On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes

by Andrew Y. Ng and Michael I. Jordan

This paper is a little bit theoretical. The main result is that the discriminative model usually has a lower error rate than its corresponding generative model given enough data (asymptotically speaking, the generalization capability of the former is better than the latter). However, it's easier to get to the bound for the latter model (it converges faster).

But please do notice there are something not validated. The paper compares the model of Naive Bayes and logistic regression. The conclusion might not generalize to other models. And the training of logistic regression is not MLE (minimizing the classification errors, though).

The gap between the two asymptotic bounds is O( 1 / m * log n)., where m is the number of samples and n the dimension of feature space.

This implies with limited data, a generative model might be better. With plenty of data, a discriminative model might be more suitable.