An Empirical Comparison of Supervised Learning Algorithms

by Rich Caruana and Alexandru Niculescu-Mizil

I think the most important thing in this paper is that it shows lots of information that I don't know, but need indeed. Here is some quoted information:

We evaluate the performance of SVMs, neural nets, logistic regression, naive bayes, memory-based learning, random forests, decision trees, bagged trees, boosted trees, and boosted stumps on eleven binary classification problems using a variety of performance metrics: accuracy, F-score, Lift, ROC Area, average precision, precision/recall break-even point, squared error, and cross-entropy... we compare the performance of each algorithm both before and after calibrating its predictions with Platt Scaling and Isotonic Regression.

Here, we need to know more about those algorithms and evaluation methods.

With excellent performance on all eight metrics, calibrated boosted trees were the best learning algorithm overall. Random forests are close second, followed by uncalibrated bagged trees, calibrated SVMs, and uncalibrated neural nets. The models that performed poorest were naive bayes, logistic regression, decision trees, and boosted stumps. Although some methods clearly perform better or worse than other methods on average, there is significant variability across the problems and metrics. Even the best models sometimes perform poorly, and models with poor average performance occasionally perform exceptionally well.

So it is really sth surprising :-p