Gaussian Processes for Regression

by Christopher K. I. Williams and Carl Edward Rasmussen

This paper might be the first one introducing GP into machine learning society. Now it is very clear to me about certain techniques: how to compute the posterior and make predictions. The training of non-parametric models requires tuning the hyperparameters (MAP -,-b in a sense, or Type II MLE) or calculating the posterior of hyperparameters (fully Bayesian way). In practice the empirical Bayesian method works pretty well. But as we know, to maximize the marginal likelihood Pr( x ), we have to calculate the partial derivatives of all hyperparameters.

That is why the matlab code the author provides include a nonlinear CG algorithms. I guess it is also possible to write another with second-order optimization techniques. The connection of GP and neural nets is addressed in Neal's earlier work.