Hierarchical Mixtures of Experts and the EM Algorithm

by Michael I. Jordan and Robert A. Jacobs

This paper formulates HMoE, which can be regarded as a soft decision tree. Each leaf node is an expert, which is usually a GLIM and whose parameters are to be estimated. The internal nodes are soft partitions (something like a multinomial regression, the number of classes equals the number of children). Therefore each internal node actually corresponds to a hidden r.v. whose parameters (the coefficients for inner product) are to be estimated. On a whole the model is a Bayesian belief network (a poly-tree?). The training of this kind of model can be done with EM algorithm and the posterior of the hidden variables must be calculated (via belief propagation, a typical inference problem). This is an old paper, which was wrongly picked out -,-b