Replicated Softmax: an undirected topic model

by Ruslan Salakhutdinov and Geoffrey Hinton

This paper talks about an application of restricted Boltzmann machine. The visible layer contains a matrix of 0-1 variables (indicating whether the kth sample takes value i). And the hidden layer is again binary variables. After defining the conditional probability (since the conditional probability for visible layer is multinomial and therefore a softmax function on the hidden units, that's where the model's name comes from; replicated for the weight matrix is shared), we may use contrastive divergence to train the model. In topic model, the so-called log-perplexity (probability of observing testing samples) has to be computed so as to compare with other topic models (such as LDA). Annealed importance sampling is employed to compute the partition function.

There is some link of this model with semantic hash previously developed by the authors. Experiments shows favorable result with undirected topic model.

The Recurrent Temporal Restricted Boltzmann Machine

by Hya Sutskever, Geoffrey Hinton and Graham Taylor

This paper is succedent to the previous motion caption paper, which does not include much details about the model (at least I don't know anything about that dynamic biases). Anyway, I find more details in this paper. This dynamic bias is actually a cross term of h_t and h_t-1. So given h_t-1, it is functioning as the bias term.

The model proposed in this paper is a bit strange. I will read something about recurrent networks first. From the point of view of optimization, it uses BP instead of CD learning (exact vs stochastic approximation).

Anyway, I will return to this paper later. It is really weird to make a temporal model with RBM compared with HMM, CRF.

Implicit Mixtures of Restricted Boltzmann Machines

by Vinod Nair and Geoffrey Hinton

This is quite simple extension. It employs a three-order RBM instead of a mixture of RBM (mainly due to the complexity of training RBM). Usually we use v for input, h for hidden neurons. Now add another z for clustering. And v and h are binomial while z is actually a multinomial r.v. (so that it corresponds to the clustering index). This model can be trained with CD. We know given z and h the sampling of v is simple. The only thing we need to consider is how to sample h and z given v. This is still easy. One way is to find z first (well it has a number of configurations linear with the number of clusters, instead of exponential). By marginalize h, we get the free energy for each z. Then a softmax function (or just a normalization of the probability) will tell us the distribution of z. Then we can sample h.

Therefore, the three-order RBM is trained with CD. They address a problem of the training. One RBM might be too strong and others won't be chosen forever. They use a technique from annealing. By dividing the free energy with a temperature T (first high later getting smaller), we might avoid this problem.

Generative versus Discriminative Training of RBMs for Classification of fMRI Images

by Tanya Schmah, Geoffrey E. Hinton, Richard S. Zemel, Steven L. Small, Stephen Strother

This is much simpler than the previous RBM. It simply train one RBM for each class. Then they use Bayes theorem to calculate the posterior probability to classify (with the assumption that the prior is equal of each class). The only difficult part is the normalization factors. They use a discriminative way of learning for this ratio. In the discriminative learning, the fraction is something like a logistic function and the partial derivatives can be calculated very easily.

They carry out the experiments on fMRI data, on which logistict regression with regularization does not work well.

Sparse Deep Belief Net Model for Visual Area V2

by Honglak Lee, Chaintanya Ekanadham and Andrew Y. Ng

In fact the algorithm is quite simple. The sparsity is enforced by a norm regularizer (the conditional expectation of the hidden states, don't know why it works). This will not modify the CD learning process much. We simply add another update of the parameters, which is deterministic and very easy to compute.

The paper argues that the sparse RBM and DBN could learn Gabor-like filters from the natural images. And the corner-oriented filters are believed to be the V2 area in human cortex.

Modeling Human Motion Using Binary Latent Variables

by Graham W. Taylor, Geoffrey E. Hinton and Sam Roweis

This paper introduces a method that enables RBM to deal with temporal data. The concept proposed is conditional RBM (CRBM): adding two flavors of directed edges to the RBM model (which is undirected). I am not sure how they design the whole model. It is said the two flavors of connections are treated as dynamical biases and the updating rule is quite similar to the bias in normal RBM.

The thing is that given previous frames, it is now possible to influence the distribution of the current frame (I guess that is where the `conditional' term comes in). This model does something quite similar to HMM, since they are both generative and models for temporal data.

In Hinton's course, he suggests a stacked version of CRBM (just like DBN to RBM).

I am not sure whether this idea is really good. It looks reasonable but I have no idea about the details. I will put it there and wait for new progress.

Classification using Discriminative Restricted Boltzmann Machines

by Hugo Larochelle and Yoshua Bengio

This paper formulates a ``discriminative'' RBM. The visible layer contains the features and the labels (in a softmax way). It can be trained in a generative way and a discriminative one. The generative one maximizes Pr( x, y ) while the latter maximizes Pr( y | x ). As we can see, Pr( x, y ) could be maximized with contrastive divergence as the normal RBM. And I remember Hinton's course last year includes an example of classification using free energy directly with this model. Now for the discriminative case, when the number of classes is small, a exhaustive sum over y is feasible. The good news is that the partial differentials can be obtained with explicit expressions.

Regarding last scanned paper, they propose a hybrid model that relies on two loss functions (negated log-likelihoods, introducing a coefficient for the generative part). Then the discriminative part has a non-stochastic gradient and the generative part has a stochastic one (due to CD).

As for semi-supervised learning, we just marginalize out the label y from the RBM. Then the gradient becomes the CD averaged over y. This negated likelihood of Pr(x) will function as a regularizer.

It's amazing the DRBM alone could achieve 1.81% error rate on MNIST with only 500 samples. The HDRBM could achieve even lower. They mention a sparse version which I will scan soon.

Modeling Image Patches with a Directed Hierarchy of Markov Random Fields

by Simon Osindero and Geoffrey Hinton

As I have read previously, RBM is the building block of DBM. Here a semi-restricted Boltzmann machine is proposed to model image patches. An SRBM allow lateral connection between visible states.

In this case, the hidden neurons are still conditionally independent given the visible layer. And it's easily seen that it is still a PoE and could be trained with contrastive divergence. The problematic point is that after we sample the hidden states, it's not as easy as in the RBM to sample the visible states. In RBM, a simple mean field approximation will be enough, but SRBM has connections between visible neurons, which means we can't sample each state independent of others. Here a damped mean field approximation is used: each time the mean field approximation and the last time probability (inintial probability is the training sample).

With SRBM, we could model patches from natural images easily, as with RBM. The point of using SRBM instead of RBM is that the correlation between pixels can't be modeled via RBMs. Their experiment shows this point.