Structural adaptation in mixture of experts

The "mixture of experts" framework provides a modular and flexible approach to function approximation. However, the important problem of determining the appropriate number and complexity of experts has not been fully explored. In this paper, we consider a localized form of the gating network that can perform function approximation tasks very well with only one layer of experts. Certain measures for the smooth functioning of the training algorithm to train this model are described first. We then propose two techniques to overcome the model selection problem in the mixture of experts architecture. In the first technique, we present an efficient way to grow expert networks to come up with an appropriate number of experts for a given problem. In the second approach, we start with a certain number of experts and present methods to prune experts which become less useful and also add on experts which would be more effective. Simulation results are presented which support the techniques proposed. We observe that the growing/pruning approach yields substantially better results than the standard approach even when the final network sizes are chosen to be the same.