New advances on Bayesian Ying-Yang learning system with Kullback and non-Kullback separation functionals

In this paper, we extend Bayesian-Kullback Ying-Yang (BKYY) learning into a much broader Bayesian Ying-Yang (BYY) learning system via different separation functionals instead of using only Kullback divergence, and elaborate the power of BYY learning as a general learning theory for parameter learning, scale selection, structure evaluation, regularization and sampling design. Improved criteria are proposed for selecting number of densities on finite mixture and Gaussian mixtures, for selecting number of clusters in MSE clustering, for selecting subspace dimension in PCA related methods, for selecting number of expert nets in mixture of experts and its alternative model and for selecting number of basis functions in RBF nets. Three categories of non-Kullback separation functionals namely convex divergence, L/sub p/ divergence and decorrelation index, are suggested for BYY learning as alternatives for those learning models based on Kullback divergence, with some properties discussed. As examples, the EM algorithms for finite mixture, mixture of experts and its alternative model are derived with convex divergence.