Gradient Learning in Structured Parameter Spaces: Adaptive Blind Separation of Signal Sources

The present paper discusses the natural gradient descent learning rules in parameter spaces which have Riemannian geometrical structures. A modi cation is necessary for de ning the steepest descent (gradient) direction in a Riemannian parameter space. Parameter spaces of multilayer perceptrons are good examples of the Riemannian nature. Another example is the space of matrices on which adaptive blind separation of mixtured signals takes place. The ICA (independet component analysis) learning method is discussed from the geometrical point of view. We also show other evidences how the natural gradient learning method is e ective. 1 Gradient Descent Learning Rule Let us consider an observable random variable z and a network speci ed by a parameter vector w 2 R n . Let l(z;w) be a loss function when we use parameter w in processing z. A typical example is the multilayer neural network, where z consists of input vector x and the desired or teacher output vector y, z = (x; y), w is the modi able parameters of the network such that the output of the network is speci ed by