Stochastic Gradient Algorithm Under (h,φ)-Entropy Criterion

Abstract Motivated by the work of Erdogmus and Principe, we use the error (h,φ)-entropy as the supervised adaptation criterion. Several properties of the (h,φ)-entropy criterion and the connections with traditional error criteria are investigated. By a kernel estimate approach, we obtain the nonparametric estimator of the instantaneous (h,φ)-entropy. Then, we develop the general stochastic information gradient algorithm, and derive the approximate upper bound for the step size in the adaptive linear neuron training. Moreover, the (h,φ) pair are optimized to improve the performance of the proposed algorithm. For the finite impulse response identification with white Gaussian input and noise, the exact optimum φ function is derived. Finally, simulation experiments verify the results and demonstrate the noticeable performance improvement that may be achieved by the optimum (h,φ)-entropy criterion.