INFORMATION THEORETIC LEARNING: RENYI'S ENTROPY AND ITS APPLICATIONS TO ADAPTIVE SYSTEM TRAINING

[1]  Deniz Erdogmus,et al.  Lower and Upper Bounds for Misclassification Probability Based on Renyi's Information , 2004, J. VLSI Signal Process..

[2]  E. Oja,et al.  Independent Component Analysis , 2001 .

[3]  A. K. Rigler,et al.  Accelerating the convergence of the back-propagation method , 1988, Biological Cybernetics.

[4]  Deniz Erdoğmuş,et al.  COMPARISON OF ENTROPY AND MEAN SQUARE ERROR CRITERIA IN ADAPTIVE SYSTEM TRAINING USING HIGHER ORDER STATISTICS , 2004 .

[5]  Deniz Erdogmus,et al.  Generalized information potential criterion for adaptive system training , 2002, IEEE Trans. Neural Networks.

[6]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..

[7]  Amparo Alonso-Betanzos,et al.  A Global Optimum Approach for One-Layer Neural Networks , 2002, Neural Computation.

[8]  José Carlos Príncipe,et al.  Fast algorithm for adaptive blind equalization using order-α Renyi's entropy , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[9]  Jing Zhao,et al.  Simultaneous extraction of Principal Components using givens rotations and output variances , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[10]  Deniz Erdogmus,et al.  Blind source separation of time-varying instantaneous mixtures using an on-line algorithm , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Deniz Erdogmus,et al.  Entropy minimization for supervised digital communications channel equalization , 2002, IEEE Trans. Signal Process..

[12]  Deniz Erdogmus,et al.  A Neural Network Perspective to Extended Luenberger Observers , 2002 .

[13]  Santamaria,et al.  A fast algorithm for adaptive blind equalization using order-/spl alpha/ Renyi's entropy , 2002 .

[14]  Deniz Erdogmus,et al.  Insights on the relationship between probability of misclassification and information transfer through classifiers , 2002, Int. J. Comput. Syst. Signals.

[15]  D. Erdogmus,et al.  Convergence analysis of the information potential criterion in Adaline training , 2001, Neural Networks for Signal Processing XI: Proceedings of the 2001 IEEE Signal Processing Society Workshop (IEEE Cat. No.01TH8584).

[16]  Deniz Erdoğmuş,et al.  Information transfer through classifiers and its relation to probability of error , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[17]  Chong-Yung Chi,et al.  Cumulant-based inverse filter criteria for MIMO blind deconvolution: properties, algorithms, and application to DS/CDMA systems in multipath , 2001, IEEE Trans. Signal Process..

[18]  Deniz Erdoğmuş,et al.  Blind source separation using Renyi's mutual information , 2001, IEEE Signal Processing Letters.

[19]  José Carlos Príncipe,et al.  Optimization in companion search spaces: the case of cross-entropy and the Levenberg-Marquardt algorithm , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[20]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[21]  Deniz Erdogmus,et al.  AN ON-LINE ADAPTATION ALGORITHM FOR ADAPTIVE SYSTEM TRAINING WITH MINIMUM ERROR ENTROPY: STOCHASTIC INFORMATION GRADIENT , 2001 .

[22]  K. Torkkola.  Visualizing class structure in data using mutual information , 2000, Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501).

[23]  John W. Fisher,et al.  Learning from Examples with Information Theoretic Criteria , 2000, J. VLSI Signal Process..

[24]  Mark R. Titchener.  A measure of information , 2000, Proceedings DCC 2000. Data Compression Conference.

[25]  Jose C. Principe,et al.  Neural and adaptive systems : fundamentals through simulations , 2000 .

[26]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[27]  Christian Jutten,et al.  Source separation techniques applied to linear prediction , 2000 .

[28]  Richard J. Duro,et al.  Discrete-time backpropagation for training synaptic delay-based artificial neural networks , 1999, IEEE Trans. Neural Networks.

[29]  Chun Chen,et al.  A new hybrid recurrent neural network , 1999, ISCAS'99. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems VLSI (Cat. No.99CH36349).

[30]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[31]  Jean-François Bercher,et al.  Estimating the entropy of a signal with applications , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[32]  B. Farhang-Boroujeny,et al.  Adaptive Filters: Theory and Applications , 1999 .

[33]  J. Príncipe,et al.  Blind source separation using information measures in the time and frequency domains , 1999 .

[34]  Aapo Hyvärinen,et al.  Survey on Independent Component Analysis , 1999 .

[35]  Chen-Fang Chang,et al.  Observer-based air fuel ratio control , 1998 .

[36]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[37]  S. Amari,et al.  Flexible Independent Component Analysis , 1998, Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378).

[38]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[39]  John W. Fisher,et al.  A novel measure for independent component analysis (ICA) , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[40]  J. Príncipe,et al.  Training neural networks with additive noise in the desired signal , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[41]  S. Haykin,et al.  Making sense of a complex world [chaotic events modeling] , 1998, IEEE Signal Process. Mag..

[42]  W. Edmonson,et al.  A global least mean square algorithm for adaptive IIR filtering , 1998 .

[43]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[44]  J. Príncipe,et al.  Energy, entropy and information potential for neural computation , 1998 .

[45]  Durbin,et al.  Biological Sequence Analysis , 1998 .

[46]  William R. Saunders,et al.  Adaptive Structures: Dynamics and Control , 1998 .

[47]  Shun-ichi Amari,et al.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information , 1997, Neural Computation.

[48]  K. Loparo,et al.  Optimal state estimation for stochastic systems: an information theoretic approach , 1997, IEEE Trans. Autom. Control..

[49]  Christian Cachin,et al.  Smooth Entropy and Rényi Entropy , 1997, EUROCRYPT.

[50]  Prasanna K. Sahoo,et al.  Threshold selection using Renyi's entropy , 1997, Pattern Recognit..

[51]  J. Príncipe,et al.  Nonlinear extensions to the minimum average correlation energy filter , 1997 .

[52]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[53]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[54]  Shun-ichi Amari,et al.  Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient , 1996, NIPS.

[55]  B. Ripley.  Pattern Recognition and Neural Networks , 1996 .

[56]  Paul A. Viola,et al.  Empirical Entropy Manipulation for Real-World Problems , 1995, NIPS.

[57]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[58]  H. Vincent Poor,et al.  A lower bound on the probability of error in multihypothesis testing , 1995, IEEE Trans. Inf. Theory.

[59]  David B. Fogel,et al.  Alternative Neural Network Training Methods , 1995, IEEE Expert.

[60]  D. T. Kaplan,et al.  Understanding Nonlinear Dynamics , 1995 .

[61]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[62]  P. Gács.  The Boltzmann entropy and randomness tests , 1994, Proceedings Workshop on Physics and Computation. PhysComp '94.

[63]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[64]  A. Tsybakov,et al.  Root-N consistent estimators of entropy for densities with unbounded support , 1994, Proceedings of 1994 Workshop on Information Theory and Statistics.

[65]  Sergio Verdú,et al.  Generalizing the Fano inequality , 1994, IEEE Trans. Inf. Theory.

[66]  Samy Bengio,et al.  Use of genetic programming for the search of a new learning rule for neural networks , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[67]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[68]  Wray L. Buntine,et al.  Computing second derivatives in feed-forward networks: a review , 1994, IEEE Trans. Neural Networks.

[69]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[70]  C. Beck,et al.  Thermodynamics of chaotic systems : an introduction , 1993 .

[71]  N. Merhav,et al.  Relations Between Entropy and Error Probability , 1993, Proceedings. IEEE International Symposium on Information Theory.

[72]  C. L. Nikias,et al.  Higher-order spectra analysis : a nonlinear signal processing framework , 1993 .

[73]  Frank Bärmann,et al.  A learning algorithm for multilayered neural networks based on linear least squares problems , 1993, Neural Networks.

[74]  Sandro Ridella,et al.  Statistically controlled activation weight initialization (SCAWI) , 1992, IEEE Trans. Neural Networks.

[75]  Chris Bishop,et al.  Exact Calculation of the Hessian Matrix for the Multilayer Perceptron , 1992, Neural Computation.

[76]  Etienne Barnard,et al.  Optimization for training neural nets , 1992, IEEE Trans. Neural Networks.

[77]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[78]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[79]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[80]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[81]  M. A. Styblinski,et al.  Experiments in nonconvex optimization: Stochastic approximation with function smoothing and simulated annealing , 1990, Neural Networks.

[82]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[83]  P. Jones,et al.  A Diary on Information Theory , 1989 .

[84]  D. V. Gokhale,et al.  Entropy expressions and their estimators for multivariate distributions , 1989, IEEE Trans. Inf. Theory.

[85]  Anuradha M. Annaswamy,et al.  Stable Adaptive Systems , 1989 .

[86]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[87]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[88]  L. Györfi,et al.  Density-free convergence properties of various estimators of entropy , 1987 .

[89]  Ralph Linsker,et al.  Towards an Organizing Principle for a Layered Perceptual Network , 1987, NIPS.

[90]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[91]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[92]  Jan Beirlant,et al.  The empirical distribution function and strong laws for functions of order statistics of uniform spacings , 1985 .

[93]  S. Qureshi,et al.  Adaptive equalization , 1982, Proceedings of the IEEE.

[94]  Shun-ichi Amari,et al.  Differential-geometrical methods in statistics , 1985 .

[95]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[96]  P. Hall.  Limit theorems for sums of general functions of m-spacings , 1984, Mathematical Proceedings of the Cambridge Philosophical Society.

[97]  Simon Haykin,et al.  Introduction to Adaptive Filters , 1984 .

[98]  Graham C. Goodwin,et al.  Adaptive filtering prediction and control , 1984 .

[99]  J. Treichler,et al.  A new approach to multipath correction of constant modulus signals , 1983 .

[100]  P. Bickel,et al.  Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit Test , 1983 .

[101]  Erkki Oja,et al.  Subspace methods of pattern recognition , 1983 .

[102]  Reuven Y. Rubinstein,et al.  Simulation and the Monte Carlo method , 1981, Wiley series in probability and mathematical statistics.

[103]  D. Donoho.  ON MINIMUM ENTROPY DECONVOLUTION , 1981 .

[104]  D. Godard,et al.  Self-Recovering Equalization and Carrier Tracking in Two-Dimensional Data Communication Systems , 1980, IEEE Trans. Commun..

[105]  A. Benveniste,et al.  Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications , 1980 .

[106]  M. Ben-Bassat,et al.  Renyi's entropy and the probability of error , 1978, IEEE Trans. Inf. Theory.

[107]  Pushpa N. Rathie,et al.  On the entropy of continuous probability distributions (Corresp.) , 1978, IEEE Trans. Inf. Theory.

[108]  Ibrahim A. Ahmad,et al.  A nonparametric estimation of the entropy for absolutely continuous distributions (Corresp.) , 1976, IEEE Trans. Inf. Theory.

[109]  D. P. Mittal.  On additive and non-additive entropies , 1975, Kybernetika.

[110]  Yuriy G. Dmitriev,et al.  On the Estimation of Functionals of the Probability Density and Its Derivatives , 1974 .

[111]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[112]  Alfréd Rényi,et al.  Probability Theory , 1970 .

[113]  F. P. Tarasenko.  On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable, and the distribution-free entropy test of goodness-of-fit , 1968 .

[114]  R. Gallager.  Information Theory and Reliable Communication , 1968 .

[115]  L. L. Campbell,et al.  A Coding Theorem and Rényi's Entropy , 1965, Inf. Control..

[116]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[117]  G. A. Barnard,et al.  Transmission of Information: A Statistical Theory of Communications. , 1961 .

[118]  S. Kullback.  Information Theory and Statistics , 1959 .

[119]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series, with Engineering Applications , 1949 .

[120]  F. Attneave,et al.  The Organization of Behavior: A Neuropsychological Theory , 1949 .

[121]  Claude E. Shannon,et al.  A Mathematical Theory of Communications , 1948 .