AN EFFICIENT RANK-DEFICIENT COMPUTATION OF THE PRINCIPLE OF RELEVANT INFORMATION

One of the main difficulties in computing information theoretic learning (ITL) estimators is the computational complexity that grows quadratically with data. Considerable amount of work has been done on computation of low rank approximations of Gram matrices without accessing all their elements. In this paper we discuss how these techniques can be applied to reduce computa- tional complexity of Principle of Relevant Information (PRI). This particular objective function involves estimators of Renyi's sec- ond order entropy and cross-entropy and their gradients, therefore posing a technical challenge for implementation in a realistic sce- nario. Moreover, we introduce a simple modification to the Nystrom method motivated by the idea that our estimator must perform accu- rately only for certain vectors not for all possible cases. We show some results on how this rank deficient decompositions allow the application of the PRI on moderately large datasets. A major issue, which we address in this paper, is that the amount of computation associated to the PRI grows quadratically with the size of the available sample. This limits the scale of the applications if one were to apply the formulas directly. The problem of poly- nomial growth on complexity has also received attention within the machine learning community working on kernel methods. Conse- quently, approaches to compute approximations to positive semidefi- nite matricesbased on kernels have been proposed (6, 7). The goal of these methods is to accurately estimate large Gram matrices without computing their n 2 elements, directly. It has been observed that in practice the eigenvalues of the Gram matrix drop rapidly and there- fore replacing the original matrixby alow rank approximation seems reasonable(7, 8). In our work, we derive an algorithm for the princi- ple of relevant information based on rank deficient approximations of a Gram matrix. We also propose a simple modified version of the Nystrom method particularly suited for estimation in ITL. The paper starts with a brief introduction to Renyi's Entropy and the associated information quantities with their corresponding rank deficient approximations. Then, the objective function for the prin- ciple of relevant information (PRI) is presented. Following, we pro- pose an implementation of the optimization problem based on rank deficient approximations. The algorithm is tested on simulated data for various accuracy regimes (different ranks) followed by some re- sults on realistic scenarios. Finally, we provide some conclusions along with future work directions.