Adaptive Projected Subgradient Method for Asymptotic Minimization of Sequence of Nonnegative Convex Functions

Abstract This paper presents an algorithm, named adaptive projected subgradient method that can minimize asymptotically a certain sequence of nonnegative convex functions over a closed convex set in a real Hilbert space. The proposed algorithm is a natural extension of the Polyak's subgradient algorithm, for nonsmooth convex optimization problem with a fixed target value, to the case where the convex objective itself keeps changing in the whole process. The main theorem, showing the strong convergence of the algorithm as well as the asymptotic optimality of the sequence generated by the algorithm, can serve as a unified guiding principle of a wide range of set theoretic adaptive filtering schemes for nonstationary random processes. These include not only the existing adaptive filtering techniques; e.g., NLMS, Projected NLMS, Constrained NLMS, APA, and Adaptive parallel outer projection algorithm etc., but also new techniques; e.g., Adaptive parallel min-max projection algorithm, and their embedded constraint versions. Numerical examples show that the proposed techniques are well-suited for robust adaptive signal processing problems.

[1]  Z. Opial.  Weak convergence of the sequence of successive approximations for nonexpansive mappings , 1967 .

[2]  Boris Polyak,et al.  The method of projections for finding the common point of convex sets , 1967 .

[3]  J. Nagumo,et al.  A learning method for system identification , 1967, IEEE Transactions on Automatic Control.

[4]  Boris Polyak.  Minimization of unsmooth functionals , 1969 .

[5]  D. Meeter.  Stochastic Approximation and Nonlinear Regression , 1969 .

[6]  T. Hinamoto,et al.  Extended theory of learning identification , 1975 .

[7]  D. Youla,et al.  Image Restoration by the Method of Convex Projections: Part 1ߞTheory , 1982, IEEE Transactions on Medical Imaging.

[8]  Guy Pierra,et al.  Decomposition through formalization in a product space , 1984, Math. Program..

[9]  S. Reich,et al.  Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings , 1984 .

[10]  Kazuhiko Ozeki,et al.  An adaptive filtering algorithm using an orthogonal projection to an affine subspace and its properties , 1984 .

[11]  H. Trussell,et al.  The feasible solution in signal restoration , 1984 .

[12]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[13]  Patrick L. Combettes,et al.  The use of noise properties in set theoretic estimation , 1991, IEEE Trans. Signal Process..

[14]  Dan Butnariu,et al.  Convergence criteria for generalized gradient methods of solving locally Lipschitz feasibility problems , 1992, Comput. Optim. Appl..

[15]  P. L. Combettes,et al.  Foundation of set theoretic estimation , 1993 .

[16]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[17]  P. L. Combettes.  The foundations of set theoretic estimation , 1993 .

[18]  Markus Rupp,et al.  The behavior of LMS and NLMS algorithms in the presence of spherically invariant processes , 1993, IEEE Trans. Signal Process..

[19]  P. L. Combettes.  Construction d'un point fixe commun à une famille de contractions fermes , 1995 .

[20]  V. V. Vasin,et al.  Ill-posed problems with a priori information , 1995 .

[21]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[22]  Ali H. Sayed,et al.  H∞ optimality of the LMS algorithm , 1996, IEEE Trans. Signal Process..

[23]  Patrick L. Combettes,et al.  Convex set theoretic image recovery by extrapolated iterations of parallel subgradient projections , 1997, IEEE Trans. Image Process..

[24]  John F. Doherty,et al.  Generalized projection algorithm for blind interference suppression in DS/CDMA communications , 1997 .

[25]  Shirish Nagaraj,et al.  Set-membership filtering and a set-membership normalized LMS algorithm with an adaptive step size , 1998, IEEE Signal Processing Letters.

[26]  Markus Rupp.  A family of adaptive filter algorithms with decorrelating properties , 1998, IEEE Trans. Signal Process..

[27]  George-Othon Glentis,et al.  Efficient least squares adaptive algorithms for FIR transversal filtering , 1999, IEEE Signal Process. Mag..

[28]  Jacob Benesty,et al.  Acoustic signal processing for telecommunication , 2000 .

[29]  I. Yamada.  The Hybrid Steepest Descent Method for the Variational Inequality Problem over the Intersection of Fixed Point Sets of Nonexpansive Mappings , 2001 .

[30]  Heinz H. Bauschke,et al.  A Weak-to-Strong Convergence Principle for Fejé-Monotone Methods in Hilbert Spaces , 2001, Math. Oper. Res..

[31]  Patrick L. Combettes,et al.  An adaptive level set method for nondifferentiable constrained image recovery , 2002, IEEE Trans. Image Process..

[32]  Isao Yamada,et al.  An efficient robust adaptive filtering algorithm based on parallel subgradient projection techniques , 2002, IEEE Trans. Signal Process..

[33]  I. Yamada.  A numerically robust hybrid steepest descent method for the convexly constrained generalized inverse problems , 2002 .

[34]  I. Yamada,et al.  NON-STRICTLY CONVEX MINIMIZATION OVER THE FIXED POINT SET OF AN ASYMPTOTICALLY SHRINKING NONEXPANSIVE MAPPING , 2002 .

[35]  José Antonio Apolinário,et al.  Constrained adaptation algorithms employing Householder transformation , 2002, IEEE Trans. Signal Process..

[36]  A. Ekpenyong,et al.  Frequency-domain adaptive filtering: a set-membership approach , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[37]  Isao Yamada,et al.  Adaptive projected subgradient method and its applications to set theoretic adaptive filtering , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[38]  Two Generalizations of the Projected Gradient Method for Convexly Constrained Inverse Problems--Hybrid steepest descent method, Adaptive projected subgradient method (数値解析と新しい情報技術 研究集会報告集) , 2004 .

[39]  I. Yamada,et al.  Two Generalizations of the Projected Gradient Method for Convexly Constrained Inverse Problems : Hybrid steepest descent method, Adaptive projected subgradient method (Numerical Analysis and New Information Technology) , 2004 .

[40]  Patrick L. Combettes,et al.  Image restoration subject to a total variation constraint , 2004, IEEE Transactions on Image Processing.

[41]  Isao Yamada,et al.  A fast blind multiple access interference reduction in DS/CDMA systems by adaptive projected subgradient method , 2004, 2004 12th European Signal Processing Conference.

[42]  Isao Yamada,et al.  Efficient Adaptive Stereo Echo Canceling Schemes Based on Simultaneous Use of Multiple State Data , 2004 .

[43]  I. Yamada,et al.  Hybrid Steepest Descent Method for Variational Inequality Problem over the Fixed Point Set of Certain Quasi-nonexpansive Mappings , 2005 .