Regime Switching Stochastic Approximation Algorithms with Application to Adaptive Discrete Stochastic Optimization

This work is devoted to a class of stochastic approximation problems with regime switching modulated by a discrete-time Markov chain. Our motivation stems from using stochastic recursive algorithms for tracking Markovian parameters such as those in spreading code optimization in CDMA (code division multiple access) wireless communication. The algorithm uses constant step size to update the increments of a sequence of occupation measures. It is proved that least squares estimates of the tracking errors can be developed. Assume that the adaptation rate is of the same order of magnitude as that of the time-varying parameter, which is more difficult to deal with than that of slower parameter variations. Due to the time-varying characteristics and Markovian jumps, the usual stochastic approximation (SA) techniques cannot be carried over in the analysis. By a combined use of the SA method and two-time-scale Markov chains, asymptotic properties of the algorithm are obtained, which are distinct from the usual SA ...

[1]  S. Andradóttir,et al.  A Simulated Annealing Algorithm with Constant Temperature for Discrete Stochastic Optimization , 1999 .

[2]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[3]  Yu-Chi Ho,et al.  Stochastic Comparison Algorithm for Discrete Optimization with Estimation , 1999, SIAM J. Optim..

[4]  B. S.  Simulated Annealing with Noisy or Imprecise Energy Measurements , 2004 .

[5]  Andrew L. Rukhin,et al.  Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach , 2001, Technometrics.

[6]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[7]  Gang George Yin,et al.  Singularly Perturbed Discrete-Time Markov Chains , 2000, SIAM J. Appl. Math..

[8]  D. Yan,et al.  Stochastic discrete optimization , 1992 .

[9]  Xuan Kong,et al.  Adaptive Signal Processing Algorithms: Stability and Performance , 1994 .

[10]  A Orman,et al.  Optimization of Stochastic Models: The Interface Between Simulation and Optimization , 2012, J. Oper. Res. Soc..

[11]  Gang George Yin,et al.  Adaptive spreading code optimization in multiantenna multipath fading channels in CDMA , 2003, IEEE International Conference on Communications, 2003. ICC '03..

[12]  G. Pflug.  Stochastic Approximation Methods for Constrained and Unconstrained Systems - Kushner, HJ.; Clark, D.S. , 1980 .

[13]  Sigrún Andradóttir,et al.  Accelerating the convergence of random search methods for discrete stochastic optimization , 1999, TOMC.

[14]  Harold J. Kushner,et al.  Approximation and Weak Convergence Methods for Random Processes , 1984 .

[15]  Gang George Yin,et al.  Spreading code optimization and adaptation in CDMA via discrete stochastic approximation , 2004, IEEE Transactions on Information Theory.

[16]  Tamer Basar,et al.  Analysis of Recursive Stochastic Algorithms , 2001 .

[17]  Gang George Yin,et al.  Recursive algorithms for estimation of hidden Markov models and autoregressive models with Markov regime , 2002, IEEE Trans. Inf. Theory.

[18]  D. Vere-Jones.  Markov Chains , 1972, Nature.

[19]  George Yin,et al.  Discrete-time singularly perturbed Markov chains: aggregation, occupation measures, and switching diffusion limit , 2003, Advances in Applied Probability.

[20]  Sigrún Andradóttir,et al.  A Global Search Method for Discrete Stochastic Optimization , 1996, SIAM J. Optim..

[21]  Lee W. Schruben,et al.  A survey of simulation optimization techniques and procedures , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[22]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[23]  Han-Fu Chen.  Stochastic approximation and its applications , 2002 .