Least mean square algorithms with switched Markov ODE limit

We analyze the tracking performance of a least mean square (LMS) algorithm for tracking a parameter that evolves according to a Markov chain with infrequent jumps. By allowing the Markov chain to evolve as the same rate of change as the LMS algorithm, we use a combined approach of two-time-scale Markov chains and stochastic approximation method to derive the limit dynamics satisfied by continuous-time interpolation of the estimates. Unlike most previous analyses of stochastic approximation algorithms, the limit we obtain is a system of ordinary differential equations with regime switching controlled by a continuous-time Markov chain. To further analyze the tracking errors, we take a continuous-time interpolation of a scaled sequence of the error sequence and derive its diffusion limit. Somewhat remarkably, for correlated regression vectors we obtain a system of switching diffusions.