On the convergence and ODE limit of a two-dimensional stochastic approximation

We consider a two-dimensional stochastic approximations scheme of the Robbins-Monro type which naturally arises in the study of steering policies for Markov decision processes. Making use of a decoupling change of variables, we establish its almost sure convergence by ad-hoc arguments that combine standard results on one-dimensional stochastic approximations with a version of the law of large numbers for martingale differences. We use this direct analysis to guide us in selecting the test function which appears in standard convergence results for multidimensional schemes. Furthermore, although a blind application of the ordinary differential equation (ODE) method is not possible here due to a lack of regularity properties, the aforementioned change of variables paves the way for an interpretation of the behavior of solutions to the associated limiting ODE