Asymptotic analysis of stochastic approximation algorithms under violated Kushner-Clark conditions with applications

Motivated by the problem of the asymptotic behavior of temporal-difference learning algorithms with nonlinear function approximation, the local almost sure asymptotic properties of stochastic approximation algorithms are analyzed for violated Kushner-Clark conditions (1978). First, the algorithms with additive noise are analyzed for the case where the noise is state-dependent. The obtained results are then applied to the analysis of the algorithms with nonadditive noise. Using these general results, the analysis of temporal-difference learning algorithms is carried out for the case of a general nonlinear function approximation and under the assumptions allowing the underlying Markov chain to be positive Harris. The general results are also illustrated by an example where the noise is nonadditive, correlated and satisfies strong mixing conditions.