The LMS, PNLMS, and exponentiated gradient algorithms

Sparse impulse responses are encountered in many applications (network and acoustic echo cancellation, feedback cancellation in hearing aids, etc). Recently, a class of exponentiated gradient (EG) algorithms has been proposed. One of the algorithms belonging to this class, the so-called EG± algorithm, converges and tracks much better than the classical stochastic gradient, or LMS, algorithm for sparse impulse responses. In this paper, we show how to derive the different algorithms. We analyze the EG± algorithm and explain when to expect it to behave like the LMS algorithm. It is also shown that the proportionate normalized LMS (PNLMS) algorithm proposed by Duttweiler in the context of network echo cancellation is an approximation of the EG±.