Contravariant adaptation on structured parameter spaces

The contravariant vector associated with the conventional gradient vector (covector) via the Riemannian metric is the appropriate direction for gradient descent learning. This fact is the basis for Amari´s familiar natural gradient learning algorithms. The language of differential geometry is used to derive the contravariant gradient rule for parameterizations of invertible matrices as pullbacks of invariant Riemannian metrics. Contravariant adaptation rules are derived for several structured matrices - including Toeplitz, inverse Toeplitz (Bezout) and orthogonal matrices - which can be used for such problems as source separation when the mixing or unmixing matrices are structured.