Semi-tied covariance matrices

A standard problem in many classification tasks is how to model feature vectors whose elements are highly correlated. If multi-variate Gaussian distributions are used to model the data then they must have full covariance matrices to accurately do so. This requires a large number of parameters per distribution which restricts the number of distributions that may be robustly estimated, particularly when high dimensional feature vectors are required. This paper describes an alternative to full covariance matrices in these situations. An approximate full covariance matrix is used. The covariance matrix is now split into two elements, one full and one diagonal, which may be tied at completely separate levels. Typically, the full elements are extensively tied, resulting in only a small increase in the number of parameters compared to the diagonal case. Thus dramatically increasing the number of distributions that may be robustly estimated. Simple iterative re-estimation formulae for all the parameters within the standard EM framework are presented. On a large vocabulary speech recognition task a 10% reduction in word error rate over a standard system was achieved