Regularization with hard penalty functions for on-line estimation

We present two on-line methods for detecting changes and estimating parameters in AR(X) models. The methods are based on the assumption of piecewise constant parameters resulting in a sparse structure of their derivative. Smoothly Clipped Absolute Deviation (SCAD) and hard thresholding (HT) penalty functions are two alternatives to give a sparse structure of the estimate. We use local quadratic and linear approximations of the penalty function, and the optimization is carried out by using a modified Newton-Raphson algorithm. To illustrate the algorithms and their performance, we apply them to estimate changing parameters of an ARX model. We investigate the convergence, squared error, and sparsity of the methods. The examples indicate that the local linear approximation gives better performance and is more encouraging than the local quadratic approximation, and SCAD provides faster and better estimation of parameters.