Estimation with Information Theoretic Criteria: An Interpetation

The estimation problem on arbitrary statistical environment is considered when the performance of the estimator is measured using information theoretical criteria. Existence and uniqueness conditions are discussed for continuous ensembles. Analytic sufficient conditions for the minimum error entropy estimator are derived. In a few circumstances the minimum error entropy estimator is derived explicitly. In discrete probability ensembles, it is shown that the results of the estimation with information criteria must be interpreted from a coding point of view.