Ranking estimation performance by estimator randomization and attribute support

To rank performance of estimators, we propose to use weighted averaging based on estimator randomization and attribute support, respectively. We assume that the "best" estimator random and may be any of the considered estimators. Different error metrics provide different observations of the random variable. A better estimator has a larger probability to be the "best" one according to an error metric, and thus we can translate the data of the error metric to the probability mass function (pmf) of the random variable conditioned on that error metric. We combine the pmfs by corresponding weights to obtain a fused pmf that can be used to rank the estimators. The weights are determined by an observation support vector (OSV) obtained by an observation support matrix (OSM) that is composed of similarity of each pmf pair in terms of the proposed Kullback-Leibler ratio divergence (KLRD). Weighted averaging based on attribute support needs data normalization, and its weights are determined by an attribute support vector (ASV) obtained by an attribute support matrix (ASM) that is composed of pairwise cosine similarity of attributes. The idea of estimator randomization and attribute support can also be used to solve other multiple-attribute ranking problems.