A general procedure to combine estimators

We propose a general method to combine several estimators of the same quantity in order to produce a better estimate. In the spirit of model and forecast averaging, the final estimator is computed as a weighted average of the initial ones, where the weights are constrained to sum to one. In this framework, the optimal weights, minimizing the quadratic loss, are entirely determined by the mean square error matrix of the vector of initial estimators. The averaging estimator is derived using an estimation of this matrix, which can be computed from the same dataset. We show that the solution satisfies a non-asymptotic oracle inequality and is asymptotically optimal, provided the mean square error matrix is suitably estimated.