Can We Define a Best Estimator in Simple One-Dimensional Cases? [Lecture Notes]

What is the best estimator for assessing a parameter of a probability distribution from a small number of measurements? Is the same answer valid for a location parameter like the mean as for a scale parameter like the variance? It is sometimes argued that it is better to use a biased estimator with low dispersion than an unbiased estimator with a higher dispersion. In which cases is this assertion correct? To answer these questions, we will compare, on a simple example, the determination of a location parameter and a scale parameter with three "optimal" estimators: the minimum-variance unbiased estimator, the minimum square error estimator, and the a posteriori mean.