A Rigorous Analysis of Least Squares Sine Fitting Using Quantized Data: The Random Phase Case

This paper considers least square (LS) based estimation of the amplitude and square amplitude of a quantized sine wave, done by considering random initial record phase. Using amplitude- and frequency-domain modeling techniques, it is shown that the estimator is inconsistent, biased, and has a variance that may be underestimated if the simple model of quantization is applied. The effects of both sine wave offset values and additive Gaussian noise are considered. General estimator properties are derived, without making simplifying assumptions on the role of the quantization process, to allow assessment of measurement uncertainty, when this LS procedure is used.