Effects of FFT coefficient quantization on sinusoidal signal detection

Detection of a sinusoid of unknown frequency in wideband noise is performed efficiently by the FFT (fast Fourier transform). The detector performs a hypothesis test on the magnitude of the FFT output. When the FFT is implemented, error due to arithmetic roundoff coefficient quantization limits the accuracy of the transform and degrades the detection performance. When the FFT is used as a detector of an unknown sinusoidal signal, the coefficient quantization error is significant and increases with the FFT length. The decimation is analyzed in time for a radix-2 FFT. The FFT output error is defined to be the maximum magnitude of the difference between the true FFT and FFT computed with the quantized coefficients. An upper bound on the error is derived by a deterministic analysis and is verified to be close to the actually measured error. Using the functional form of the bound and scaling it to fit the measured error, an empirical formula for the error is derived. The probability of detection of the quantized-coefficient FFT is computed using the empirical error formula. The probability of detection curves is presented as a function of the FFT length. Simulations indicate that when a sufficient number of bits is used to quantize the coefficients, the probability of detection does not significantly degrade