Colored Noise and Regularization Parameter Selection for Waveform Metrology

We study six algorithms to select a regularization parameter for deconvolution problems appearing in high-speed communication measurement applications. We investigate these algorithms in the presence of unspecified noise correlation analyzing their performance as components of a multivariate random variable and study their joint distribution by Monte Carlo. We find that several parameter selection algorithms, despite their widespread use, are not robust to unspecified noise correlations. Specifically, the discrepancy principle fails to return adequate regularizations for rough noise, while the generalized cross validation (GCV), unbiased predictive risk, and information complexity selectors can fail for smooth noise. For some experimental configurations, GCV failed completely, returning zero successful inversions out of 500 noise instantiations. These parameter selection algorithms share in the characteristic that they do not contain mechanisms to monitor quantities derived from the parameter-dependent solution vector. By contrast, the L-curve and quasi-optimality criteria do contain such mechanisms, and furthermore exhibited significantly fewer failures and correlated highly with the optimal inversion across all noise levels and correlations.