Nonlinear Equalization of Binary Signals in Gaussian Noise

A maximum a posteriori probability (MAP) approach is used to derive the optimal equalizer for binary signals transmitted at a constant rate over a dispersive and noisy channel. The optimal equalizer is shown to consist of a matched filter followed by a very complex nonlinear transversal filter. Various approximations are made to simplify the transversal filter. Thus the optimal linear equalizer and two suboptimal nonlinear equalizers are obtained. The performance characteristics of these equalizers are compared in terms of eye patterns and error probabilities. These investigations exhibit a clear superiority of the nonlinear equalizers. Up to a certain peak distortion, intersymbol interference can be eliminated by nonlinear equalization With negligible noise enhancement. The error probability then closely approaches the error probability in the case of noninterfering signals.