Generalized Maximum Likelihood for Cross-Polarization Modulation Effects Compensation

We investigate the mitigation of nonlinearities with advanced digital signal processing focusing in particular on cross-polarization effects. Based on a relaxation of an analytical model derived for cross-polarization effects, this paper proposes a novel compensation method called generalized maximum likelihood. It performs a joint blind channel estimation and symbol detection, and it additionally accounts for the statistical prior distributions of the cross-polarization crosstalk coefficients. This avoids an overestimation of these crosstalk coefficients. A practical method for both fast computations and optimal performance is then presented, which allows nonlinear compensation for high-order modulations. Next, we present Monte-Carlo simulations showing that the proposed algorithm performs close to the theoretical limits. Large performance improvement can be obtained and this is particularly emphasized with higher order modulation such as a 16-ary quadrature amplitude modulation. Finally, using Nyquist pulse shaping and polarization-division multiplexed with a quadrature phase-shift keying modulation, the experiments are shown to be in accordance with the simulations and show up to 0.7 dB improvement in Q-factor for the worst-case samples.