Signal Design and Detection in Presence of Nonlinear Phase Noise

In optical fiber transmission systems using inline amplifiers, the interaction of a signal and amplifier noise through the Kerr effect leads to nonlinear phase noise that can impair the detection of phase-modulated signals. We present analytical expressions for the maximum-likelihood (ML) decision boundaries and symbol-error rate (SER) for phase-shift keying and differential phase-shift keying systems with coherent and differentially coherent detection, respectively. The ML decision boundaries are in the form thetas(r) = c₂r² + c₁r + c₀, where thetas and r are the phase and the amplitude of the received signal, respectively. Using the expressions for the SER, we show that the impact of phase error from carrier synchronization is small, particularly for transoceanic links. For modulation formats such as 16-quadrature amplitude modulation, we propose various transmitter and receiver phase rotation strategies such that the ML detection is well approximated by using straight-line decision boundaries. The problem of signal constellation design for optimal SER performance is also studied for a system with four signal points.