Design of APSK Constellations for Coherent Optical Channels with Nonlinear Phase Noise

We study the design of amplitude phase-shift keying (APSK) constellations for a coherent fiber-optical communication system where nonlinear phase noise (NLPN) is the main system impairment. APSK constellations can be regarded as a union of phase-shift keying (PSK) signal sets with different amplitude levels. A practical two-stage (TS) detection scheme is analyzed, which performs close to optimal detection for high enough input power. We optimize APSK constellations with 4, 8, and 16 points in terms of symbol error probability (SEP) under TS detection for several combinations of input power and fiber length. For 16 points, performance gains of 3.2 dB can be achieved at a SEP of 10^-2 compared to 16-QAM by choosing an optimized APSK constellation. We also demonstrate that in the presence of severe nonlinear distortions, it may become beneficial to sacrifice a constellation point or an entire constellation ring to reduce the average SEP. Finally, we discuss the problem of selecting a good binary labeling for the found constellations.