Analytical evaluation of transmission penalty due to group velocity dispersion, self-phase modulation, and amplifier noise in optical heterodyne CPFSK systems

An analytical bit error rate evaluation of an optical heterodyne continuous-phase frequency shift keying (CPFSK) transmission system affected by group-velocity dispersion (GVD), self-phase modulation (SPM), and erbium-doped fiber amplifier (EDFA) amplifier spontaneous emission noise (ASE) in a nonlinear fiber medium is carried out following a perturbative and a transfer matrix approach. The utility of both approaches has been exemplified by applying them to different dispersion compensation schemes in optical and electronic domains. The theoretical penalty estimates are found to have good agreement with both the reported experimental results and those obtained by the Q-value simulation.