A spectral shaping nonlinear binary coded modulation with Gray-mapped QAM signals

A simple 4-state nonlinear trellis code with Gray-mapped QAM signals is designed which restricts certain symbol transitions in the constellation in order to shape the spectrum. The resulting spectral shaping subset selection code employs sign bit (most significant bit) coding, and is realized as a nonlinear binary convolutional code followed by an M-QAM modulator. The closed form of the power spectral density for the coded modulation is derived and different variations on the coding scheme are compared. The average signal power is analyzed and shown to be unchanged by the coding.