Error Rate Performance of Multilevel Signals with Coherent Detection

In this paper, coherent detection for multilevel correlated signaling sets in additive white Gaussian noise is addressed. From the decision rule, a general analytical expression for the symbol error probability (SEP) is derived, which is in the form of a single integral. Our new result is not only in agreement with a known equivalent expression, but it also has a much simpler analytical form. The structure of the correlation matrix under consideration of the signaling set is quite generic, including various known signaling sets as special cases. Hence the derived SEP expression is useful for analyzing the performance of various coherent modulation schemes such as multilevel phase-shift and frequency-shift keying. Specific correlation structures which minimize the SEP are also studied. Based on eigendecomposition or LU decomposition, generic methods for constructing a correlated signaling set for any correlation matrix under consideration are also provided.