A new approximation to the symbol error probability for coded modulation schemes with maximum likelihood sequence detection

The power efficiency of coded modulation schemes in additive white Gaussian noise depends on the signal space distribution of their most common error events. Symbol error probability calculation allowing for the pairwise interaction of these error events is discussed. Two optimality criteria are considered for detectors. The first minimizes the probability of symbol error for each symbol decision. This is called the symbol-to-symbol detector. The second (which is superior) is the maximum likelihood sequence detector (MLSD). A lower bound for the symbol-to-symbol detector and an approximation to the MLSD symbol error probability are described. The theoretical performance difference between these two detectors is given. The results are more accurate than minimum squared Euclidean distance predictions, especially at low and intermediate signal-to-noise ratios. The MLSD symbol error probability approximation is obtained for considerably less cost than computer simulation and gives more insight into the signal space structure of the scheme being analyzed. Numerical results are presented for a continuous phase modulation (CPM) example