An upper bound on BER in a coded two-transmission scheme with same-size arbitrary 2D constellations

Constellation design is a well studied topic. For instance, it is known that, when 2-dimensional (2D) signalling schemes are used, the performance gains that the optimal constellations yield in comparison to the commonly employed square constellations are rather small. However, most of the earlier literature in this area considers rather simple communication protocols (for instance, without retransmissions), even in the absence of channel coding. With the advent of advanced communication protocols as well as signal processing techniques, there has been a rejuvenated interest in constellation design in recent years. In this paper, we consider a generic two-transmission scheme which may correspond to a relay, HARQ (hybrid automatic repeat request), or CoMP (coordinated multipoint) based transmission scenario, with a maximum likelihood receiver. The system has the flexibility of using a different 2D constellation in each transmission (however, the constellation size, i.e., the number of bits per symbol, remains the same). A generic channel coding scheme for which an encoder transfer function can be written (such as, convolutional and turbo codes) is considered for the versatile Nakagami-m fading channel. The main contribution of this paper is the derivation of an upper bound on the bit error rate (BER) which is expressed as a function of the distances between the constellation points. Using proper optimization techniques, this bound (tight for the high SNR values) which captures the impact of the distances between the constellation points can enable the design of good constellations for a given coding scheme in the above explained two-transmission setting.