An adaptive soft-decision quantizer for digital communications with convolutional coding on a Rayleigh fading channel

An adaptive soft-decision quantizer with quantization thresholds controlled by a supervised learning neural network co-processor which enhances the performance of a soft-decision Viterbi decoder used for forward error-correction in a digital communication channel subject to Rayleigh fading and additive white Gaussian noise (AWGN) has been investigated and designed. The neural network is designed to cooperate with a differentially phase shift keying (DPSK) demodulator, and a soft-decision quantizer which is an analog-to-digital convertor (ADC). The quantization thresholds in the quantizer are adaptively adjusted by the neural network according to the statistical characteristics of the analog symbol outputs of the DPSK demodulator. The channel cutoff rate R₀, created by the demodulation system (DPSK demodulator and quantizer), is employed to determine the best quantization threshold step-size Δ_BEST that results in the minimization of the Viterbi decoder output bit error rate (BER). The neural network is trained to learn Δ_BEST as a function of the mean and standard deviation of the analog symbol outputs of the DPSK demodulator. Consistent and substantial performance improvements, 3% to 30%, by computer simulations, have been demonstrated for various channel conditions. This neural network co-processor approach is easily generalized and applied to any digital signal processing system, and thus decrease the performance loss caused by quantization and/or signal instability.