Decision-feedback equalization via separating hyperplanes

The design of finite-length decision-feedback equalization (DFE) forward and feedback filters under the assumption of genie-aided feedback and independent and equally likely transmitted symbols is considered. It is shown that the problem of determining DFE filters that minimize the probability of symbol error at high signal-to-noise ratio (SNR) is equivalent to finding the hyperplane that maximally separates two given finite groups of points in a finite-dimensional Euclidean space. The latter task can be formulated as a quadratic program which is readily solved numerically. It is also shown that the problem of finding finite-length DFE filters that minimize the probability of symbol error at any SNR subject to a certain separation condition is a convex optimization problem. The case where the transmitted data is coded using a runlength-limited code is also investigated. Examples show that this criterion yields a performance that is better than zero-forcing DFE on severely distorted channels at high SNR