Prototype Filter Optimization to Minimize Stopband Energy With NPR Constraint for Filter Bank Multicarrier Modulation Systems

Recently, filter bank multicarrier (FBMC) modulations have attracted increasing attention. The filter banks of FBMC are derived from a prototype filter that determines the system performance, such as stopband attenuation, intersymbol interference (ISI) and interchannel interference (ICI). In this paper, we formulate a problem of direct optimization of the filter impulse-response coefficients for the FBMC systems to minimize the stopband energy and constrain the ISI/ICI. Unfortunately, this filter optimization problem is nonconvex and highly nonlinear. Nevertheless, observing that all the functions in the optimization problem are twice-differentiable, we propose using the $\alpha$ -based Branch and Bound ( $\alpha$ BB) algorithm to obtain the optimal solution. However, the convergence time of the algorithm is unacceptable because the number of unknowns (i.e., the filter coefficients) in the optimization problem is too large. The main contribution of this paper is that we propose a method to dramatically reduce the number of unknowns of the optimization problem through approximation of the constraints, so that the optimal solution of the approximated optimization problem can be obtained with acceptable computational complexity. Numerical results show that, the proposed approximation is reasonable, and the optimized filters obtained with the proposed method achieve significantly lower stopband energy than those with the frequency sampling and windowing based techniques.