Using synthetic data and dimensionality reduction in high-dimensional classification via logistic regression

Traditional logistic regression is plugged with degenerates and violent behavior in high-dimensional classification, because of the problem of non-invertible matrices in estimating model parameters. In this paper, to overcome the high-dimensionality of data, we introduce two new algorithms. First, we improve the efficiency of finite population Bayesian bootstrapping logistic regression classifier by using the rule of majority vote. Second, using simple random sampling without replacement to select a smaller number of covariates rather than the sample size and applying traditional logistic regression, we introduce the other new algorithm for high-dimensional binary classification. We compare the proposed algorithms with the regularized logistic regression models and two other classification algorithms, i.e., naive Bayes and K-nearest neighbors using both simulated and real data.