L1 vs. L2 Regularization in Text Classification when Learning from Labeled Features

In this paper we study the problem of building document classifiers using labeled features and unlabeled documents, where not all the features are helpful for the process of learning. This is an important setting, since building classifiers using labeled words has been recently shown to require considerably less human labeling effort than building classifiers using labeled documents. We propose the use of Generalized Expectation (GE) criteria combined with a L₁ regularization term for learning from labeled features. This lets the feature labels guide model expectation constraints, while approaching feature selection from a regularization perspective. We show that GE criteria combined with L₁ regularization consistently outperforms - up to 12% increase in accuracy - the best previously reported results in the literature under the same setting, obtained using L₂ regularization. Furthermore, the results obtained with GE criteria and L₁ regularizer are competitive to those obtained in the traditional instance-labeling setting, with the same labeling cost.