Document vector compression and its application in document clustering

Document clustering organizes documents into groups such that each group contains documents with similar content. The majority of document clustering algorithms require a vector representation for each document. Each vector has well over 10,000 elements. Consequently, the memory required during clustering can be extremely high when clustering hundreds of thousands of documents. This paper introduces document vector compression, which is based on the discrete cosine transform (DCT). Document vector compression reduces the run-time memory requirements by as much as 60%. Document vector compression does not degrade the final cluster quality (total F-measure) as does other document vector reduction techniques