Estimating intrinsic dimension via clustering

Estimating the intrinsic dimension of a data set from pairwise distances is a critical issue for a wide range of disciplines, including genomics, finance, and networking. Current estimation techniques are agnostic to structure in the data, failing to exploit properties that can improve efficiency. In this paper, we present a methodology that uses inherent clustering present in data to efficiently and accurately estimate intrinsic dimension. Our experiments show that this approach has greater accuracy and better scalability than prior techniques, even when the data does not conform to an obvious clustering structure.