A cell-MST-based method for big dataset clustering on limited memory computers

This paper presents a new clustering algorithm, called Cell-MST-Based Method that is a combination of a Cell-based method and Minimum Spanning Tree based (MST-based) methods. The algorithm is dedicated for Big Datasets on a limited memory computer, especially for thin big datasets which have a small number of attributes but a very large number of instances. Firstly, a Cell-based method converts a big dataset to a small grid of cells in such a way that the required memory to store an edge-weighted graph created from the grid which is less than the available memory of a computer. Then MST-based methods obtain an optimal threshold, estimate the number of clusters and determine the initial centroids. The proposed Cell-MST-based methods can reduce more than 99% of the required memory of the previous similarity-based and MST-based cluster number estimation methods. Moreover, this new Cell-MST-based method also outperforms the quantization error modeling method in terms of executing time and estimated accurate level.