Determining an optimal value of K in K-means clustering

Partitioning data into a finite number of k homogenous and separate clusters (groups) without use of prior knowledge is carried out by some unsupervised partitioning algorithm like the k-means clustering algorithm. To evaluate these resultant clusters for finding optimal number of clusters, properties such as cluster density, size, shape and separability are typically examined by some cluster validation methods. Mainly the aim of clustering analysis is to find the overall compactness of the clustering solution, for example variance within cluster should be a minimum and separation between the clusters should be a maximum. In this study, for k-means clustering we have developed a new method to find an optimal value of k number of clusters, using the features and variables inherited from datasets. The new proposed method is based on comparison of movement of objects forward/back from k to k+1 and k+1 to k set of clusters to find the joint probability, which is different from the other methods and indexes that are based on the distance. The performance of this method is also compared with some existing methods through two simulated datasets.