Unbalanced sample size effect on the genome-wide population differentiation studies

The fixation index (Fst) is one of the most widely used measurements of genetic distance between populations. The data set from the international HapMap project has been served as a reference data set for population differentiation studies. Fst is commonly used in order to compare the sample data with HapMap data. In this study, however, we show that Fst without consideration of sample sizes may mislead the result. In particular, we first demonstrate that Fst suffers from imbalance of sample sizes through simulation studies and through the analysis of a large scale Korean genome-wide association (GWA) data. Then, we propose a modified version of Fst which is shown to be more robust to imbalance of sample size. In addition, the chi-square test commonly used for homogeneity test is shown to perform similarly to the modified version of Fst.