Color and texture based segmentation of molecular pathology images usING HSOMS

Prostate cancer is the most common cancer among men, excluding skin cancer. It is diagnosed by histopathology interpretation of Hematoxylin and Eosin (H&E)-stained tissue sections. Gland and nuclei distributions vary with the disease grade, and the morphological features vary with the advance of cancer. A tissue microarray with known disease stages can be used to enable efficient pathology slide image analysis. We focus on an intuitive approach for segmenting such images, using the Hierarchical Self-Organizing Map (HSOM). Our approach introduces the use of unsupervised clustering using both color and texture features, and the use of unsupervised color merging outside of the HSOM framework. The HSOM was applied to segment 109 tissues composed of four tissue clusters: glands, epithelia, stroma, and nuclei. These segmentations were compared with the results of an EM Gaussian clustering algorithm. The proposed method confirms that the self-learning ability and adaptability of the HSOM, coupled with the information fusion mechanism of the hierarchical network, leads to superior segmentation results for tissue images.