A coupled schema of probabilistic atlas and statistical shape and appearance model for 3D prostate segmentation in MR images

A hybrid framework of probabilistic atlas and statistical shape and appearance model (SSAM) is proposed to achieve 3D prostate segmentation. An initial 3D segmentation of the prostate is obtained by registering the probabilistic atlas to the test dataset with deformable Demons registration. The initial results obtained are used to initialize multiple SSAMs corresponding to the apex, central and base regions of the prostate gland to incorporate local variabilities. Multiple mean parametric models of shape and appearance are derived from principal component analysis of prior shape and intensity information of the prostate from the training data. The parameters are then modified with the prior knowledge of the optimization space to achieve 2D segmentation. The 2D labels are registered to the 3D labels generated using probabilistic atlas to constrain the pose variation and generate valid 3D shapes. The proposed method achieves a mean Dice similarity coefficient value of 0.89±0.11 and mean Hausdorff distance of 3.05±2.25 mm when validated with 15 prostate volumes of a public dataset in a leave-one-out validation framework.