Cognitive tracking of surgical instruments based on stereo vision and depth sensing

It is significant to have accurate localization of surgical instruments in navigated minimally invasive surgery. Moreover, instrument tracking modules are essential for cognitive surgical robotic systems. Commercial optical trackers have been developed with sub-millimeter accuracy, but typically work only at single spectrum - either visible spectrum or infrared spectrum, which limits the sensing and perception in surgical environment. The objective of this research is to incorporate multiple sensors at broad-spectrum, including stereo infrared (IR) cameras, color (or RGB) cameras and depth sensors to perceive the surgical environment. Features extracted from each modality can contribute to the cognition of complex surgical environment or procedures. Additionally, their combination can provide higher robustness and accuracy beyond what is obtained from single sensing modality. As a preliminary study, we propose a multi-sensor fusion approach for localizing surgical instruments. We developed an integrated dual Kinect tracking system to validate the proposed hierarchical tracking approach.