Coordinates tracking and augmented reality system using bipolar X-ray fluoroscopy and stereo vision for image-guided neurosurgery

In this paper multiple optic sensor system and algorithms to match coordinate systems for neurosurgery robot is proposed for O-arm or C-arm fluoroscopy. As a specific device mounted on fluoroscopy we make the optical axis of X-ray source and the optical sensor identical. This makes it capable of achieving simultaneously X-ray vision and camera vision of the same area without any other complex mapping between two images. This can be used to make augmented vision to show X-ray vision to surgeons under operation. Also because of the characteristic of neurosurgery the required measurement area is relatively small but neurosurgery is very demanding in terms of accuracy, so the measurement should be within small numbers of micrometers. For accurate measurement specially designed stereo vision is implemented with IR illumination and retro-reflective marker spheres. With a pair of stereo images from the sensor system the three-dimensional coordinates and pose of the markers are calculated. By attaching these markers to patient, surgical instruments, and robot each set of the coordinate systems can be tracked with the proposed sensor system.