Teleoperation system with virtual reality based on stereo vision

In this paper, a virtual reality (VR) assisted teleoperation system is investigated. The system consists of a servo-controlled construction robot, two joysticks for controlling the robot, and a 3-dimensional virtual environment. The operator performs remote operation of the construction robot by manipulating the graphic robot directly in virtual scene using the joysticks. In order to ensure the consistency between virtual environment and actual world, a stereo vision based image procession, recognition, and real time 3D reconstruction method is applied. The virtual scene not only involves a graphic robot, task objects, but also include a 3D shape for background working field. Finally, a teleoperation task is carried out comparing 3D virtual display method with 2D image feedback only based on the results of evaluation experiments (task efficiency and risk indexes). These experiments confirmed that the virtual environment reconstructed via stereo vision could be applied to guide operator to accomplish remote operation task effectively with better safety, and reduction of stress.