Hyperspectrum image fusion for sensor guided mobile manipulations

The hyperspectrum images from across the spectrum collect object characteristics and are capable of performing target recognition in the environments and target localization autonomously. In this paper, hyperspectrum image fusion for mobile manipulation is proposed. The fusion sensor including a single camera and an IR projector runs in two spectral bands (visible and IR) alternatively. After color images of the environments are captured from visible light, the region of interest (ROI) of the target will be recognized. Then in the ROI the target will be localized with the one-shot images of the pattern from IR light. A stereo fusion vision model of the fusion sensor and its calibration algorithm with a high precision are also presented. The localization is performed by 3D reconstruction based on the calibrated system. The calibration precision of the fusion sensor is [0.40, 0.24, 1.39]mm in x, y, z direction respectively. A manipulation experiment of grasping a handcream tub is carried out on the mobile platform and the experiment results verify the effectiveness of the hyperspectrum image fusion.