A vision-based 2D-3D registration system

In this paper, we propose an automatic system for robust alignment of 2D optical images with 3D LiDAR (light detection and ranging) data. Focusing on applications such as data fusion and rapid updating for GIS (geographic information systems) from diverse sources when accurate georeference is not available, our goal is a vision-based approach to recover the 2D to 3D transformation including orientation, scale and location. Two major challenges of 2D-3D registration systems are different sensors problem and large 3D viewpoint changes. Due to the challenging nature of the problem, we achieve the registration goal through a two-stage solution. The first stage of the proposed system uses a robust region matching method to handle different sensor problems registering 3D data onto 2D images with similar viewing directions. Robustness to 3D viewpoint changes is achieved by the second stage where 2D views with sensor variations compensated by the first stage are trained. Other views are rapidly matched to them and therefore indirectly registered with 3D LiDAR data. Experimental results using four cities´ datasets have illustrated the potential of the proposed system.