A hypothesize-and-bound algorithm for simultaneous object classification, pose estimation and 3D reconstruction from a single 2D image

We consider the problems of 3D reconstruction, pose estimation and object classification, from a single 2D image. In sharp contrast with state of the art methods that solve each of these problems separately or iteratively, we propose a mathematical framework that solves these problems jointly and simultaneously. Since the joint problem is ill posed unless “prior knowledge” is considered, the proposed framework incorporates “prior knowledge” about the 3D shapes of different object classes. This knowledge is used to define a function L(H) that encodes how well each hypothesis H (object class and pose) “explains” the input image. To efficiently maximize L(H) without having to exactly evaluate it for each hypothesis H, we propose a H&B algorithm that computes and refines upper and lower bounds for L(H) at a much lower cost. In this way suboptimal hypotheses are disregarded with little computation. The resulting algorithm integrates information from the 2D image and the 3D prior, is efficient, and is guaranteed to find the optimal solution.