Neural network based modeling and color rendering for mixed reality

This paper describes a new approach to color image rendering for mixed reality applications. The approach has two steps. The first uses neural network based photometric stereo to obtain both a 3D shape model and color reflectance factors simultaneously. The second uses neural network based image rendering to generate realistic virtual images for arbitrary viewpoint and direction of light source. The proposed approach achieves high quality 3D modeling of real objects and subsequent virtual image rendering based on empirical measurements obtained in a simple experimental environment. The approach is empirical using only a standard digital camera and light sources. Assumptions about surface reflectance are reasonably generic. Experiments using real data are demonstrated.