Understanding of mechanical assembly instruction manual by integrating vision and language processing and simulation

The paper reports a system which builds assemblies by simulating in virtual space the assembly operations specified in an assembly manual. A manual consists of both illustration and explanation. Illustration consists of not only figures but also phrases and symbols which relate the phrases to the figures. This research uses the 3D models of the mechanical parts generated by a modeling tool. A noun phrase in the illustration specifies the name of a mechanical part and determines the model used. A verb phrase imposes an initial condition to be held before the execution of an assembly operation. Auxiliary lines in an illustration are virtual lines, which are used to mate the parts at both ends. Both the assembly process and terminal condition of the operation are elucidated in explanation. The state of a subassembly obtained is hard to infer due to geometric variations of the parts involved in the operation. The configuration obtained with simulation is propagated to inference mechanism and this allows the system to recognize the state of the machine assembled