Looking for trouble: Using causal semantics to direct focus of attention

Vision should provide an explanation of the scene in terms of a causal semantics. The authors propose a characterization of what constitutes visual understanding. The cornerstone of the proposal is that visual understanding is fundamentally a matter of developing a causal explanation of the scene, i.e., of determining the causal significance of the elements in a scene, and the causal relationships among those elements. Simple, naive physical knowledge is used as the basis of a vertically integrated vision system that explains arbitrarily complex stacked block structures. The semantics provides a basis for controlling the application of visual attention, and forms a framework for the explanation that is generated. It is shown that the program sequentially explores scenes of complex blocks structures, identifies functional substructures such as arches and cantilevers, and develops an explanation of why the whole construction stands and the role of each block in its stability