The data analysis computing hierarchy

With the dramatic increases in simulation complexity and resolution comes an equally dramatic challenge for resources, both computational and storage, needed to facilitate analysis and understanding of the results. Traditionally these needs have been met by powerful workstations equipped with sophisticated analysis tools and special purpose visualization hardware. More and more these personal computing resources are unable to manage the so-called data deluge without significant support from additional high-end resources. The response to this crisis is taking shape in the form of an additional layer in the analysis pipeline: the visual and data analysis cluster. This High Performance Computing resource stands somewhere between the source of the data deluge (a supercomputer simulation or a large data collection experiment) and the user´s personal computer. In this paper we discuss (1) the scale and character of a few current large-data enterprises, (2) a descriptive hierarchical model of the computation and analysis pipeline, and (3) some of the capabilities that will need to be developed in order for such an architecture to meet the challenges of efficient analysis in the face of huge datasets, I/O bottlenecks, and remote users.