XChange: coupling parallel applications in a dynamic environment

Modern computational science applications are becoming increasingly multidisciplinary, involving widely distributed research teams and their underlying computational platforms. A common problem for the grid applications used in these environments is the necessity to couple multiple, parallel subsystems, with examples ranging from data exchanges between cooperating, linked parallel programs, to concurrent data streaming to distributed storage engines. This work presents the XChange_mxn middleware infrastructure for coupling componentized distributed applications. XChange_mxn implements the basic functionality of well-known services like the CCA Forum´s MxN project, by providing efficient data redistribution across parallel application components. Beyond such basic functionality, however, XChange_mxn also addresses two of the problems faced by wide area scientific collaborations, which are (1) the need to deal with dynamic application/component behaviors, such as dynamic arrivals and departures due to the availability of additional resources, and (2) the need to ´match´ data formats across disparate application components and research teams. In response to these needs, XChange_mxn uses an anonymous publish/subscribe model for linking interacting components, and the data being exchanged is dynamically specialized and transformed to match end point requirements. The pub_{paradigm makes it easy to deal with dynamic component arrivals and departures. Dynamic data transformation enables the ´inflight´ correction of data or needs mismatches for cooperating components. This work describes the design and implementation of XChange}sub mxn/, and it evaluates its implementation compared to those of less flexible transports like MPI. It also highlights the utility ofXChange_mxn´s ´inflight´ data specialization, by applying it to the SmartPointer parallel data visualization environment developed at our institution. Interestingly, using XChange_mxn did not significantly affect performance but led to a reduction in the size of the code base.