Creating a software ecosystem for Data Intensive Science

“Data Intensive Science” is an acknowledgement that as simulations and experiments continue to generate larger amounts of data, we must turn our attention on how to move, store, manage, analyze and visualize this data in a timely fashion. The cost of “write once read never” is becoming too expensive and we must start to create software ecosystems to help us cope with this flood of data from scientific instruments and calculations. “Big Data” can be characterized by the five Vs of data intensive science: volume, velocity, variety, veracity, and value. Understanding how applications and experiments will change in the next ten years is critical in order to develop a software ecosystem that can handle the implications of “Big Data” in order to allow for knowledge discovery. In this talk, we will explore different software systems used by scientists today, and the extension of many of these software artifacts to bring them together into a complete software ecosystem that can be amenable to next-generation science. In particular we will focus on the ADIOS software system as a framework for the management of large data, and the use of operations on the data using the next-generation VTK-M visualization software system, and explore its usage in both the short and long term.