Evaluating I/O characteristics and methods for storing structured scientific data

Author

Ching, Avery ; Choudhary, Alok ; Liao, Wei-keng ; Ward, Lee ; Pundit, Neil

Author_Institution

Dept. of EECS, Northwestern Univ., Evanston, IL

fYear

2006

fDate

25-29 April 2006

Abstract

Many large-scale scientific simulations generate large, structured multi-dimensional datasets. Data is stored at various intervals on high performance I/O storage systems for checkpointing, post-processing, and visualization. Data storage is very I/O intensive and can dominate the overall running time of an application, depending on the characteristics of the I/O access pattern. Our NCIO benchmark determines how I/O characteristics greatly affect performance (up to 2 orders of magnitude) and provides scientific application developers with guidelines for improvement. In this paper, we examine the impact of various I/O parameters and methods when using the MPI-IO interface to store structured scientific data in an optimized parallel file system

Keywords

file organisation; input-output programs; parallel processing; I/O access pattern; checkpointing; high performance I/O storage systems; large-scale scientific simulations; optimized parallel file system; structured scientific data storage; Adaptive mesh refinement; Character generation; Computational modeling; Concurrent computing; Data visualization; File systems; Guidelines; Laboratories; Large-scale systems; Testing;

fLanguage

English

Publisher

ieee

Conference_Titel

Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. 20th International

Conference_Location

Rhodes Island

Print_ISBN

1-4244-0054-6

Type

conf

DOI

10.1109/IPDPS.2006.1639306

Filename

1639306