Applying Rules as Policies for Large-Scale Data Sharing

Large scientific projects need collaborative data sharing environments. For projects like the Ocean Observations Initiative (OOI), the Temporal Dynamics of Learning Center (TDLC) and Large-scale Synoptic Survey Telescope (LSST) the amount of data collected will be on the order of Petabytes, stored across distributed heterogeneous resources under multiple administrative organizations. Policy-oriented data management is essential in such collaborations. The integrated Rule-Oriented Data System (iRODS) is a peer-to-peer, federated server-client architecture that uses a distributed rule engine for data management to apply policies encoded as rules. The rules are triggered on data management events (ingestion, access, modifications, annotations, format conversion, etc) as well as periodically (to check integrity of the data collections, intelligent data archiving and placement, load balancing, etc). Rules are applied by system administrators (e.g. for resource creation, user management, etc.) and by individual users, groups and data providers to tailor the sharing and access of data for their own needs. In this paper, we will discuss the architecture of the iRODS middleware system and discuss some of the applications of the software.