Title :
The sqlLoader Data-Loading Pipeline
Author :
Szalay, Alex ; Thakar, Ani R. ; Gray, Jim
Author_Institution :
Johns Hopkins Univ., Baltimore
Abstract :
Using a database management system (DBMS) is essential to ensure the data integrity and reliability of large, multidimensional data sets. However, loading multiterabyte data into a DBMS is a time-consuming and error-prone task that the authors have tried to automate by developing the sqlLoader pipeline-a distributed workflow system for data loading.
Keywords :
SQL; data integrity; pipeline processing; relational databases; data integrity; database management system; distributed workflow system; error-prone task; multidimensional data set; sqlLoader data-loading pipeline; Data mining; Database systems; Image converters; Image databases; Information retrieval; Multidimensional systems; Observatories; Pipelines; Publishing; Warehousing; SDSS; Sloan Digital Sky Survey Science Archive; astronomy; database management systems; large-scale databases;
Journal_Title :
Computing in Science & Engineering
DOI :
10.1109/MCSE.2008.18
