Title :
An Erasure Coded Archival Storage System
Author :
Misra, Prasant ; Roy, Nicholas ; Naskar, Sourav ; Dey, Shuvashis
Author_Institution :
TCS Innovation Lab., Tata Consultancy Services Ltd., Kolkata, India
Abstract :
There is an ever increasing need of storage capacity for storage of digital archives and historical data-digital preservation, because of regulatory and compliance requirements. There is an increasing interest in disk based archival system. Major technical challenges in creating large disk based storage archive are - providing large capacity at low costs, large read and write throughput, data integrity and sustaining hardware and operating system refresh. In this paper we present the architecture and working principle of an archival storage system that uses an erasure-coded redundancy scheme. We present the design of a Quality of Service (QoS) framework that tries to achieve an optimum balance between file availability, performance and system availability. The design includes a file encoding and placement scheme that allows files to be read from the archive without the need to access any metadata. Finally, we present the results obtained from running an experimental setup on Amazon Web Services.
Keywords :
disc storage; information retrieval systems; quality of service; records management; QoS; compliance requirements; data integrity; digital archives; disk based archival system; erasure coded archival storage system; file encoding; hardware system; historical data-digital preservation; metadata; operating system; quality of service; storage capacity; Availability; Bandwidth; Encoding; Operating systems; Quality of service; Servers; Throughput; QoS; archival storage; erasure coding; long term storage; regeneration;
Conference_Titel :
Parallel and Distributed Systems (ICPADS), 2012 IEEE 18th International Conference on
Conference_Location :
Singapore
Print_ISBN :
978-1-4673-4565-1
Electronic_ISBN :
1521-9097
DOI :
10.1109/ICPADS.2012.112