Hierarchical Storage Systems and File Formats for Web Archiving

Many national libraries are making efforts to crawl and store various born-digital information, there are many difficult problems of the social, legal and technical aspects. In this paper, from the view points of long-term preservation of digital contents, we focus on the the urgent task of storage system, since the size of the web archive is increasing exponentially. In order to archive monotonously increasing contents, we discuss management of storage devices and file formats in web archive systems. Firstly, we propose an architecture of hierarchical storage system based on characteristics of storage devices and file compression formats. Next, we modify the file moving algorithm by using file access frequency. We also evaluate the performance of our proposed algorithm with predicted data based on actual statistics of a web archive system.