Creating digital libraries: content generation and re-mastering

This article has two main goals: to describe the automatic creation of a digital library and to provide an overview of the metaalgorithmic patterns that can be applied to increase the accuracy of its creation. Automating the creation of useful digital libraries - that is, digital libraries affording searchable text and reusable ("repurposable") output - is a complicated process, whether the original library is paper-based or already available in electronic form. We outline the steps involved in the creation of a deployable digital library (>1.2 × 10⁶ pages) for MIT Press, as well as its implications to other aspects of digital library creation, management, use and repurposing. Input, transformation, information extraction, and output processes are considered in light of their utility in creating layers of content. Interestingly, in some aspects, scanning directly from paper offers extra opportunities for error-checking through feedback-feedforward combination. Strategies for quality assurance (QA) at the document, chapter and book level are also discussed. We emphasize the use of metaalgorithmic design patterns for application towards improving the content generation, extraction and remastering. This approach also increases the ease with which modules and algorithms are added to and deprecated from the system.