A high performance implementation of an OAI-based federation service

As the number of OAI-PMH compliant digital libraries are growing, we are encountering significant performance bottlenecks. Our earlier experience of implementing the searching and indexing for Arc (X. Liu et al., 2001) on a single processor utilizing the standard database technology like Oracle does not scale. In this paper, we describe how the cluster of PCs can be used to improve indexing and searching performance for an OAI based federation service. For supporting parallel indexing and searching, we use the Lucene search engine. We evaluated our architecture on a total number of 10 million harvested records on a Sun cluster. We compare the performance of the new implementation with the existing single processor implementation that uses Oracle 9i as the underlying database for searching and indexing. We demonstrate significant performance gain over the single processor implementation.