A Performance Evaluation Framework for Library Search Engines

Libraries offer valuable resources to library patrons. Unfortunately, formulating library queries that match the rigid keywords chosen by the Library of Congress in library records to retrieve relevant results can be difficult. In solving this problem, we have developed a library search engine, called EnLibS, which allows library patrons to post a query Q with commonly-used words and ranks the retrieved library records according to their degrees of resemblance with Q. To evaluate the performance of EnLibS, it is imperative to conduct a thorough assessment. However, this performance evaluation cannot be conducted due to the lack of benchmark datasets and standardized metrics. To address this issue, in this paper we introduce an evaluation framework which (i) statistically determines the size of a test dataset, (ii) includes a controlled experiment that employs technically-sound approaches for calculating the ideal number of appraisers and queries to be used in the experiment, and (iii) establishes standard metrics for evaluating a library search engine. The proposed evaluation model can be applied to assess the performance of library search engines in (i) reducing the number of keyword queries that retrieve no results, (ii) obtaining high precision in retrieving and accurately ranking relevant library records, and (iii) achieving an acceptable query processing time. We present a case study in which we apply the proposed evaluation framework on the library search engine at Brigham Young University and EnLibS to assess, compare, and contrast their performance.