A comparison of handsearching versus EMBASE searching of the Annals of Saudi Medicine to identify reports of randomized controlled trials

Healthcare decision-making needs to be informed by high quality and timely research evidence. The randomized controlled trial has long been considered the ‘gold standard’ in the hierarchy of evidence. Trials involving sufficient numbers of participants are essential to distinguish reliably between the effects of healthcare interventions and the effects of bias or chance. The synthesis of the results of these trials in systematic reviews can provide reliable evidence about the effects of these interventions. The Cochrane Collaboration, an international organization dedicated to improving healthcare through the preparation of systematic reviews, has focused on the systematic electronic searching of Medline and EMBASE and the systematic handsearching of over 2000 general and specialized healthcare journals for reports of randomized controlled trials. This involves reading each document in a journal to decide if a report is a randomized trial, according to Cochrane eligibility criteria.1 The efforts of the many volunteers working within the Cochrane Collaboration have added a substantial number of previously ‘buried’ reports of randomized controlled trials to the Cochrane Central Register of Controlled Trials (CENTRAL) published in The Cochrane Library. Some of these reports of trials may have been ‘buried’ as a result of inconsistencies in indexing (indexing bias), a lack of cover-to-cover indexing, or because they have been published in journals not indexed in the major healthcare databases such as Medline and EMBASE (database bias) or in journals published in languages other than English (language bias). A good example of language bias was provided in a study of 68 Spanish general medicine journals, which found that only six were indexed in Medline.2 The Bahrain Branch of the UK Cochrane Centre is contributing to efforts to minimize bias through a comprehensive handsearching of journals published in the Arab world. Methods All issues of the Annals of Saudi Medicine (July 1981 to April 2005) were searched by hand from cover to cover for reports of trials involving prospective assignment to one of two or more interventions. These were classified as randomized controlled trials (RCTs) or controlled clinical trials (CCTs) according to Cochrane eligibility criteria.1 The handsearcher classified reports of trials as RCTs if the author reported that the groups compared in the trial were established by random allocation. If the author(s) did not state explicitly that the trial was randomized but randomization could not be ruled out, the report was classified as a CCT. CCT was also applied to quasi- or possibly quasi-randomized studies where the method of allocation was known but not considered strictly random (e.g. alternation, date of birth, medical record number). Photocopies of the bibliographic details and of the pages describing the study design were sent to the UK Cochrane Centre for verification and processing for submission to the US Cochrane Center and publication in CENTRAL. EMBASE (via Ovid Web) and Medline were also searched to determine if reports found by the handsearch were already included in either of these databases. The Annals has been indexed in EMBASE since January 1990, but in Medline only since the beginning of 2004. We then determined the added value of the handsearch in minimizing the effects of indexing bias by assessing the sensitivity (proportion of total number of known randomized trials identified by the search)3 of (i) the EMBASE index term RANDOMIZED CONTROLLED TRIAL and (ii) the Medline index term RANDOMIZED CONTROLLED TRIAL. The “gold standard” for calcuation of sensitivity was the handsearch. Added value was defined as the total number of reports of trials published in these journals and not indexed as randomized controlled trials in EMBASE or appearing before indexing, and therefore not easily identified except through a handsearch of these journals. Results We found 26 RCTs and 3 CCTs by handsearching. The distribution by country of principal investigator was highest for Saudi Arabia (18), followed by Turkey (2) and Iran (2). Of the 26 reports of RCTs found by the handsearch, 21 were in EMBASE but only 8 of these had been given the index term RANDOMIZED CONTROLLED TRIAL (Figure 1). Six of the trials (all randomized controlled trials) identified in the handsearch were published before January 1990, so the sensitivity of the EMBASE search was 40% (8/20). The overall added value of the handsearch in relation to EMBASE was 21 additional articles found, so that 72% of controlled trials in the journal were discovered by handsearching. Only 3 of the 29 reports found by the handsearch were in Medline, but indexing started in January of 2004. The added value of the handsearch in relation to Medline was 26 additional articles found (90% of controlled trials identified by handsearching).