Background:

Within the scope of a series of systematic reviews of test accuracy studies for predicting pre-eclampsia, we compiled a database of 15,375 possibly relevant references. Four reviewers had to select references for ordering as full papers. Time constraints precluded us from performing the preliminary selection, based on titles and abstracts, using pairs of reviewers working independently. Therefore, we decided to use the latter gold standard method in a sample of references only. In that way we tried to estimate the proportion of times that discordant decisions on whether a full text paper should be ordered, leads to exclusion of useful papers from the review.

Objectives:

To investigate the risk of excluding relevant papers from a series of systematic reviews of diagnostic accuracy if single reading of titles and abstracts were to replace double reading.

Methods:

Preliminary selection, based on titles and abstracts, was performed independently by three pairs of reviewers on 1,004 references (Table 1). Final inclusion based on full text paper was performed independently by these pairs on 247 selected references. Selection and inclusion criteria were predefined and pilot tested. By admixture of concordant references, reviewers did not know which references were discordant.

(Table 1)

Results:

Disagreement between reviewers existed on 139 selected references. Interobserver agreement was 0,53 (SE 0.04, CI 95% 0.45 0.60). The proportion of papers at risk of being inadvertently excluded will be reported at the Colloquium.

Conclusions:

The absence of relevant studies among discordant references would justify single reading in preliminary selection. The generalizability of these results will be discussed. The quantitative impact on the conclusion of the review will be explored as an add-on in the upcoming reviews.