Objective: During systematic review elaboration information contained in clinical trial reports must be interpreted by the reviewer. Subjective factors and interpretation are forcibly involved in the data extraction. This could be the source of differences of interpretation and data extraction by different reviewers. During the Validata project an attempt was made to evaluate this inter-reviewer variation.

Methods: At the end of the Validata project, 12 clinical trial reports were distributed to two experts. One of the products of the Validata project was the definition of the information concerning a clinical trial that is necessary to characterise the trial, its interest, its validity and its results. The experts were asked to extract this information from the twelve articles and to complete trial summary forms. The content of these forms was then entered into a database and compared.

Results: The results were disappointing worrying and somewhat surprising: while there was no discordance in the identification of the primary outcomes of the trials, trial population sizes were often found to be of different sizes by the readers, the same selection criteria were not identified, secondary outcomes, trial design and starting and ending dates were all regularly discordant.

Discussion: Inter-reviewer variation exists and it is important to evaluate the size of the effect it might have on existing and future systematic reviews. Methods to reduce the contribution of subjective interpretation in the extraction of information for reviews need to be investigated and potentially applied in the methodology of systematic reviewing. These results are worrying as to the reproducibility of meta-analyses. It would be feasible to set up a controlled trial to evaluate the effect of different reviewers on the findings of meta-analyses. Potential modifications reviewing methodology might include systematic double data extraction as well the provision of factual standard trial summaries by the trial designers or promoters. The existence of inter-reviewer variation is one of the foundations on our insistence that high quality systematic reviews should be based on standard summaries of the included trials and is one of the reasons for the development of the information system TriSum. Initiatives for the standard reporting of trial findings such as the Consort statement should help to reduce inter-reviewer variation. This could also be evaluated in a controlled trial.