Introduction: A study of the literature is by definition non-experimental and therefore open to the same forms of bias as other observational studies. In order to prevent review bias some authors require blinded assessment of the quality of the articles. Systematic reviews performed in our department are blinded for authors, institutes where the study had been performed, the journal in which the paper was published and all information about the results of the intervention. This form of blinding is very time consuming. The aim of this study is to assess if our method of blinded reviewing is essential in preventing review bias.

Objective: To assess if blinded reviewing differs from non-blinded reviewing.

Methods: We have used articles from a systematic review about the efficacy of balneotherapy in patients with arthritis. We excluded cross-over designs. In total 18 reviewers participated. They were university staff members and graduated students with epidemiological backgrounds. All reviewers followed training to ensure that the reviewers used the forms and procedures in identical ways. The computer designed a random table to guarantee that each article was scored 10 times (5 times blinded and 5 times non-blinded).

Results: A set of 12 experimental studies were used. All 18 reviewers completed the criteria lists and each of them scored on average 8 articles. The quality scores between the blinded and open assessment of the articles do not differ much. There is hardly any difference in ranking between the blinded and non-blinded assessment.

Discussion: Although we reviewed only a small number of trials using reviewers with a high level of epidemiological knowledge, we found no evidence for review bias in quality assessment of clinical trials.