Background: An overview of reviews compiles evidence from multiple reviews into one accessible document, thus helping decision makers to choose effective healthcare interventions. Overviews are challenging. For instance, results from individual studies are not always adequately presented in reviews to allow optimal compilation of data.

Objectives: To describe the difficulties of preparing an overview of reviews. Methods and Results: We summarised evidence from systematic reviews on the effect of financial incentives on professional practice and patient outcomes, including both Cochrane (CR) and non-Cochrane reviews (NCR). From a large number of seemingly eligible reviews, only 2 CRs and 2 NCRs, presenting quantitative data and fulfilling the DARE criteria for an adequate search, were included. Excluded reviews typically did not present numerical data or failed to present data at the level of individual studies. Comparison of the quality of the evidence reported in included reviews was difficult because they used different assessment tools. We used the AMSTAR tool to assess the methodological quality of reviews, and found the quality to be high even though the presentation of numerical data was incomplete for many outcomes. We attempted to retrieve missing data by consulting individual studies or their authors. Still, we could not retrieve the information needed for meta-analysis. Instead, results were summarised using vote counting within financial incentives across type of outcome, and within outcomes across types of financial incentives.

Conclusions: Inadequately reported data in reviews is a major problem in the preparation of overviews over reviews. Differences in the tools used by reviewers to assess study limitations, is another. This makes it very difficult to summarise and evaluate large amounts of heterogeneous data, in order to present it in an accessible format. Researchers and reviewers alike must improve the reporting of data.