Article type
Year
Abstract
Background:
In 2005, John Ioannides shook the scientific world with his landmark publication “Why most research findings are false”. This finding seems to be true for all types of research, including clinical research. Mistakes in methodology or statistical approach can lead to false conclusions. When using these false claims as evidence for medical decisions, these mistakes may have major impact on clinical care.
Objectives:
The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews. We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through OVID. All reviews reported a summary estimate of sensitivity and specificity.
Methods:
We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through OVID. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. We evaluated the following items of included studies: (1) the availability of a 2x2 table, (2) availability of full text papers, (3) correctness of information included in the primary papers and (4) the correctness of the pooled estimate. Full replication was met if all four items were reported and the point estimate differed < 1% point from the reported point estimates. We studied reproduction of MA’s in which the 2x2 tables were available. If the information for the 2x2 tables was not available we returned to the primary papers to search for information to comprise the 2x2 tables.
Results:
Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2x2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced.
In 32 (63%) of MA’s the information to comprise 2x2 tables was not available. In 17 (33%) of those the primary papers were available. Of those 0 analyses could be replicated.
Considering the correctness of the numbers from the primary papers and the complete reporting of the search strategy, only one meta-analysis was fully replicable.
Conclusions:
Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used; and partly because of mistakes in the data extraction or data reporting.
Patient or healthcare consumer involvement:
Because of the nature of this study no patients were involved
In 2005, John Ioannides shook the scientific world with his landmark publication “Why most research findings are false”. This finding seems to be true for all types of research, including clinical research. Mistakes in methodology or statistical approach can lead to false conclusions. When using these false claims as evidence for medical decisions, these mistakes may have major impact on clinical care.
Objectives:
The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews. We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through OVID. All reviews reported a summary estimate of sensitivity and specificity.
Methods:
We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through OVID. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. We evaluated the following items of included studies: (1) the availability of a 2x2 table, (2) availability of full text papers, (3) correctness of information included in the primary papers and (4) the correctness of the pooled estimate. Full replication was met if all four items were reported and the point estimate differed < 1% point from the reported point estimates. We studied reproduction of MA’s in which the 2x2 tables were available. If the information for the 2x2 tables was not available we returned to the primary papers to search for information to comprise the 2x2 tables.
Results:
Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2x2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced.
In 32 (63%) of MA’s the information to comprise 2x2 tables was not available. In 17 (33%) of those the primary papers were available. Of those 0 analyses could be replicated.
Considering the correctness of the numbers from the primary papers and the complete reporting of the search strategy, only one meta-analysis was fully replicable.
Conclusions:
Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used; and partly because of mistakes in the data extraction or data reporting.
Patient or healthcare consumer involvement:
Because of the nature of this study no patients were involved