Article type
Year
Abstract
Background: Systematic reviewers often have to extract data from less than ideal trial reports, especially when dealing with older studies. These may present their findings without absolute numbers, but as relative effect measures, percentages, or sometimes only in graphical figures. We wanted to test the precision and accuracy of purpose-built software now available to perform graphical data extraction, and to compare it with traditional manual 'by hand and eye' methods.
Methods: Using figures from trial reports produced within our own Department and published in the BMJ and the Lancet, two authors (KC and MS) independently extracted data manually 'by hand and eye', using rulers, protractors and sharp pencils. We then processed the same graphs within two commercially available software packages (DigitizeIt and Grab It!), and compared the results obtained. We then evaluated the accuracy of each method against the source data from which the original graphs were produced.
Results: The software packages demonstrated some benefits in accuracy and precision, although care in setting up the extraction process and authorial decisions about key parameters were necessary, and were inevitably susceptible to variation in personal judgements. Occasional errors were made in identifying the correct data points for extraction, but these were related to the intricacies of the graphical figures, and were equally likely to occur whether a digital or a manual extraction method was used. One of the manual data extractors and one of the digital packages achieved high and comparable precision and accuracy, compared with the source material. The mean difference (i.e. accuracy) between manual extraction and source for Extractor B was 0.160, with a precision of 7.5 estimated by the total size of the 95% reference range (-3.581 to +3.902). The mean difference between the source data and the Grab It! extraction was 0.497, with a precision of 4.5 (95% reference range -1.770 to +2.765). Bland-Altman plots were used to visualise the relative accuracy and precision of these extractions. A secondary outcome was the functionality and user-friendliness of each of the software packages, including one which was available for demonstration purposes, but which, because of cost, we did not use for our extraction exercise (UN-SCAN-IT). We explored the relative merits of the available software, and agreed on the greater flexibility and simplicity of the DigitizeIt software over the Grab It! software, although in our testing the latter delivered greater precision and accuracy.
Conclusions: There may be a place for digital software in graphical data extraction. However, its value is contingent upon transparent decision-making by the reviewer, and upon an accurate set-up procedure. The results may be no more reliable than those produced by an experienced 'by hand-and-eye' extractor, but the digital software could offer the benefits of consistency and replicability.
Methods: Using figures from trial reports produced within our own Department and published in the BMJ and the Lancet, two authors (KC and MS) independently extracted data manually 'by hand and eye', using rulers, protractors and sharp pencils. We then processed the same graphs within two commercially available software packages (DigitizeIt and Grab It!), and compared the results obtained. We then evaluated the accuracy of each method against the source data from which the original graphs were produced.
Results: The software packages demonstrated some benefits in accuracy and precision, although care in setting up the extraction process and authorial decisions about key parameters were necessary, and were inevitably susceptible to variation in personal judgements. Occasional errors were made in identifying the correct data points for extraction, but these were related to the intricacies of the graphical figures, and were equally likely to occur whether a digital or a manual extraction method was used. One of the manual data extractors and one of the digital packages achieved high and comparable precision and accuracy, compared with the source material. The mean difference (i.e. accuracy) between manual extraction and source for Extractor B was 0.160, with a precision of 7.5 estimated by the total size of the 95% reference range (-3.581 to +3.902). The mean difference between the source data and the Grab It! extraction was 0.497, with a precision of 4.5 (95% reference range -1.770 to +2.765). Bland-Altman plots were used to visualise the relative accuracy and precision of these extractions. A secondary outcome was the functionality and user-friendliness of each of the software packages, including one which was available for demonstration purposes, but which, because of cost, we did not use for our extraction exercise (UN-SCAN-IT). We explored the relative merits of the available software, and agreed on the greater flexibility and simplicity of the DigitizeIt software over the Grab It! software, although in our testing the latter delivered greater precision and accuracy.
Conclusions: There may be a place for digital software in graphical data extraction. However, its value is contingent upon transparent decision-making by the reviewer, and upon an accurate set-up procedure. The results may be no more reliable than those produced by an experienced 'by hand-and-eye' extractor, but the digital software could offer the benefits of consistency and replicability.