Background: The option of automating individual working steps when creating systematic reviews has been discussed for many years [1,2]. The use of machine-learning algorithms for the identification of randomized controlled trials (RCTs) appears to be particularly promising. The publication by Marshall et al. [3], for example, shows that the RCT classifier provided by RobotSearch [4] can be reliably used (in terms of sensitivity) instead of conventional database search filters. But what does this mean in practice?
Objectives: The aim of our analysis was to determine whether the use of an RCT classifier could be integrated into the standard working process at the German Institute for Quality and Efficiency in Health Care (IQWiG).
Methods: In two projects (1. Biomarkers in breast cancer; 2. Osteoporosis drugs), we compared the results yielded using the RCT classifier (sensitive model) with those yielded using our standard working process (see Table 1) with conventional search filters. In addition, we tested the practical applicability of the RCT classifier in our daily project work.
Results: Using the sensitive CHSSS filter in MEDLINE and an optimized filter in Embase for Project 1 (non-drug assessment), the number of hits with the RCT classifier was lower than the number with the standard process. However, using the optimized filters for MEDLINE and Embase for Project 2 (drug assessment), the number of hits increased considerably with the RCT classifier (4803 vs. 2589 in MEDLINE). Applying the standard process together with the RCT classifier would have significantly reduced the number of hits (- 29%); see Table 2.
In addition, in the practical application of the RCT classifier, a number of new issues arose:
- longer processing times due to a high number of hits
- higher workload due to increased documentation and error-proneness due to media disruption
- less information in the data file due to the Research Information System (RIS) format required
Conclusions: We explicitly support the automation of work processes and the RobotSearch interface is easy to use. However, routine use of the RCT classifier at IQWiG is unlikely in the near future, as the advantages (e.g. fewer hits to screen) do not seem to outweigh the disadvantages. Further testing of the reliability of the RCT classifier as a complementary tool to the standard study filters would be useful.

1. Tsafnat G. Systematic review automation technologies. Syst Rev 2014; 3: 74.
2. Beller E. Making progress with the automation of systematic reviews. Syst Rev 2018; 7(1): 77.
3. Marshall I. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Res Syn Meth 2018; 9(4): 602-614.
4. RobotSearch RCT classifier. URL: https://robotsearch.vortext.systems.
5. Lefebvre C. Technical supplement to chapter 4: searching for and selecting studies. URL: https://training.cochrane.org/handbook
6. Wong SSL. Comparison of top-performing search strategies. J Med Libr Assoc 2006; 94(4): 451-455.

Patient or healthcare consumer involvement: no