Article type
Year
Abstract
Background:
Preliminary filtering of searches is one of the most time consuming aspects of systematic reviewing. Cochrane Review authors use a variety of approaches (manual and electronic) but no single method satisfactorily fulfills the principal requirements of speed with accuracy.
Objectives:
Pilot testing of Rayyan (rayyan.qcri.org) focused on usability and assessment of how accurately the tool performed against manual methods, and evaluation of the added benefit of the prediction feature.
Methods:
Searches from two published Cochrane Reviews (1030 and 273 records, respectively) were used to test the app (December 2013 to March 2014). Included studies had been previously selected manually for the reviews; original searches were uploaded into Rayyan.
Results:
One recently updated Cochrane Review (273 records) was used as a taster, allowing a quick overview of the look and feel of Rayyan and for early feedback on usability to be addressed by the developers. The second Cochrane Review (1030 records) required several iterations to identify the 11 trials that had previously been selected manually. The suggestions and hints, based on the prediction rules, appeared as the testing progressed beyond five included studies. The selection process was responsive and effective; the options undecided / included / excluded / suggested were clearly displayed. Search functions included limiters relevance / title / date. Innovative features include: word clouds as graphical indicators of key terms; translation option linked to Google Translate to enable a quick translation or forwarding to a translator; similarity-based exploration of studies; labelling of studies, including reasons for exclusion. Key functionality includes the unambiguous way in which studies could be viewed in context together with the completed selections, and how undecided studies could be fed back into the system and were then highlighted as hint.
Conclusion:
Rayyan (beta-testing phase) is responsive, and largely intuitive to use, with a significant potential to lighten the load of review authors by speeding up this tedious part of the process.
Preliminary filtering of searches is one of the most time consuming aspects of systematic reviewing. Cochrane Review authors use a variety of approaches (manual and electronic) but no single method satisfactorily fulfills the principal requirements of speed with accuracy.
Objectives:
Pilot testing of Rayyan (rayyan.qcri.org) focused on usability and assessment of how accurately the tool performed against manual methods, and evaluation of the added benefit of the prediction feature.
Methods:
Searches from two published Cochrane Reviews (1030 and 273 records, respectively) were used to test the app (December 2013 to March 2014). Included studies had been previously selected manually for the reviews; original searches were uploaded into Rayyan.
Results:
One recently updated Cochrane Review (273 records) was used as a taster, allowing a quick overview of the look and feel of Rayyan and for early feedback on usability to be addressed by the developers. The second Cochrane Review (1030 records) required several iterations to identify the 11 trials that had previously been selected manually. The suggestions and hints, based on the prediction rules, appeared as the testing progressed beyond five included studies. The selection process was responsive and effective; the options undecided / included / excluded / suggested were clearly displayed. Search functions included limiters relevance / title / date. Innovative features include: word clouds as graphical indicators of key terms; translation option linked to Google Translate to enable a quick translation or forwarding to a translator; similarity-based exploration of studies; labelling of studies, including reasons for exclusion. Key functionality includes the unambiguous way in which studies could be viewed in context together with the completed selections, and how undecided studies could be fed back into the system and were then highlighted as hint.
Conclusion:
Rayyan (beta-testing phase) is responsive, and largely intuitive to use, with a significant potential to lighten the load of review authors by speeding up this tedious part of the process.