How to rate the certainty of prediction modelling studies in a systematic review?

Article type
Authors
Van Remoortel H1, Scheers H1, De Buck E2, Vandekerckhove P3
1Centre for Evidence-Based Practice, Belgian Red Cross, Mechelen
2Centre for Evidence-Based Practice, Belgian Red Cross, Mechelen, Department of Public Health and Primary Care, Faculty of Medicine, KU Leuven, Cochrane First Aid
3Belgian Red Cross, Mechelen, Department of Public Health and Primary Care, Faculty of Medicine, KU Leuven
Abstract
Background: mass gatherings attended by large crowds are an increasingly common feature of society. In parallel, an increased number of studies have been conducted to identify those variables that are associated with increased medical usage rates.

Objectives:
1) to conduct a systematic review answering the PICO (patient, intervention, comparison, outcome) question ‘Which predictive models (I) are available for emergency services planning (O) during mass gathering events (P)?’;
2) to apply the GRADE approach to rate the certainty of the included prediction modelling studies.

Methods: we retained prediction modelling studies from six databases following systematic searching. We selected for analysis predictors for patient presentation rate (PPR) and/or transfer to hospital rate (TTHR) that were included in a multivariate regression model, and summarized the evidence narratively. We assessed methodological limitations by making a judgment on the risk of bias items of the CHARMS checklist (outcome(s) to be predicted, candidate predictors, missing data, model development). We assessed indirectness, imprecision, inconsistency, publication bias and the three upgrade criteria (large effect, dose-response gradient, plausible confounding) by the GRADE guidelines when evidence for an effect is summarized narratively. The initial certainty level was set at ‘high’ (association between predictors and outcomes irrespective of any causal connection).

Results: we identified 12 prediction modelling studies, performed in the USA (n =7 ), Australia (n = 3), Japan (n = 1) and Singapore (n = 1), with a combined audience of more than 32 million people in over 1500 mass gatherings. Statistically significant variables (P < 0.05) to predict PPR and/or TTHR were: accommodation, type of event, weather conditions, crowd size, day versus night, demographic variables, sports event distance, level of competition, free water availability and TTHR-predictive factors (number of patient presentations, type of injury). We downgraded the evidence (-1) due to methodological limitations, identified by using the CHARMS checklist. Since the studies were mainly conducted in the USA/Australia, results cannot be extrapolated to other contexts, continents, countries (downgrading (-1) for indirectness). No reason for upgrading or further downgrading due to imprecision, inconsistency or publication bias was present. Therefore, the final certainty in the effect estimates was considered as ‘low’.

Conclusions: the GRADE approach and the CHARMS checklist allow researchers to rate the certainty of prediction modelling studies.

Patient or healthcare consumer involvement: further formal guidance from the GRADE working group is recommended to use the GRADE approach on prediction modelling studies. This will improve evidence-based mass gathering medicine by more effective pre-event planning and resource provision.