Article type
Year
Abstract
Background: decision makers are interested not only in whether an intervention works but also how it works and its components. Programme and implementation descriptions, and other descriptive information, can provide valuable evidence on these questions. To include these sources in evidence syntheses, we need to assess the extent to which the evidence they provide is reliable. However, there are few tools available for critical appraisal of programme descriptions, implementation descriptions and other ‘non-conventional’ types of evidence. The ‘Ways of Evaluating Important and Relevant Data’ (WEIRD) tool addresses this gap.
Objectives: to describe a new tool to assess the limitations of non-conventional evidence sources such as programme, implementation and systems reform descriptions.
Methods: we developed WEIRD in several stages. Stage 1: we reviewed related tools, including the NICE (National Institute for Health and Care Excellence) tool for critical appraisal of grey literature, the AACODS checklist for grey literature, and the TIDieR checklist for describing interventions. Stage 2: drawing from criteria described in existing tools and team discussions, we created a first draft of WEIRD. The WEIRD tool comprises a set of questions that can be applied to source materials. Stage 3: the draft tool was piloted in three evidence syntheses and further changes then made. Stage 4: we conducted group discussions to obtain feedback from potential users and stakeholders, including health systems managers. A consensus process with a reference group is proposed as a final stage.
Results: the WEIRD tool includes 11 criteria that can be applied to source materials to assess their limitations: 10 criteria are applicable to all sources while one is applicable only to sources that include empirical data. For each criterion, users choose one of YES, NO or UNCLEAR. Users should provide a justification for the assessment made, preferably supported by extracts from the source materials. Based on the judgements made for each criterion, the user makes an overall assessment of the limitations of the source. Overall limitations can be assessed as: no or very minor concerns; minor concerns; moderate concerns; or serious concerns. Users should give a justification for the overall assessment.
Conclusions: the WEIRD tool supports the utilization of sources such as programme and implementation descriptions in evidence syntheses to inform decision processes. WEIRD assessments can feed into an assessment of how much confidence to place in findings from a synthesis, for example using the GRADE-CERQual approach.
Patient or healthcare consumer involvement: health system managers have been involved in this work through providing feedback on the tool criteria and their application. Other types of consumers were not involved in this work. However, the tool facilitates the inclusion of non-conventional data sources in evidence syntheses - these sources can help better design programmes that address people's needs.
Objectives: to describe a new tool to assess the limitations of non-conventional evidence sources such as programme, implementation and systems reform descriptions.
Methods: we developed WEIRD in several stages. Stage 1: we reviewed related tools, including the NICE (National Institute for Health and Care Excellence) tool for critical appraisal of grey literature, the AACODS checklist for grey literature, and the TIDieR checklist for describing interventions. Stage 2: drawing from criteria described in existing tools and team discussions, we created a first draft of WEIRD. The WEIRD tool comprises a set of questions that can be applied to source materials. Stage 3: the draft tool was piloted in three evidence syntheses and further changes then made. Stage 4: we conducted group discussions to obtain feedback from potential users and stakeholders, including health systems managers. A consensus process with a reference group is proposed as a final stage.
Results: the WEIRD tool includes 11 criteria that can be applied to source materials to assess their limitations: 10 criteria are applicable to all sources while one is applicable only to sources that include empirical data. For each criterion, users choose one of YES, NO or UNCLEAR. Users should provide a justification for the assessment made, preferably supported by extracts from the source materials. Based on the judgements made for each criterion, the user makes an overall assessment of the limitations of the source. Overall limitations can be assessed as: no or very minor concerns; minor concerns; moderate concerns; or serious concerns. Users should give a justification for the overall assessment.
Conclusions: the WEIRD tool supports the utilization of sources such as programme and implementation descriptions in evidence syntheses to inform decision processes. WEIRD assessments can feed into an assessment of how much confidence to place in findings from a synthesis, for example using the GRADE-CERQual approach.
Patient or healthcare consumer involvement: health system managers have been involved in this work through providing feedback on the tool criteria and their application. Other types of consumers were not involved in this work. However, the tool facilitates the inclusion of non-conventional data sources in evidence syntheses - these sources can help better design programmes that address people's needs.
PDF