Article type
Year
Abstract
Introduction/Objective: Assessing the quality of the published evidence for evidence reports and technology assessments is critical to ensuring that the strength of the knowledge base about a given topic is properly and adequately reflected in the final document. Two important, linked tasks for documenting the strengths of the science base on efficacy and effectiveness of health care services are: (1) grading articles - clarifying the quality of the individual articles that ultimately are reviewed/abstracted and (2) grading evidence - combining information about the individual articles in ways that will convey the overall strength of the evidence about specific clinical questions. This paper presents issues for consideration by the US Agency for Health Care Policy and Research (AHCPR) in structuring a systematic approach to these tasks for the AHCPR Evidence-based Practice Centers (EPCs).
Methods: The author: solicited the EPCs for their approaches and documents; reviewed a selected set of published, unpublished, and Internet-based materials; and developed a policy paper for AHCPR that identified key issues, offered some options for the Agency's consideration, and outlined a research agenda in this area.
Results: Issues/or grading articles include: defining "quality"; specifying the reasons for grading quality of articles containing efficacy or effectiveness data; moving beyond a quality hierarchy based solely on study design; developing quality ratings for different types of health interventions (prevention; screening; diagnosis; treatment); devising quality ratings for policy-related questions; conducting quality ratings (who, when, how); creating categories, scales, or global/index scores; coping with poorly reported work; applying quality grades to meta-analysis, cost-effectiveness, or similar studies; and showing the feasibility of different approaches. Further issues for grading overall evidence include: showing comparative efficacy/effectiveness for alternative services; dealing with harms as well as benefits; finding confirmatory evidence using different types of studies; distinguishing between "no evidence of effect" and "evidence of no effect"; and finding a simple but unambiguous labelling scheme. Needed research includes methodologic and evaluation studies to determine (1) if various approaches to grading the literature and the strength of the science base are reliable, valid, and easy to use and interpret; should have "fatal flaws" or be accompanied by sensitivity analyses; can be universally applied or need to be tailored to prevention/screening, diagnosis, or therapy; and (2) what difference using such grading schemes makes to the final conclusions or recommendations in evidence reports or practice guidelines.
Discussion: These topics will be discussed in the context of their possible applicability to the work of the EPCs and the evolution of the international evidence-based practice movement.
Methods: The author: solicited the EPCs for their approaches and documents; reviewed a selected set of published, unpublished, and Internet-based materials; and developed a policy paper for AHCPR that identified key issues, offered some options for the Agency's consideration, and outlined a research agenda in this area.
Results: Issues/or grading articles include: defining "quality"; specifying the reasons for grading quality of articles containing efficacy or effectiveness data; moving beyond a quality hierarchy based solely on study design; developing quality ratings for different types of health interventions (prevention; screening; diagnosis; treatment); devising quality ratings for policy-related questions; conducting quality ratings (who, when, how); creating categories, scales, or global/index scores; coping with poorly reported work; applying quality grades to meta-analysis, cost-effectiveness, or similar studies; and showing the feasibility of different approaches. Further issues for grading overall evidence include: showing comparative efficacy/effectiveness for alternative services; dealing with harms as well as benefits; finding confirmatory evidence using different types of studies; distinguishing between "no evidence of effect" and "evidence of no effect"; and finding a simple but unambiguous labelling scheme. Needed research includes methodologic and evaluation studies to determine (1) if various approaches to grading the literature and the strength of the science base are reliable, valid, and easy to use and interpret; should have "fatal flaws" or be accompanied by sensitivity analyses; can be universally applied or need to be tailored to prevention/screening, diagnosis, or therapy; and (2) what difference using such grading schemes makes to the final conclusions or recommendations in evidence reports or practice guidelines.
Discussion: These topics will be discussed in the context of their possible applicability to the work of the EPCs and the evolution of the international evidence-based practice movement.