Background: There has been considerable discussion about how to evaluate the methodological rigour of primary studies included in reviews. This has focused on both the criteria and the method for establishing high versus other quality. The Effective Public Health Practice Project ( EPHPP) has developed and tested an instrument for use with randomized and observational studies. The reliability and some validity of the criteria included in this instrument have been established. There is some consensus that one should not sum the results for these criteria to determine overall quality (and therefore inclusion in analysis), but include all relevant studies or those that meet a priori determined criteria deemed important. To date, according to a standardized set of criteria, the EPHPP has labeled studies strong, moderate or weak based on the number of methodological criteria that are judged to be strong, moderate or weak. Our reviews do not have a meta-analysis, but are narratively synthesized.

Objectives: The objective of this presentation is to determine the difference made in identifying high quality studies from other studies by either summing results for the six criteria or identifying a priori the most relevant criteria and using them to determine study inclusion.

Methods: The EPHPP has recently completed a large review to answer the following question: What is the effectiveness of interventions for promoting healthy weight, preventing overweight/obesity and increasing physical activity among school-aged children and youth? In this review, we have included all relevant studies in the tables, but only discussed those we identified as strong in the text. We are in the process of determining the differences between those studies we have selected to discuss by this method compared to those that would have been selected by our previous method. Analysis will include a correlation matrix to test the level of agreement of the methodologically strong studies identified by the new process with those identified by the previous process.

Results: During the review, it was difficult for the investigators to agree upon the most important criteria. Or more specifically, to agree on those that were unimportant beyond those where there was little variation among studies. The results of the correlation will be presented at the Colloquium.

Conclusions: It is possible that because public health interventions tend to be complex, this process is not feasible for these studies. On the other hand, it may be that one or two criteria are more important than the others and this would simplify future quality assessment procedures in this field.