Background: Apparent sub-group effects often arise in systematic reviews of RCTs, suggesting the need for a formal instrument for rating their credibility. Objectives: To develop an instrument for evaluating reports of both primary RCTs and systematic reviews resulting in a rating of sub-group effects credibility from extremely implausible to extremely likely. Methods: To generate an item pool, we reviewed methods articles addressing the issue of sub-group analysis credibility and consulted with a group of 18 methodologically trained individuals participating in a review of sub-group analysis in RCTs. The steering group of the study decided on item wording and response options. We plan to ask 5 external senior methodologists to evaluate the draft instrument for clarity, completeness, and redundancy. We will apply the instrument to a sample of primary studies and meta-analyses that make claims regarding sub-group effects. Two raters will independently evaluate each article; results will allow estimation of reliability and conduct of a factor analysis that will guide possible item reduction. Exploration of construct validity will include 1) examining the relation between application of the instrument to particular sub-group analyses versus authors’ claims in those same analyses 2) in situations in which conclusions regarding sub-group effects are definitive, examining the extent to which application of the instrument at an earlier time – when fewer studies were available – predicts the ultimate outcome. Results: Item generation yielded 17 items including issues of power and statistical significance, a priori hypothesis generation, within or between study comparison, and the methodological quality of the original studies (Table). We have decided on a four-point scale for response options (e.g., definitely not to definitely yes). We anticipate data from subsequent stages of the project will be available at the time of the Colloquium. Conclusion: Our instrument should prove helpful to Cochrane reviewers evaluating possible sub-group effects.