Background: Providing evidence-based practice (EBP) training programs to health care practitioners has been proposed as an effective way of facilitating an evidence-based approach to clinical practice. Educators who provide training require standard robust instruments to evaluate the effectiveness of EBP training programs, and to document changes in competence of the practitioners being trained. There is only one objective measure of EBP knowledge and skills which has been tested and applied in allied health, the Adapted Fresno Test (AFT). The AFT is a seven-item instrument for assessing knowledge in the major domains of EBP, such as formulating questions, searching for and critically appraising research evidence.

Objectives: This study examined the inter-rater reliability of The AFT using a number of raters with different professional experience.

Methods: The AFT was completed by physiotherapists and occupational therapists and a random sample of 12 tests were scored by four raters with different professional experience. Inter-rater reliability was calculated with the use of intra-class correlation coefficients (ICC [2, 1]) for the subtest scores and the total AFT score.

Results: Inter-rater reliability was moderate to excellent for items 1 and 7 (ICC, 0.63–0.95). Questionable levels of agreement among raters were found in other items and the total score. For these items, the raters were clustered into two groups—‘experienced’ and ‘inexperienced’, and then examined for agreement. The reliability estimates for rater 1 and rater 2 (‘inexperienced’) increased slightly for items 2, 5 and total score but not for all other items. For raters 3 and 4 (‘experienced’), ICCs increased considerably, indicating excellent agreement for all items and total score (0.80–0.99), except for item 4 where a further decrease in ICC was obtained.

Conclusions: The use of AFT to assess one’s knowledge and skills in EBP may be problematic unless raters are carefully selected and trained.