Background: Numerous tools have been developed to evaluate competence in EBM. Two widely used scales are the Fresno test and Berlin questionnaire. Both scales were validated and purport to assess a comprehensive range of EBM knowledge. Fresno test consists of mainly short-answer questions and Berlin questionnaire entirely multiple-choice questions. Objectives: To determine the agreement between the two scales when administered to medical undergraduates. Methods: We recruited two cohorts of final-year medical students undergoing six-month clerkship training (September 2006 to February 2007 and March to August 2007). We administered the Fresno test and Berlin questionnaire concurrently, midway through their clerkship, within which they received a structured EBM training. Two authors, having scored the Fresno test scripts independently, determined the final scores via consensus. We set the following pass marks: Fresno test: 50% (106/212) and Berlin questionnaire: 53% (8/15). We randomly sequenced the test scripts, so half started with the Fresno test and the other half Berlin questionnaire. Results: Eighty-three students participated in the study. Inter-rater correlation for Fresno test scores was 0.89. Students performed similarly in both tests (averages: Fresno: 45% (SD 10.1) and Berlin: 44.7% (SD 14.9), p=0.735), with 36% passing the Fresno test and 45% passing the Berlin questionnaire. The overall correlation between the two scales was poor (r=0.207, p=0.08). Comparing questions that examine knowledge on the same domains of EBM, correlations remained poor (r:-0.178 to 0.244, p:0.026 to 0.653). Conclusions: Scores in Fresno test and Berlin questionnaire correlated poorly when administered to medical students. This could not be explained by different levels of difficulty, as performances were similar in both tests. Overall, students scored poorer than expected of our graduates, possibly because the tests were administered before they completed their EBM training within the clerkship. Further work is needed to develop a truly comprehensive and reliable scale for measuring competence in EBM.