Article type
Year
Abstract
Background: Evidence-based medicine (EBM) and information mastery (IM) skills are taught in many medical training programs, but there are few published assessment measures of these skills with known psychometric properties. Further, none have been used in simulated objective structured clinical examination (OSCE) settings.
Objectives: To develop and test an OSCE to assess the skills of information mastery and use of best evidence among medical residents.
Methods: An 'Integrated OSCE' that assessed the skills of information mastery and use of best evidence using standardized patients was developed. In the 'EBM OSCE' station residents had 30 minutes to find the best evidence answer to a clinical question arising from the simulated OSCE case. They completed a standardized form documenting a 4-part question, search strategies, resources searched, brief critical appraisal, efficiency of the search, and best evidence answer, including rationale for choosing it. Using theoretical underpinnings of EBM, a 10-item checklist measure and a single item global measure were developed to assess this OSCE. Use of the Cochrane Database of Systematic Reviews (CDSR) and other sources of high quality systematic reviews were highest scored. Three independent raters were used. In phase one the measure was tested with 12 1st year, 21 2nd year residents, and then edited. In phase two the refined measures were tested on 33 residents in all three post-graduate years from three residency centers.
Results: The full 10-item measure had good face validity, an inter-rater reliability of 0.901, and a Cronbach alpha of .56 for all 3 raters. A final 4-item 'EBM' factor emerged from factor analysis with an inter-rater reliability of .96, and a Cronbach alpha of .73. There were no differences in scores among the three years of residents, and the Cochrane Library was little used.
Conclusions: A new 'EBM OSCE' measure that uses resources such as the Cochrane Library contained good psychometric properties. The findings of no variation in scores between residency levels and low Cochrane Library utilization are likely the result of little EBM/IM training in these programs or in their feeder schools until very recently.
Objectives: To develop and test an OSCE to assess the skills of information mastery and use of best evidence among medical residents.
Methods: An 'Integrated OSCE' that assessed the skills of information mastery and use of best evidence using standardized patients was developed. In the 'EBM OSCE' station residents had 30 minutes to find the best evidence answer to a clinical question arising from the simulated OSCE case. They completed a standardized form documenting a 4-part question, search strategies, resources searched, brief critical appraisal, efficiency of the search, and best evidence answer, including rationale for choosing it. Using theoretical underpinnings of EBM, a 10-item checklist measure and a single item global measure were developed to assess this OSCE. Use of the Cochrane Database of Systematic Reviews (CDSR) and other sources of high quality systematic reviews were highest scored. Three independent raters were used. In phase one the measure was tested with 12 1st year, 21 2nd year residents, and then edited. In phase two the refined measures were tested on 33 residents in all three post-graduate years from three residency centers.
Results: The full 10-item measure had good face validity, an inter-rater reliability of 0.901, and a Cronbach alpha of .56 for all 3 raters. A final 4-item 'EBM' factor emerged from factor analysis with an inter-rater reliability of .96, and a Cronbach alpha of .73. There were no differences in scores among the three years of residents, and the Cochrane Library was little used.
Conclusions: A new 'EBM OSCE' measure that uses resources such as the Cochrane Library contained good psychometric properties. The findings of no variation in scores between residency levels and low Cochrane Library utilization are likely the result of little EBM/IM training in these programs or in their feeder schools until very recently.