Review Article

 

Evidence Based Medicine Teaching in Undergraduate Medical Education: A Literature Review

 

Misa Mi

Medical Librarian

Oakland University William Beaumont School of Medicine Library

Rochester, Michigan, United States of America

Email: mi@oakland.edu

Received: 22 Apr. 2012 Accepted: 21 June 2012

 

 

Description: Description: Description: cc-ca_logo_xl2012 Mi. This is an Open Access article distributed under the terms of the Creative CommonsAttributionNoncommercialShare Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

 


 

Abstract

 

Objectives To determine the year when evidence based medicine (EBM) was introduced and the extent to which medical students were exposed to EBM in undergraduate medical education and to investigate how EBM interventions were designed, developed, implemented, and evaluated in the medical curriculum.

 

Methods A qualitative review of the literature on EBM interventions was conducted to synthesize results of studies published from January 1997 to December 2011. A comprehensive search was performed on PubMed, CINAHL, Web of Science, Cochrane Library, ProQuest Dissertations & Theses, PsycINFO, and ERIC. Articles were selected if the studies involved some form of quantitative and qualitative research design. Articles were excluded if they studied EBM interventions in medical schools outside the United States or if they examined EBM interventions for allied health profession education or at the levels of graduate medical education and continuing medical education. Thirteen studies which met the selection criteria were identified and reviewed. Information was abstracted including study design, year and setting of EBM intervention, instructional method, instruction delivery format, outcome measured, and evaluation method.

 

Results EBM was introduced to preclinical years in three studies, integrated into clinical clerkship rotations in primary care settings in eight studies, and spanned preclinical and clinical curricula in two studies. The duration of EBM interventions differed, ranging from a workshop of three student contact hours to a curriculum of 30 student contact hours. Five studies incorporated interactive and clinically integrated teaching and learning activities to support student learning. Diverse research designs, EBM interventions, and evaluation methods resulted in heterogeneity in results across the 13 studies.

Conclusions The review reveals wide variations in duration of EBM interventions, instructional methods, delivery formats for EBM instruction, implementation of an EBM intervention, outcomes measured, and evaluation methods, all of which remain relevant issues for further research. It is important for medical educators and health sciences librarians to attend to these issues in designing and delivering a successful EBM intervention in the undergraduate medical curriculum.

 

 


 

Introduction

 

Evidence based medicine (EBM) is healthcare practice based on integrating knowledge gained from the best available research evidence, clinical expertise, and patients values and circumstances (Dickersin, Straus, & Bero, 2007). Evidence based practice emphasizes new skills that learners must acquire and use: question formation, search and retrieval of the best available evidence, critical appraisal of the evidence, and application of the evidence to an individual or a group of patients (Finkel, Brown, Gerber, & Supino, 2003; Montori & Guyatt, 2008). EBM serves as a powerful educational tool or strategy that allows students and clinicians to become lifelong self-directed learners and information masters able to fill their knowledge gaps and practice high quality medicine (Barnett, Smith, & Swartz, 1999; Bordley, Fagan, & Theige, 1997).

 

EBM as an approach to practicing medicine has spawned widespread attention from medical educators who consider EBM as an important content area to be addressed in the undergraduate medical education curriculum. The Association of American Medical Colleges (AAMC) established the Medical School Graduation Questionnaire (GQ), a national questionnaire, for medical schools to use in program evaluation and to improve the medical student experience (Association of American Medical Colleges, 2012). The questionnaire contains specific items asking graduating medical students to self-report the level of adequacy in EBM instruction in their medical education curriculum. The Liaison Committee on Medical Education (LCME) that accredits medical education programs in the United States and Canada considers GQ data as part of important evidence demonstrative of the educational quality of new and established medical educational programs (Melnyk, 2006). The LCME also includes education standards for EBM related skills being appropriately addressed in a medical education program (Liaison Committee on Medical Education, 2012). To develop and implement innovations to medical schools across the United States, the Undergraduate Medical Education for the 21st Century project identified the practice of EBM and population-based medicine as one of nine key content areas to be integrated into medical school curricula in the clinical years (O'Connell & Pascoe, 2004).

 

At the level of graduate medical education, the accreditation body for residency programs in the United States the Accreditation Council for Graduate Medical Education (ACGME) endorsed six core competencies for residents in residency programs, one of which is the competency in practice based learning and improvement (Steward, 2001). The competency requirement stipulates that residents demonstrate the ability to locate, appraise, and apply evidence from scientific studies to their patient health problem. To better prepare medical students for beginning a medical residency program, a medical education program is expected to incorporate EBM into its curriculum to teach students basic skills in clinical decision making and application of evidence based information to medical practice.

 

As consumers of health/medical information, physicians are inundated with diverse and abundant information resources with variable quality and clinical relevance. They face a formidable challenge of staying current with biomedical knowledge and applying the growing knowledge to specific patients. The skills needed to find potentially relevant studies quickly and reliably, to separate the wheat from the chaff, and to apply sound research findings to patient care have today become as essential as skills with a stethoscope (Glasziou, Burls, & Gilbert, 2008, p. 704). Given the health care environment that is changing constantly and becoming increasingly sensitive to optimizing the quality and cost of patient care, medical educators face an important task of developing curricula that provide a sound scientific foundation for clinical practice and scholarly investigation and that prepare medical students to become physicians to practice medicine competently in the 21st century (Mahoney et al., 2004).

 

Medical students need to master an enormous volume of medical knowledge in their medical education from various sources. They may be very comfortable in using Google or Wikipedia to search for information. However, they lack proficiency in identifying, searching, and filtering information appropriate to address different types of clinical questions. Thus, it is essential for them to develop high level information literacy and acquire fundamental, important skills for lifelong, self-directed, problem based learning from the early years of their medical education. Basic EBM skills of searching, appraising, and applying research evidence to individual patients should be taught early and applied as an integral part of learning throughout the four years of the medical education (Glasziou et al., 2008). EBM integrated into the curriculum will be able to educate students who will become more effective residents and, ultimately, practicing physicians best able to make informed clinical decisions about the care of patients and to handle the exponential growth of biomedical knowledge.

 

It is important for medical educators and health sciences librarians to become cognizant of the available education research evidence and current practice in teaching EBM in undergraduate medical education. The knowledge will aid them as they launch their efforts of designing, developing, and implementing an EBM intervention, whether it is in the format of a curriculum, course, or series of workshops in preclinical or clinical years. A comprehensive literature search yielded no systematic or other types of reviews on EBM in the undergraduate medical education. This qualitative review of studies on EBM was conducted to determine the year when EBM was introduced and the extent to which medical students were exposed to EBM in undergraduate medical education. The review also attempted to examine how EBM interventions were designed, developed, implemented, and evaluated in the medical curriculum.

 

Methods

 

Data Sources

 

The review of the literature covered the period from 1997, the publication year for David Sacketts seminal work Evidence-Based Medicine: How to Practice and Teach EBM, to 2011, when the literature review for this article was performed. Since 1997, Sacketts work has helped drive the paradigm shift in the practice of medicine and has had a far reaching impact on the evolution of EBM teaching across the continuum of medical education.

 

The literature search included PubMed, CINAHL, Web of Science, Cochrane Library, ProQuest Dissertations & Thesis, PsycINFO, and ERIC. PubMed was searched by subject with MeSH terms (medical subject headings): education, medical, undergraduate; students, medical; evidence-based practice; and evidence-based medicine. The search results were limited to the language of English. PubMed and other databases were searched with the text words of evidence-based medicine, evidence-based practice, and evidence. These words were combined with medical education, undergraduate medical education, medical school, clerkship, medical students, as well as variations of the following text words: teach*, learn*, train*, educat*, instruct*, curricul*, and program. The search of these databases was supplemented by hand searching of reference lists of all included articles. Cited references were also searched in Web of Science to identify relevant articles that cited those studies selected for the review.

 

Selection Criteria

 

Studies were included in the review if they evaluated the effect of formal EBM teaching in undergraduate medical education programs leading to a medical degree accredited by the LCME. Formal EBM teaching was defined as any intervention in the form of a curriculum, required or elective course, EBM components integrated into a preclinical course, curriculum, or clinical clerkship rotation.

 

The types of research design utilized in clinical research may be inadequate in researching, investigating, and reporting educational phenomena with complexities of educational programs and wide variations in instructional methods, learning outcomes, and differences in learner characteristics. Qualitative research methods or alternative educational research methods are underutilized and valuable research tools in medical education research. As Hatala and Guyatt pointed out, well-designed qualitative studies using these methods are equally valuable in contributing evidence to the research of teaching evidence based medicine (2002). Therefore, studies that utilized the following types of research design (Table 1) commonly employed in education research were selected for the review (Campbell & Stanley, 1963; Gall, Gall, & Borg, 2007). Further information on each research design type can be found in Appendix A.

 

 

Table 1

Types of Research Design

Control-group designs with random assignment

1.       Pretest-posttest control-group design

R O X O

R O O

2.       Posttest-only control-group design

R X O

R O

Quasi-experimental designs

3.       Static-group comparison design
X O

O

4.       Nonequivalent control-group design

O X O

O O

Single-group designs

5.       One-short case study

X O

6.       One-group pretest-posttest design
O X O

Note. R = randomization; O = pretest or posttest; X = experimental treatment/intervention.


Articles were excluded from the review if they were studies on:

         EBM teaching at postgraduate and continuing education levels

         EBM teaching that is not part of, or not integrated into a curriculum, an existing/required course, program, or clinical clerkship rotation in undergraduate medical education

         EBM in medical schools not accredited by the LCME

         Effect of EBM teaching in osteopathic medical education or other allied health professional education

 

Data Extraction

 

Data were extracted from each eligible study on study design, year of EBM introduced in the curriculum, setting, EBM intervention, outcome measured, instructional method, and evaluation method. Furthermore, information on instructional strategies was examined using the hierarchy of evidence based medicine teaching and learning activities (Table 2) as described by Khan and Coomarasamy (2006).

 

Kirkpatricks four levels of evaluation (Table 3) were adapted to determine the extent to which EBM outcomes were measured (Kirkpatrick, 1994, 1996). The variation in research designs, interventions, and outcome measures in the selected studies precludes quantitatively combining results with a meta-analysis technique.

 

Table 2

Hierarchy of Evidence Based Medicine Teaching and Learning Activities

Level 1

Interactive, and clinically integrated teaching and learning activities

Level 2

a)       Interactive, classroom based teaching and learning activities

b)       Didactic, but clinically integrated teaching and learning activities

Level 3

Didactic, and classroom or standalone teaching and learning activities

 

Table 3

Four Levels of Evaluation

Level 1

Learner satisfaction (reaction)

Perceptions, opinions, satisfaction

Level 2

Learning outcomes (learning)

Change in knowledge, skills, attitudes

Level 3

Performance improvement

(behaviour)

Change in behaviours, transfer of learning to a patient care setting, performance in a practice setting

Level 4

Patient/health outcomes (results)

Tangible results--improvement in the health and wellbeing of patients

 


Results

 

Literature Search

 

The search strategies identified 13 studies on EBM in undergraduate medical education which met the inclusion criteria. These articles were derived from a large pool of 1,279 articles through a process of title scanning, abstract reading, hand searching, and elimination of duplicate articles from multiple databases. The review of the studies was conducted within the framework of the instructional design model ADDIE (the five phases or steps in designing effective instruction): analysis, design, development, implementation, and evaluation in instructional design (Gustafson & Branch, 2007). The phase of analysis is beyond the scope of this review. Table 4 provides a summary of the studies selected for the review.

 

Table 4

Summary of Study Design, Year of EBM in the Curriculum, and Settings

Author (Year)

Study Design

Year of EBM in the Curriculum and Setting

Wadland, Barry, Farquhar, Holzman, and White (1999)

 

Static-group comparison design

Year 3: clinical campuses in 6 communities

Barnett et al. (2000)

Static-group comparison design

Year 1

Year 3 and Year 4: clinical clerkships in medicine, pediatrics, psychiatry, surgery, obstetrics and gynecology, neurology, community medicine, geriatrics

 

Ghali et al. (2000)

Nonequivalent control-group design

Year 3: ambulatory care sites during internal medicine clerkship

 

Thomas and Cofrancesco (2001)

One-shot case study

Year 3 and Year 4: ambulatory medicine clerkship rotation

 

Srinivasan et al. (2002)

One-shot case study

Year 1

 

Holloway, Nesbit, Bordley, and Noyes (2004)

One-shot case study

Year 1

Year 2 (same cohorts of first year medical students followed up throughout Year 2)

 

Dorsch, Aiyer, and Meyer (2004)

One-shot case study

Year 3: internal medicine clerkship

 

Cayley (2005)

One-group pretest-posttest design

Year 3: family medicine clerkship

 

Schilling, Wiecha, Polineni, and Khalil (2006)

Pretest-posttest control-group design

Year 3: family medicine clerkship

 

Nieman,Cheng, and Foxhall (2009)

One-shot case study

Year 1

 

Aronoff et al. (2010)

One-group pretest-posttest design

Year 3: core clinical clerkships

 

West, Jaeger, and McDonald (2011)

One group pretest-posttest design

Year 2

Year 3: clinical rotations of internal medicine, surgery, pediatrics, obstetrics and gynecology, neurology, and psychiatry

Sastre, Denny, McCoy, McCoy, and Spickard (2011)

One group pretest-posttest design

Year 3: inpatient portion of internal medicine clerkship blocks

 

 


 

 

Description of Studies

 

The 13 studies demonstrated variability in methodology. One study had the pretest-posttest control group design with random assignment; three studies had the quasi-experimental design; four studies used one group pretest-posttest design; and five studies utilized one shot case study design. Participants of the studies ranged from first-year to fourth-year medical students. Three studies reported the integration of EBM into preclinical education curricula. EBM was introduced to third-year clinical clerkships in eight studies.

 

Two studies reported a longitudinal EBM intervention that spanned preclinical and clinical years. Barnett et al. (2000) evaluated EBM instruction integrated into the first-year medical education curriculum and clinical education; while the study by West et al. (2011) reported the outcomes of a longitudinal EBM into the second-year preclinical education and third-year core clinical rotations. Studies on EBM during clinical rotations were conducted at various settings of clerkships of internal medicine, family medicine, and pediatrics on community campuses (Wadland et al., 1999), outpatient internal medicine clerkship (Ghali et al., 2000; Thomas & Cofrancesco, 2001), inpatient portion of internal medicine clerkship blocks (Sastre et al., 2011), family medicine clerkship (Cayley, 2005; Schilling et al., 2006), or core clinical clerkship rotations such as internal medicine, surgery, pediatrics, obstetrics and gynecology, neurology, and psychiatry (Aronoff et al., 2010; Barnett et al., 2000; West et al., 2011).

 

Design and Development of EBM Interventions

 

The primary goal of EBM instruction in the 13 studies was to develop medical students essential skills for evidence based practice and evidence based problem solving; other goals included developing lifelong learners (Barnett et al., 2000; Dorsch et al., 2004), introducing basic concepts of epidemiology and biostatistics (West et al., 2011), and developing and presenting a research proposal (Wadland et al., 1999). EBM skills were the main focus of the EBM interventions in the studies; however, the EBM interventions reported in three studies included no content on the second step of the EBM process acquiring the evidence (Srinivasan et al., 2002; Wadland et al., 1999; West et al., 2011).

 

There were a varied number of contact hours devoted to the EBM interventions in the 13 studies. The time ranged from three student contact hours in a workshop (Sastre et al., 2011) to 30 contact hours in an EBM curriculum (Holloway et al., 2004). EBM components were introduced into preclinical years in various ways: offered as a short EBM course of eight contact hours enhanced with problem based learning small group discussion sessions facilitated by clinicians (Srinivasan et al., 2002); integrated into courses such as Library Science and Medical Informatics, Epidemiology, Microbiology (Barnett et al., 2000), and Mastering Medical Information (Holloway et al., 2004); taught as part of a series of educational interventions in an elective family medicine preceptorship program for first and second-year medical students (Nieman, Cheng, & Foxhall, 2009); and offered as a course of 22 contact hours at the end of Year 1 (West et al., 2011). In clinical years, EBM was integrated with clinical educational experiences through inpatient and outpatient primary care clerkship rotations of internal medicine, family medicine, and several other core clinical clerkships (Aronoff et al., 2010; Cayley, 2005; Dorsch et al., 2004; Ghali et al., 2000; Sastre et al., 2011; Schilling et al., 2006; Thomas & Cofrancesco, 2001; Wadland et al., 1999). Among the 13 studies, only two studies, one by Barnett et al. (2000) and the other by West et al. (2011), had a longitudinal EBM curriculum that spanned the preclinical curriculum and clinical core clerkships.

 

The content domain of the EBM interventions in the 13 studies was derived from multiple sources. They included Users Guides to the Medical Literature (Guyatt & Rennie, 2002), a series of articles on harnessing MEDLINE (McKibbon, Walker-Dilks, Haynes, & Wilczynski, 1995; McKibbon & Walker-Dilks, 1994a, 1994b), EBM steps as outlined by Sackett, Rosenberg, Gray, Haynes, and Richardson (1996), and journal articles on EBM teaching. Four studies failed to report how EBM content was developed, although the EBM interventions in these studies reflected some basic EBM principles and processes widely recognized in the medical literature on EBM teaching and practice. The EBM curriculum in one study comprised only online content: a literature searching tutorial, the website of the National Guideline Clearinghouse (www.guideline.gov/), and the website on the calculation of NNT (number needed to treat). Three out of the 13 studies included instruction on using pre-appraised EBM resources such as DynaMed, ACP Journal Club, Cochrane Library, and the National Guideline Clearinghouse (Nieman et al., 2009; Sastre et al., 2011; Schilling et al., 2006).

 

EBM knowledge and component skills in the EBM interventions were taught with various methods such as didactic lecture, live demonstration, hands-on practicum, small group discussion, and case based presentation. In addition, journal club (Barnett et al., 2000), problem based small group discussion (Srinivasan et al., 2002), and worksheets (Dorsch et al., 2004; Nieman et al., 2009; Srinivasan et al., 2002) were also reported as interactive teaching and learning activities in the EBM interventions. To categorize the EBM interventions in the 13 studies with the hierarchy of effective teaching and learning activities proposed by Khan and Coomarasamy (2006), five studies incorporated EBM teaching and learning activities of level 1, which represents the most effective practice of teaching and learning of EBM; one study used activities of level 2a; six studies integrated activities of level 2b; and one study fell under level 3 activities.

 

Out of the 13 studies, three explored the alternative instruction delivery format to provide EBM learning experience for students on the Internet or through a web content management system (WebCT or Blackboard). Given the variability in facultys availability in teaching EBM, online EBM instruction could provide consistent and equivalent learning experiences for students rotating across multiple clinical training sites or offer students a tool to enhance their learning in a traditional classroom setting. Srinivasan et al. (2002) used a 20 page Web-based EBM curriculum and an online practice examination developed in WebCT to supplement a short EBM course made of components of didactic lectures and interactive, problem based small group discussion sessions. First-year medical students used the online EBM curriculum as an independent study tool to reinforce their EBM learning (Srinivasan et al., 2002). Schilling et al. (2006) investigated the use of a Web-based curriculum to teach EBM. The entire curriculum encompassed an online tutorial and information from two other websites. In the study, an asynchronous discussion board moderated by faculty was also used as a learning activity to encourage students discussion of their patient case encounter, clinical questions, search process, and findings. Although the curriculum was intended as a Web-based EBM instruction tool, EBM content covered was incomplete since essential EBM skills of critical appraisal and evidence application were not addressed in the EBM curriculum.

 

Another study reported an EBM intervention featuring a two part EBM course for third-year medical students who went through core clinical clerkship rotations (Aronoff et al., 2010). The first part was delivered in six didactic online modules of materials via Blackboard (Blackboard Academic Suite; Blackboard, Inc., Washington, DC), each of which was followed by a focused practicum that students completed and submitted to an online faculty mentor for review and feedback. The second part of the course required that each student generate a clinical question on a patient seen during each of four clinical rotations and complete four evidence summaries using the Critically Appraised Topic format (CAT) developed by Sackett, Richardson, Rosenberg, and Haynes (1997). The students sent the evidence summaries to their mentor for review and correction, if necessary.

 

Implementation of EBM Interventions

 

Nine out of the 13 studies did not provide any information on any faculty development opportunities nor delineate the preparation or qualification of any faculty involved in teaching EBM. Only four studies explained the process of implementing their EBM intervention, particularly faculty development efforts, on varying levels of detail. The first of the four studies was conducted at the College of Human Medicine at Michigan State University, which is a community-based medical school without a central university hospital. The EBM intervention of a nine session curriculum was delivered on clinical campuses across six communities. A course director and course coordinator provided central management for faculty development and course implementation. Each community campus had a research director who served as the community course coordinator and participated in the course content and format selection. A two day faculty development event was held in which four McMaster University faculty members led intensive training seminars to train and prepare 50 faculty members to teach EBM locally (Wadland et al., 1999).

 

The development of a four year longitudinal EBM curriculum reported by Barnett et al. (2000) started with the establishment of an Evidence Based Medicine Working Group at the Mount Sinai School of Medicine. Its multidisciplinary team members were charged with the task of revamping the EBM component in the traditional medical curriculum in collaboration with all preclinical course and clinical clerkship directors. Faculty development opportunities were provided on different levels, including a full day retreat for course and clerkship directors to receive EBM training provided by faculty from McMaster University. These trained course and clerkship directors in turn provided training for other faculty in their own department.

 

Srinivasan et al. (2002) investigated the effect of a one month EBM course integrated into the preclinical medical curriculum at the Indiana University School of Medicine. The course combined traditional didactic lectures with interactive small group, problem based learning sessions to teach 138 first-year medical students EBM principles and skills. The small group, problem based sessions were facilitated by 16 faculty members who represented four departments of emergency medicine, pediatrics, internal medicine, and library sciences at the medical school. To prepare faculty for the small group sessions, a facilitators handbook was developed to provide faculty with consistent small group experiences, detailed objectives, timelines, commonly asked questions and answers, sample dialog, completed Users Guides to the Medical Literature worksheets (Guyatt & Rennie, 2002), critical concept summaries, background reading material, and small group teaching strategies and references. Furthermore, dedicated secretarial support was available to faculty in the early introduction of the EBM course to preclinical medical students. All faculty facilitators who participated in the small group sessions had EBM experience through taking formal EBM courses, teaching EBM, holding a MPH (Master of Public Health) degree, or writing about EBM for peer reviewed journals.

 

In another study on an EBM longitudinal curriculum reported by West et al. (2011), EBM faculty were given supported time for each graded assignment and administrative time for direct student contact during the period of teaching and maintaining the short course in the curriculum. Each instructor involved in EBM teaching had extensive EBM knowledge and skills through their advanced training in biostatistics and epidemiology and participation in the workshop offered at McMaster University. These instructors also had experience in teaching basic and advanced EBM topics to residents at Mayo Clinic.

 

The EBM curriculum in the study by Wadland et al. (1999) was standardized to ensure the equivalent EBM learning experience in three primary care clerkships of internal medicine, family medicine, and pediatrics at multiple locations. Srinivasan et al. (2002) used didactic lectures in conjunction with the problem based learning approach which demanded a great deal of investment in resources (e.g., faculty and their time in facilitating small group sessions). Two other studies, one by Barnett et al. (2000) and the other by West et al. (2011), were characterized by a longitudinal curriculum integrated into both preclinical and clinical years of medical education. These four studies demonstrate that the successful implementation of an EBM intervention across multiple sites, with a problem based small group discussion component, or integrated into both preclinical and clinical education curricula, are contingent on important factors such as provision of faculty development opportunities, resources coordination, and instructional and administrative support for EBM faculty.

 

Evaluation of EBM Outcomes

 

Variations in evaluation methods used to assess the effect of the EBM interventions in the studies preclude any quantitative analysis of a pooled effect size of the results from the studies. Therefore, the outcomes reported in the studies were examined using Kirkpatricks four level evaluation model: satisfaction with a learning experience (level 1), learning in terms of any change in knowledge, skills, or attitudes (level 2), performance improvement or behavioural changes in a patient care setting (level 3), and impact on patient health outcomes (level 4). Five studies incorporated level 1 evaluation; all 13 studies examined changes in knowledge, skills, or attitudes at level 2; and three studies had level 3 evaluation of the outcomes of behavioural changes in a patient care setting. None of the studies went beyond level 3 evaluation. A summary of EBM interventions, outcomes measured, evaluation methods, and levels of evaluation is presented in Appendix B.

 

Level 1 Evaluation: Reaction and Satisfaction

 

Five out of the 13 studies evaluated learners satisfaction and reaction to the EBM intervention. Wadland et al. (1999) found that second-year medical students rating of an EBM course was not significantly different
from that of other courses offered in the medical curriculum and that there were no significant variations in responses across multiple community clinical sites. Srinivasan et al. (2002) evaluated both students and facultys satisfaction with an EBM course integrated into the first-year medical curriculum. The evaluative results showed that a majority of students reported enjoying the EBM course and felt that the course material was appropriate for their learning level and related to clinical practice. Srinivasan et al. (2002) also sought facultys feedback as an indicator of the effect of EBM instruction. In facultys opinions, medical students could perform well in EBM-related areas and use EBM concepts as well as or better than residents whom the faculty had supervised in the past year. Both faculty and student respondents agreed that the course material was appropriate for first-year medical students. In the study by Sastre et al. (2011), third-year medical students also reacted to their EBM learning positively. They felt that EBM learning was useful and that they would incorporate their acquired skills into clinical care of patients.

 

Holloway et al. (2004), however, reported that a majority of student respondents (58 out of 67 respondents) expressed negative comments about EBM instruction and evaluation that spanned the preclinical education. The students felt that there was too much emphasis on EBM (43-page syllabus) in their preclinical curriculum. They also commented that the test module with a 15-20 page evaluation packet (a five step EBM evaluation module) took too much time to complete and was of little value to their learning of good EBM skills. Thomas and Cofrancesco (2001) found that third- and fourth-year medical students rated the usefulness of their EBM learning through an EBM report less favorably than their clinical sessions with their preceptor. The researchers also found that the students perceived their clinical competence in EBM with a lower value than diagnostic decision making and self-directed learning.

 

Level 2 of Evaluation: Change in Skill and Knowledge

 

Level 2 evaluation was conducted in all 13 studies to measure any possible change in learners knowledge, skills, or attitudes as a result of EBM instruction. Learning outcomes in these studies were measured with a wide array of evaluation methods such as the AAMC Medical School Graduation Questionnaire (Wadland et al., 1999), critique of a relevant article (Barnett et al., 2000; Schilling et al., 2006), self-administered EBM skill survey (Dorsch et al., 2004; Ghali et al., 2000; Schilling et al., 2006; West et al., 2011), self-administered examination of knowledge and skills (Aronoff et al., 2010; Dorsch et al., 2004; Srinivasan et al., 2002; West et al., 2011), self-efficacy questionnaire (Cayley, 2005; Nieman et al., 2009), a five step EBM evaluation module (Holloway et al., 2004), PICO case summary (Nieman et al., 2009), and CAT (Critically Appraised Topic) summary (Aronoff et al., 2010).

 

One study reported an improved response rate for appropriate training in literature analysis and research skills on the AAMC Medical School Graduation Questionnaire since one of the objectives for the EBM curriculum in the study was to develop students research skills (Wadland et al., 1999). Five studies reported improvement in literature searching and other library skills (Barnett et al., 2000; Dorsch et al., 2004; Ghali et al., 2000; Holloway et al., 2004; Schilling et al., 2006). These skills were measured with various instruments such as a rating scale (Barnett et al., 2000; Schilling et al., 2006), evaluation rubric (Holloway et al., 2004), case scenario based skill test (Dorsch et al., 2004), and self-reported survey (Ghali et al., 2000). EBM skill in critical appraisal was evaluated as an outcome in six studies: four reported significant improvement on objective measures (Aronoff et al., 2010; Dorsch et al., 2004; Srinivasan et al., 2002; West et al., 2011); one reported improvement on self-reported skill test (Ghali et al., 2000); and one failed to report any evaluative results through a critical appraisal measure part of two evaluation modules that tested the five steps of the EBM process (Holloway et al., 2004). The results of most studies reveal improvement of varying degrees in terms of learners knowledge and skills in EBM.

 

Level 3 of Evaluation: Changes in Behaviour

 

Three studies incorporated level 3 evaluation measures in evaluating the extent to which any change was made in students behaviour in a clinical setting. Two of the three studies reported students successful use of EBM in actual clinical work through application of the five steps of the EBM process in an EBM report (Aronoff et al., 2010; Thomas & Cofrancesco, 2001). Students in the study by Thomas and Cofrancesco (2001) wrote an EBM report based on a real patient case. Students in the study by Aronoff et al. (2010) formulated a clinical question generated by a patient seen during each of four clinical rotations. A complete CAT form was required of students and reviewed but not scored by faculty mentors. In the study by Sastre et al. (2011), level 3 evaluation was conducted through evaluation of citations of EBM resources in students inpatient admission notes and the quality of the EBM component of the discussion portion in the notes. Another measure to evaluate students use of EBM resources in the same study was computer log recordings of students online use of various EBM resources via an electronic medical resource system. Analysis of students patient notes reveals a significant improvement on the overall quality of EBM incorporation into the discussion of a patients problem as a result of EBM instruction. The computer log data showed a significant increase in students usage for all electronic resources. EBM resources such as systematic peer reviewed resources (e.g., Cochrane Databases, Clinical Guidelines) represented 59% of all resources accessed following the workshop. However, there was no significant improvement in the number of citations per patient note before and after the EBM intervention.

 

Discussion

 

The review was conducted to summarize the findings of the studies retrieved from comprehensive literature searching of relevant resources. The analysis of 13 studies reveals assorted approaches to instructional design, development, implementation, and evaluation of EBM interventions in undergraduate medical education. The definition of EBM and component skills required to practice EBM are not debatable in the current medical literature. However, there is no standard or model for how EBM should be designed and developed and how it should be effectively taught and evaluated in undergraduate medical education. The studies show wide variations in the use of instructional strategies, delivery formats for EBM instruction, implementation of an EBM intervention, and outcome measures, all of which remain important issues for further research.

 

Effectiveness of an educational intervention is inescapably linked to the outcomes of educational activities (Belfield, Thomas, Bullock, Eynon, & Wall, 2001). Interactive and clinically integrated activities support active learning and could lead to medical students deep understanding of content material and superior experience in classroom and clinical settings. The review shows that several studies made efforts to incorporate clinically integrated EBM learning experiences and clinically relevant outcome measures into EBM instruction in the medical curriculum. However, more than half of the studies reviewed fell short of integrating interactive teaching and learning activities into the EBM interventions. Future research is required to determine the effect of EBM instruction on learning outcomes as a result of sound instructional design principles and active learning strategies employed in the design and development of an EBM intervention in undergraduate medical education.

 

Delivery of EBM instruction with information technology holds promise in affecting EBM learning outcomes for medical students. A study by Davis, Crabb, Rogers, Zamora, and Khan (2008) demonstrated that first-year medical students in a computer-based EBM session had gains in knowledge and attitude similar to those in the lecture-based EBM session. Cook et al. (2008) conducted a meta-analysis of Internet-based learning in the health professions. Their findings provided further evidence for the effectiveness of Internet-based instruction similar to traditional instructional methods. Three of the 13 studies utilized online learning as an approach to teaching EBM or supplementing the traditional method of teaching EBM. The results from these studies suggest implications for medical educators and health science librarians in developing and teaching EBM in the medical curriculum. Online/Internet-based EBM instruction that is properly designed, developed, and implemented could support students self-directed learning, reinforce EBM learning in a traditional classroom setting, or present great potential for standardizing the quality of EBM teaching to achieve equivalent learning experience across multiple clinical teaching sites or clerkship rotations.

 

To a great extent, effective delivery of an EBM intervention hinges on a good implementation plan addressing issues related to faculty recruitment for EBM teaching, faculty training in EBM, administrative support, time and space allocated to instruction, and procedures for recording and collecting data pertaining to course and student evaluation (Gustafson & Branch, 2007). Nine of the 13 studies failed to provide any information on the implementation process of an EBM intervention. Implementation is one of six important steps in curriculum development for medical education (Kern, Thomas, & Hughes, 2009), as well as in the process of instructional design (Gustafson & Branch, 2007). Successful integration of EBM into the medical curriculum necessitates EBM training for busy primary care clinicians or preceptors in the clinical teaching and application of EBM (Cayley, 2005; Nicholson, Warde, & Boker, 2007) and for teaching EBM in a longitudinal fashion rather than as a single innovation (Nieman et al., 2009). Given that faculty development is likely more important than any other step in implementation (Bordley et al., 1997), faculty development efforts are pivotal to any effective EBM instruction for medical students in either preclinical or clinical years, as evidenced by higher levels of learner satisfaction and knowledge gains found in the studies by Srinivasan et al. (2002) and West et al. (2011). Teaching EBM may need to focus as much on teachers as on students (Del Mar, Glasziou, & Mayer, 2004, p. 990). Implementation of a successful EBM intervention requires investment of adequate faculty time, resources (Mi, Moseley, & Green, 2012), and provision of faculty development opportunities (Green, 2000; Nicholson et al., 2007).

 

The analysis of the studies reveals a wide variation in EBM learning outcomes due to study heterogeneity in research designs, which did not lend to an estimation of a combined effect from these studies. The adapted 4 level evaluation model by Kirkpatrick was applied to examine the extent to which learning outcomes were evaluated. The results showed that the majority of the studies focused on level 1 and level 2 evaluation to measure learning that occurs separated from the real-time setting of practice. Few studies went beyond level 3 evaluation, which is more difficult to measure because it requires assessment in the practice setting (Straus et al., 2004). However, evaluation of the effectiveness of EBM as an educational intervention should aim to incorporate level 3 and level 4 evaluation into curriculum development to measure the long-term impact of EBM instruction on sustained behavioural changes in a patient care setting and possibly, patient health outcomes. West et al. (2011) stressed the need for research on the sustained increases in perceived and measured EBM knowledge and benefits beyond medical school into postgraduate medical education and practice. Clearly, students independent application of EBM skills in a clinical setting and into their residency warrants further research efforts.

 

The article only reviewed studies on EBM interventions in the medical curriculum of U.S. medical schools given the differences that exist between medical schools in the US and other countries in terms of duration of undergraduate medical education programs, requirement in competence-based learning outcomes, and program accreditation standards. Further effort could be made to systematically review the literature including international studies, related to specific topics such as the use of different instructional methods in teaching EBM (e.g, team based learning, problem based learning) and different instruction delivery formats (e.g., online instruction of EBM). Such review would afford additional insights into EBM teaching in undergraduate medical education.

 

The qualitative review was limited to full research reports published in peer review journals. The selected reports addressed formal EBM instruction as part of or integrated into preclinical or clinical curricula, which is the focus of the review. Future research could be conducted to examine how each EBM step or component (e.g., clinical question formation, searching the literature for the evidence, or critical appraisal) is instructed and evaluated in undergraduate medical education. Systematic reviews could be carried out through exhaustive searching of literature including conference proceedings and other grey literature to shed further light on teaching practices and innovations in EBM instruction in undergraduate medical education.

 

The inclusion criteria specifically established in this review circumscribed the number of studies selected. It should be pointed out that there is a view frequently held in medical education that all that is needed in medical education is commonsense (Harden, 1986, p. 522). The view may partly account for the level of evidence based educational research on EBM and other educational phenomena. Clearly, there is a need to employ different approaches to research and to break with or question existing teaching practices in medical education in general and EBM teaching in particular. Relevant research on the use of these approaches to research and creative and innovative methods of teaching EBM would add significantly to the literature in this field.

 

Conclusion

 

The review was conducted as an attempt to present research evidence on how EBM was designed, developed, implemented, and evaluated in undergraduate medical education and to help inform medical educators and health sciences librarians in their efforts to practice evidence based teaching of EBM. The review also pointed out a number of areas that warrant further research on EBM in undergraduate medical education.

 

Khan and Coomarasamy (2006) comment that the substantial heterogeneity in the types of educational interventions and their effects necessitate an analysis beyond simply focusing on the overall results of studies on EBM teaching. The article analyzed and discussed EBM interventions in educational research from the perspective of instructional design. It examined some relevant issues related to the treatment of EBM content in the undergraduate medical curriculum. These issues included EBM instructional design, implementation, faculty development, and levels of evaluation of learning outcomes. Failure to attend to these issues could undermine the effectiveness of an EBM intervention in the medical education curriculum.

 

 

References

 

 

Aronoff, S. C., Evans, B., Fleece, D., Lyons, P., Kaplan, L., & Rojas, R. (2010). Integrating evidence based medicine into undergraduate medical education: combining online instruction with clinical clerkships. Teaching & Learning in Medicine, 22(3), 219-223. doi:10.1080/10401334.2010.488460

 

Association of American Medical Colleges. (2012). Medical School Graduation Questionnaire: 2012 All Schools Summary Report. In Association of American Medical Colleges. Retrieved 4 Aug. 2012 from https://www.aamc.org/download/300448/data/2012gqallschoolssummaryreport.pdf

 

Barnett, S. H., Kaiser, S., Morgan, L. K., Sullivant, J., Siu, A., Rose, D., Rico, M., Smith, L., Schechter, C., Miller, M., Stagnaro-Green, A. (2000). An integrated program for evidence-based medicine in medical school. Mount Sinai Journal of Medicine, 67(2), 163-168.

 

Barnett, S. H., Smith, L. G., & Swartz, M. H. (1999). Teaching evidence-based medicine skills to medical students and residents. International Journal of Dermatology, 38(12), 893-894. doi:10.1046/j.1365-4362.1999.00844.x

 

Belfield, C., Thomas, H., Bullock, A., Eynon, R., & Wall, D. (2001). Measuring effectiveness for best evidence medical education: A discussion. Medical Teacher, 23(2), 164-170. doi:10.1080/0142150020031084

 

Bordley, D. R., Fagan, M., & Theige, D. (1997). Evidence-based medicine: A powerful educational tool for clerkship education. The American Journal of Medicine, 102(5), 427-432.

 

Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research. Boston, MA: Houghton Mifflin Company.

 

Cayley, W. E., Jr. (2005). Evidence-based medicine for medical students: Introducing EBM in a primary care rotation. Wisconsin Medical Journal, 104(3), 34-37. Retrieved 18 Jul. 2012 from http://www.wisconsinmedicalsociety.org/_WMS/publications/wmj/issues/wmj_v104n3/Cayley.pdf

 

Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008). Internet-based learning in the health professions: A meta-analysis. Journal of American Medical Association, 300(10), 1181-1196. doi:10.1001/jama.300.10.1181

 

Davis, J., Crabb, S., Rogers, E., Zamora, J., & Khan, K. (2008). Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: A randomized controlled trial. Medical Teacher, 30(3), 302-307. doi:10.1080/01421590701784349

 

Del Mar, C., Glasziou, P., & Mayer, D. (2004). Teaching evidence based medicine. British Medical Journal, 329(7473), 990. doi:10.1136/bmj.329.7473.989

 

Dickersin, K., Straus, S. E., & Bero, L. A. (2007). Evidence based medicine: Increasing, not dictating, choice. British Medical Journal, 334(Suppl 1), s10. doi:10.1136/bmj.39062.639444.94

 

Dorsch, J. L., Aiyer, M. K., & Meyer, L. E. (2004). Impact of an evidence-based medicine curriculum on medical students' attitudes and skills. Journal of the Medical Library Association, 92(4), 397-406. Retrieved 18 Jul. 2012 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC521510/pdf/i0025-7338-092-04-0397.pdf

 

Finkel, M. L., Brown, H. A., Gerber, L. M., & Supino, P. G. (2003). Teaching evidence-based medicine to medical students. Medical Teacher, 25(2), 202-204. doi:10.1080/0142159031000092634

 

Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston: Pearson/Allyn & Bacon.

 

Ghali, W. A., Saitz, R., Eskew, A. H., Gupta, M., Quan, H., & Hershman, W. Y. (2000). Successful teaching in evidence-based medicine. Medical Education, 34(1), 18-22. doi:10.1046/j.1365-2923.2000.00402.x

 

Glasziou, P., Burls, A., & Gilbert, R. (2008). Evidence based medicine and the medical curriculum. British Medical Journal, 337, a1253. doi:10.1136/bmj.a1253

 

Green, M. L. (2000). Evidence-based medicine training in internal medicine residency programs: A national survey. Journal of General Internal Medicine, 15(2), 129-133. Retrieved 17 Jul. 2012 from http://www.springerlink.com/content/r351055rj56h1130/fulltext.pdf

 

Gustafson, K. L., & Branch, R. M. (2007). What Is Instructional Design? In R. A. Reiser & J. V. Dempsey (Eds.), Trends and Issues in Instructional Design and Technology (2nd ed., pp. 10-16). Upper Saddle River, NJ: Pearson/Merrill Prentice Hall.

 

Guyatt, G., & Rennie, D. (Eds.) (2002). Users' guides to the medical literature: Essentials of evidence-based clinical practice. Chicago, IL: American Medical Association.

 

Harden, R. M. (1986). Approaches to research in medical education. Medical Education, 20(6), 521-531. doi:10.1111/j.1365-2923.1986.tb01394.x

 

Hatala, R., & Guyatt, G. (2002). Evaluating the teaching of evidence-based medicine. JAMA: Journal of the American Medical Association, 288(9), 1110-1112. doi:10.1001/jama.288.9.1110

 

Holloway, R., Nesbit, K., Bordley, D., & Noyes, K. (2004). Teaching and evaluating first and second year medical students' practice of evidence-based medicine. Medical Education, 38(8), 868-878. doi:10.1111/j.1365-2929.2004.01817.x

 

Kern, D. E., Thomas, P. A., & Hughes, M. T. (2009). Curriculum development for medical education: A six-step approach (2nd ed.). Baltimore, MD: Johns Hopkins University Press.

 

Khan, K. S., & Coomarasamy, A. (2006). A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Medical Education, 6, 59. doi:10.1186/1472-6920-6-59

 

Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.

 

Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, 50(1), 54-59.

 

Liaison Committee on Medical Education. (2012). Functions and structure of a medical school: Standards for accreditation of medical education programs leading to the M.D. Degree. In Liaison Committee on Medicual Education. Retrieved 4 Aug. 2012 from http://www.lcme.org/functions.pdf

 

Mahoney, J. F., Cox, M., Gwyther, R. E., O'Dell, D. V., Paulman, P. M., & Kowlowitz, V. (2004). Evidence-based and population-based medicine: National implementation under the UME-21 project. Family Medicine, 36(Jan. Suppl.), S31-35. Retrieved 17 Jul. 2012 from http://www.stfm.org/fmhub/fm2004/January/JohnS31.pdf

 

McKibbon, K. A., Walker-Dilks, C., Haynes, R. B., & Wilczynski, N. (1995). Beyond ACP Journal Club: How to harness MEDLINE for prognosis problems. ACP Journal Club, 123(1), A12-A14.

 

McKibbon, K. A., & Walker-Dilks, C. J. (1994a). Beyond ACP Journal Club: How to harness MEDLINE for diagnostic problems. ACP Journal Club, 121(Suppl. 2), A10-12.

 

McKibbon, K. A., & Walker-Dilks, C. J. (1994b). Beyond ACP Journal Club: How to harness MEDLINE for therapy problems. ACP Journal Club, 121(Suppl. 1), A10-12.

 

Melnyk, B. M. (2006). Calling all educators to teach and model evidence-based practice in academic settings. Worldviews on Evidence-Based Nursing, 3(3), 93-94. doi:10.1111/j.1741-6787.2006.00061.x

 

Mi, M., Moseley, J. L., & Green, M. L. (2012). An instrument to characterize the environment for residents' evidence-based medicine learning and practice. Family Medicine, 44(2), 98-104. Retrieved 17 Jul. 2012 from http://www.stfm.org/fmhub/fm2012/February/Misa98.pdf

 

Montori, V. M., & Guyatt, G. H. (2008). Progress in evidence-based medicine. Journal of American Medical Association, 300(15), 1814-1816. doi:10.1001/jama.300.15.1814

 

Nicholson, L. J., Warde, C. M., & Boker, J. R. (2007). Faculty training in evidence-based medicine: Improving evidence acquisition and critical appraisal. Journal of Continuing Education in the Health Professions, 27(1), 28-33. doi:10.1002/chp.090

 

Nieman, L. Z., Cheng, L., & Foxhall, L. E. (2009). Teaching first-year medical students to apply evidence-based practices to patient care. Family Medicine, 41(5), 332-336. Retrieved 17 Jul. 2012 from http://www.stfm.org/fmhub/fm2009/May/Linda332.pdf

 

O'Connell, M. T., & Pascoe, J. M. (2004). Undergraduate medical education for the 21st century: leadership and teamwork. Family Medicine, 36 (Suppl. January), S51-56. Retrieved 4 Aug. 2012 from http://www.stfm.org/fmhub/fm2004/january/marks51.pdf

 

Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidenced-based medicine: How to practice and teach EBM. New York, NY: Churchill Livingstone.

 

Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence-based medicine: What it is and what it isn't. British Medical Journal, 312, 71. doi:10.1136/bmj.312.7023.71

 

Sastre, E. A., Denny, J. C., McCoy, J. A., McCoy, A. B., & Spickard, A. (2011). Teaching evidence-based medicine: Impact on students' literature use and inpatient clinical documentation. Medical Teacher, 33(6), e306-312. doi:10.3109/0142159X.2011.565827

 

Schilling, K., Wiecha, J., Polineni, D., & Khalil, S. (2006). An interactive Web-based curriculum on evidence-based medicine: Design and effectiveness. Family Medicine, 38(2), 126-132. Retrieved 17 Jul. 2012 from https://www.stfm.org/fmhub/fm2006/February/Katherine126.pdf

 

Slawson, D. C., & Shaughnessy, A. F. (1999). Teaching information mastery: Creating informed consumers of medical information. Journal of American Board of Family Practice, 12(6), 444-449.

 

Srinivasan, M., Weiner, M., Breitfeld, P. P., Brahmi, F., Dickerson, K. L., & Weiner, G. (2002). Early introduction of an evidence-based medicine course to preclinical medical students. Journal of General Internal Medicine, 17(1), 58-65. Retrieved 17 Jul. from http://www.springerlink.com/content/u42tnw482l38441r/fulltext.pdf

 

Steward, M. G. (2001). Core Competencies. In Accreditation Council for Graduate Medical Education. Retrieved 4 Aug. 2012 from http://www.acgme.org/acwebsite/RRC_280/280_corecomp.asp

 

Straus, S. E., Green, M. L., Bell, D. S., Badgett, R., Davis, D., Gerrity, M., Ortiz, E., Shaneyfelt, T. M., Whalen, C. Mangrulkar, R. (2004). Evaluating the teaching of evidence based medicine: Conceptual framework. British Medical Journal, 329(7473), 1029-1032. doi:10.1136/bmj.329.7473.1029

 

Thomas, P. A., & Cofrancesco, J., Jr. (2001). Introduction of evidence-based medicine into an ambulatory clinical clerkship. Journal of General Internal Medicine, 16(4), 244-249. Retrieved 17 Jul. 2012 from http://www.springerlink.com/content/b7k81391322u64n1/fulltext.pdf

 

Wadland, W. C., Barry, H. C., Farquhar, L., Holzman, C., & White, A. (1999). Training medical students in evidence-based medicine: A community campus approach. Family Medicine, 31(10), 703-708. Retrieved 17 Jul. 2012 from http://stfm.org/fmhub/Fullpdf/NOVDEC99/mse.pdf

 

West, C. P., Jaeger, T. M., & McDonald, F. S. (2011). Extended evaluation of a longitudinal medical school evidence-based medicine curriculum. Journal of General Internal Medicine, 26(6), 611-615. doi:10.1007/s11606-011-1642-8


 


 

Appendix A

Types of Research Design

 

1.       Control group designs with random assignment

         Pretest-posttest control group design

o    Random assignment of research participants to experimental and control groups

o    Administration of a pretest to both groups

o    Implementation of an intervention to the experimental group but not to the control group

o    Administration of a posttest to both groups

         Posttest-only control group design

o    Random assignment of participants to experimental and control groups

o    Implementation of an intervention to the experimental group but not to the control group

o    Administration of a posttest to both groups

2.       Quasi-experimental designs

         Static-group comparison design

o    Participants not randomly assigned to the experimental and control groups

o    Implementation of an intervention to the experimental group but not to the control group

o    A posttest, but no pretest administered to both groups

         Nonequivalent control group design

o    Participants not randomly assigned to the experimental and control groups

o    Implementation of an intervention to the experimental group but not to the control group

o    Both groups taking a pretest and posttest

3.       Single group designs

         One short case study

o    Implementation of an intervention for participants (no control group)

o    Administration of a posttest

         One group pretest-posttest design

o    Administration of a pretest to research participants (no control group)

o    Implementation of an intervention

o    Administration of a posttest

 


 

Appendix B

Summary of EBM Interventions, Outcomes Measured, Evaluation Methods, and Levels of Evaluation

Author (Year)

EBM Interventions

Outcomes Measured

Evaluation Methods

Levels of Evaluation

Wadland et al. (1999)

Year 1 and Year 2:

         First component (8 hours) prior to formal clinical clerkships, focusing on critiquing articles and answering questions about study quality and applicability to patient care

Year 3 and Year 4:

         Second component (12 hours) during clerkships of internal medicine, family practice, and pediatrics, focusing on evaluating guidelines relevant to primary care and developing clinically relevant research proposals

         Computerized literature search

         Understanding of structure of medical

research articles

         Critical analysis of medical research articles

         Evaluation of practice

guidelines and their development

         Development and oral presentation of a research proposal

         AAMC Medical School Graduation Questionnaire

         Course evaluation questions

         Survey at two points of assessment (at graduation and at the end of first-year residency)

1, 2

Barnett et al. (2000)

Year 1 and Year 2:

         Medical informatics integrated into the Library Science and Medical Informatics Course

(7 hours)

         Critical appraisal of articles integrated into the environmental medicine module of the Epidemiology course

         Use of literature to identify unknown laboratory specimens in the Microbiology course

Year 3:

         One hour session during orientation on definition of goals and objectives of curriculum and description of components in each rotation

         EBM clerkships of medicine, pediatrics, psychiatry, surgery, obstetrics and gynecology, neurology, community medicine, geriatrics, with each clerkship teaching one of the McMaster modules such as appraising overview, prognosis, therapy, harm, and diagnostic test articles

Year 4

         EBM added to the Community and Preventive Medicine Course

         Two hour instruction in MEDLINE

         Geriatrics clerskship: questions from Course Director for small-group discussion

         Exposure to medical informatics both prior to and during medical school

         Information retrieval

         Critical reasoning

         Medical Informatics questionnaires

         Fourth-year EBM exercise with four cases: literature search strategy, 5 relevant citations, one paragraph critique of the single most relevant article

 

2

Ghali et al. (2000)

Four 90 minute sessions offered throughout Year 3, covering:

         EBM steps: developing focused clinical questions from patient care problems encountered in students clinical rotations, group discussion of actual clinical scenarios, formulating a specific question, conducting evidence based literature searching for articles, and selecting one article for critical appraisal

         Reading/library behaviours

         Skills and attitudes on issues relating to EBM

         Self-reported skill survey

2

Thomas et al. (2001)

Year 3 and Year 4:

         Instructions in the structure of an EBM report during clerkship orientation

         Ninety minute small group session modeling a five step approach with a clinical case

         Selecting a patient case with a diagnostic or therapeutic dilemma from clerkship experience

         Ninety minute small group session with students searching for the best evidence, submitting report, and presenting findings to peers

         Formulation of questions

         Application of evidence to specific situations

         Written report graded with a 12-item checklist

         Program evaluation including an end-of-clerkship assessment and self-assessment

 

1, 3

Srinivasan et al. (2002)

Short EBM course of eight student contact hours in Year 1:

         Two 1-hour lectures: 1-hour introductory lecture reviewing standard biostatistical concepts and construction of clinical questions and 1-hour introduction of clinical questions

         Three 2-hour small group sessions on evaluation skills frequently used by clinicians: assessment of risks/benefits of therapeutic interventions and diagnostic test, and of causation of harm; discussing a clinical vignette, developing a relevant question, and evaluating a corresponding article

         Supplemental online EBM curriculum: 20 page web-based EBM curriculum and a practice examination

         Student preparation, performance, participation

         Utilization of supplemental web curriculum

         Student/facilitator satisfaction

 

         Small-group attendance and facilitator questionnaire

         Online practice examination and written final examination

         Usage of online supplemental EBM curriculum

         Student satisfaction online questionnaire

         Facilitator satisfaction and experience questionnaire

1, 2

Holloway et al. (2004)

Twenty to 30 student contact hours of EBM instruction integrated into basic science curricula of preclinical years:

         Introduction to EBM during first four weeks of curricula: eight to ten student contact hours in the Mastering Medical Information course: an EBM overview, medical informatics, EBM databases, framing a well built searchable question, and critical appraisal of the literature

         A series of EBM reinforcements during the reminder of Year 1 as part of an ambulatory clerkship course: five student contact hours, including an advanced skills workshop, a patient-centered EBM exercise with individualized feedback, a peer comparison report, and an EBM review

         EBM reinforcement continued in Year 2: a series of seven monthly, 2-hour small group EBM tutorials led by local EBM experts to review clinical cases and practice building clinical questions, searching and appraising the literature

         Generation of PICO questions

         Medline searching

         Critical appraisal skills

         Application of results of appraisal to patients

         Self-assessment

 

         Two test modules evaluating students ability to carry out five EBM steps

         Self-assessment questions

         Student satisfaction survey and written comments

 

1, 2

Dorsch et al. (2004)

Eight 1-hour weekly seminars during a 12 week internal medicine clerkship in Year 3:

         Two sessions: defining EBM, formulating clinical questions based on a standardized case scenario, identifying and reviewing EBM search strategies and resources

         Three sessions: developing critical appraisal skills for therapy, diagnosis, and meta-analysis articles

         Three sessions: presenting critically appraised topics (CAT) based on typical patient care problems encountered during clerkship to consolidate and demonstrate learned skills

         Performance in applying EBM skills

         Pre- and post-clerkship survey

         Pre- and post-test skills assessment

2

Cayley (2005)

Year 3:

         Six session curriculum combining exploration of basic principles of EBM with application of these principles to real-life cases from students clinical experience

         Understanding the use of EBM

         Questionnaire developed by Slawson and Shaughnessy (1999)

2

Schilling et al. (2006)

Year 3:

Week 1:

         Two course integrated, Web-based learning tutorials on MEDLINE and EBM database information retrieval skills, requiring 40-60 minutes to complete

Week 4

         Introduction to the National Guideline Clearinghouse (www.guideline.gov/) from the US Department of Health and Human Services, Agency for Healthcare Research and Quality (AHRQ)

Week 5

         Learning how to calculate and interpret NNT (number need to treat) statistic from Michigan State University Department of Family Practices tutorial Introduction to Information Mastery (www.poems.msu.edu/InfoMastery/)

         Information retrieval skills

         EBM practice skills

         Perception about clerkship experience

         MEDLINE literature search strategies

         Articles identified by students as providing the best evidence to address a clinical case

         Post-clerkship NNT test (calculating a NNT from a hypothetical clinical trial)

         Post-clerkship survey

 

1, 2

Nieman et al. (2009)

Year 1:

         Mandatory 2-hour orientation workshop: asking a clinical question about a sample case, accessing databases (e.g., DynaMed), and discussing appraisals of relevant medical literature

         Elective family medicine preceptorship: completing EBM case summaries for four patients whom students would select with the agreement of their preceptor

         Documentation of EBM process

         Self-efficacy

         Level of EBM learning

         Preceptors attitudes toward using an EBM project as a focus of their feedback

 

         Four PICO case summaries analyzed with Blooms taxonomies of cognitive and affective domains

         Student self-efficacy questionnaire

         Preceptor Questionnaire

2

Aronoff et al. (2010)

Year 3:

Two parts of an EBM course:

         Part 1 (18 weeks to complete): six online modules, each of which contained didactic material and a focused practicum/assignment

         Part 2 (24 weeks to complete): formulating a clinical question generated by a patient seen during each of four clerkship rotations; completing four evidence summaries using Critically Appraised Topic (CAT) format developed by Sackett et al. (1997)

Competence in EBM

         A practicum/ assignment each student completed and submitted to an assigned online faculty mentor for review and feedback

         Completing a CAT form

         Fresno Test administered before and after the EBM course

 

 

2, 3

West et al. (2011)

End of Year 2:

         Short course of 22 contact hours over a 2-week period: didactic and small-group sessions on full range of EBM skills adapted from Users Guide to the Medical Literature

Year 3:

         EBM integrated with clinical experiences in each clinical rotation with each student generating a clinical question from a patient encounter, searching for an article addressing the question, critically appraising the article, and producing a brief summary of the evidence and its application to the patient from whom the clinical question arose

         EBM skills and knowledge

         Self assessment of the importance of EBM for medical education and clinical practice

         Berlin Questionnaire

         Fresno Test

         Self-rated EBM knowledge and assessment of importance of EBM for medical education

 

2

Sastre et al. (2011)

Year 3:

         A single, physician-led, hands on 3-hour workshop teaching clinical question formation, locating and using pre-appraised resources, discussing strengths and weakness of available pre-appraised resources

 

 

         Attitudes and knowledge about literature searching, prior use of EBM resources, and knowledge of clinical question formation and searching techniques

         Impact on patient care

         Pre- and post-surveys

         Inpatient admission notes as a surrogate marker of impact on patient care

         Computer log data of students searching of various EBM resources using hyperlinks and search tools integrated within an electronic medical resource system

1, 2, 3

 

 

 

 

 

 




Evidence Based Library and Information Practice (EBLIP) | EBLIP on Twitter