|ORIGINAL RESEARCH PAPER
|Year : 2011 | Volume
| Issue : 3 | Page : 573
Effectiveness of educational interventions in improving clinical competence of residents in an internal medicine residency program in Pakistan
M Tariq1, NA Syed1, A Motiwala1, W Jafri1, K Hameed2, N Islam1, M Riaz1, S Awan1, J Akhter1, J Talati1
1 Aga Khan University, Karachi, Pakistan
2 Ziauddin University, Karachi, Pakistan
|Date of Submission||05-Nov-2010|
|Date of Acceptance||30-Sep-2011|
|Date of Web Publication||04-Dec-2011|
Aga Khan University, Karachi
Source of Support: None, Conflict of Interest: None
Introduction: Medical education is a continuously evolving field. Training institutes and programs should have a process in place to gather continuous feedback and then make appropriate modifications in order to provide education and training effectively. Our study aimed to assess the effect of a quality improvement cycle approach in using various educational interventions within a residency. Effects were measured on the key educational outcomes of residents' medical knowledge, skills and professional attitudes using results of postgraduate examination with both written and clinical skills components.
Methodology: A number of educational interventions were implemented which included changes in work hours with increased time for self-study, new educational activities including a Residents' Hour, a Residents' Slide Session, Grand Rounds and Journal Clubs, Clinico-pathological conferences, and a two- week postgraduate course for senior residents. Newer and improved assessment tools were also implemented, including an annual in-training mock exam based on the format of the postgraduate examination. Pass rates in postgraduate examinations (Fellow of College of Physicians and Surgeons exam and Member of Royal College of Physicians exam) were compared before and after the interventions to assess the effectiveness of the interventions.
Results: The first group of residents after introduction of the educational interventions completed residency training in 2001. Postgraduate exam pass rates (sometimes after two or more attempts) were 59.2% (42 of 71 graduates) before 2001and 86.4% (38 of 44 graduates after 2001 (p=0.002). The number of candidates passing the examinations in either their first or second attempts before 2001 was 17 of 42 (40.5%), which increased to 33 of 38 (86.8%) after 2001 (p=< 0.001).
Conclusions: Our study describes a number of interventions that were successful in bringing about an improvement in the performance of our residents. These can serve as a guide for postgraduate training programs, particularly those of Internal Medicine, in implementing strategies to strengthen training and enhance the performance of trainees.
Keywords: Curriculum improvement, educational interventions, internal Medicine residency program, postgraduate examinations, postgraduate training, quality improvement cycle
|How to cite this article:|
Tariq M, Syed N A, Motiwala A, Jafri W, Hameed K, Islam N, Riaz M, Awan S, Akhter J, Talati J. Effectiveness of educational interventions in improving clinical competence of residents in an internal medicine residency program in Pakistan. Educ Health 2011;24:573
|How to cite this URL:|
Tariq M, Syed N A, Motiwala A, Jafri W, Hameed K, Islam N, Riaz M, Awan S, Akhter J, Talati J. Effectiveness of educational interventions in improving clinical competence of residents in an internal medicine residency program in Pakistan. Educ Health [serial online] 2011 [cited 2020 May 27];24:573. Available from: http://www.educationforhealth.net/text.asp?2011/24/3/573/101420
Medical education at both undergraduate and postgraduate levels is a continuously evolving field. Every postgraduate training program needs a system of continuous feedback, analysis and modification in order to keep pace with the changing trends of education and health care. Ideally, this should be a continuous cycle of analysis of trainee performance, identifying factors affecting any deficiencies in performance, developing and implementing strategies to address the identified challenges, and assessing the improvement brought about by the strategies. Optimizing training within the context of the current local health system is important not only for trainees but also for patients and for effective health care delivery.
Since 1986, Aga Khan University (AKU) has been a pioneer institution in establishing structured residency programs in Pakistan. One hundred and fifteen candidates completed postgraduate residency training from our Internal Medicine Residency Training Program from 1989 to 2007. Goals of the structured, four-year residency are to graduate physicians with adequate medical knowledge, clinical skills and an appropriate professional attitude to enable them to function as academic professionals in their careers.
Educational activities prior to the 1998 interventions
Prior to 1998, the curriculum of the Internal Medicine Residency included educational opportunities that included designated ward tutorials by faculty, weekly medicine grand rounds, morning report, monthly mortality and morbidity meetings and quarterly clinical practice guidelines. The residents received monthly in-training evaluations and had annual in-service examinations based on multiple-choice questions (MCQ). At the end of their training, residents were required to take exit examinations, mainly the FCPS-II (Fellow of College of Physicians and Surgeons) exam; others took the MRCP-II (Member of Royal College of Physicians) exam.
The past rate in the exit examinations prior to 1998 had been noted to be low. Many program graduates required several attempts to pass. Our residents also complained of inadequate self-study time during the program as well as a general lack of teaching activities and learning opportunities.
Ideally the identification of problems at any level in a residency program, whether they are related to training, assessment, results or administrative issues, should lead to an in-depth analysis into potential causes. These root causes may be identified through a special taskforce, questionnaires or meetings and discussions with the individuals involved1,2. These can subsequently lead to the formulation of a set of solutions which might take the form of guidelines, interventions or curriculum redesign to tackle the identified problems3,4. Follow-up analysis to assess the effectiveness of the interventions ideally follows.
We interpreted poor performance on the exit examinations by our residents as an indicator of their inadequate medical knowledge and skills. Using a quality improvement cycle influenced by the 'Innovations in Professional Education' described by Boyatzis, Cowen and Kolb5, we sought to determine the causes of their inadequate clinical competence and implement changes in our educational strategy to improve their performance. (Figure I).
Figure I: The process of analysis, implementation and impact of educational intervention strategies
In this study, we aimed to demonstrate the effect of using a quality improvement cycle approach on the key educational outcomes of medical knowledge and skills, using postgraduate (FCPS and MRCP) exam results as an indicator of resident graduates’ performance.
The FCPS exam, conducted by CPSP (College of Physicians and Surgeons, Pakistan), has two components6. The first written part assesses core depth of knowledge, and only candidates successful in the written part are short listed for the clinical and oral exam. The clinical and oral part includes the TOACS (Task Oriented Assessment of Clinical Skills), which comprises 15 clinical skills stations and short and long cases, both requiring detailed clinical examination of actual patients. Generally, FCPS has been very competitive, and it is not uncommon for candidates to need to appear more than once before passing. Once they pass, doctors are considered qualified as a specialist in their field.
The MRCP, a postgraduate exam conducted by RCP (Royal College of Physicians, United Kingdom), is also made up of two parts. The theory section comprises the written examination. The clinical part, termed PACES (Practical Assessment of Clinical Examination Skills), is a clinical examination in which candidates rotate through five 20-minute stations, seeing a range of real and simulated patients that they are required to interview, examine and then discuss management options7. This part is meant to assess in depth clinical knowledge and skills, as well as communication skills and ethics. These exit exams are intended to assess clinical competencies, specifically knowledge, clinical skills and professional attitude gained during the postgraduate training.
Poor pass rates in these exams were taken as evidence of resident graduates’ inadequate medical knowledge and skills. Principally guided by the program director and the Department Residency Committee, a need was felt to identify and address the causes of poor results. A series of open-ended discussions with residents and faculty were then held to pinpoint specific areas of weakness and solicit suggestions for ways to address the identified problems. Participants of the discussion group included currently enrolled senior residents, recent graduates, senior faculty and departmental leadership. The discussions focused around the question: 'What in your opinion are the causes of the low pass rate of our residents and what can be done about it?' A number of issues were raised by the participants during these discussions, including:
- Inadequate self-study time due to pressing clinical demands.
- The need for more organized interactive teaching activities throughout the training period.
- Residents being unable to attend academic activities due to conflicting demands on their time.
- Lack of uniformity in the sub-specialty rotations; some residents were unable to gain adequate exposure in certain sub-specialties.
- Although the clinical work took up most of their time, residents indicated there were not sufficient opportunities to engage in discussions on clinical cases.
- There were few opportunities for residents to interact with their program director.
- Finally, the need was identified to have more formative evaluation opportunities as well as in-service exams and assessments.
A number of interventions were subsequently implemented to enhance the educational quality of the program based on the challenges identified. These included structuring of rotations during training to provide uniform and adequate exposure to medicine and subspecialty rotations. A greater amount of time for self-study was provided to residents by limiting the number of night calls in a few rotations and shortening their hours in ambulatory clinics. As the residents had requested, a greater amount of interaction was ensured between the program director and the residents, in the form of the regular presence of the program director in morning reports and a weekly Residents’ Hour.
A series of new teaching modalities were introduced to provide additional learning opportunities for the residents. These included fortnightly 'Residents’ Hour' in which the residents presented and discussed 10 to 12 brief case histories under the supervision of senior faculty; 'Residents’ Slide Session,' which was an interactive picture slides-based quiz; and weekly lectures by faculty on designated topics. Monthly 'Residents’ Grand Rounds' were introduced to allow residents to enhance their presentation skills and provide additional opportunities for learning, and fortnightly ‘Journal Clubs’ ensured that the residents remained updated with current literature. Monthly Clinico-pathological conferences were also introduced. A comprehensive annual, two-week postgraduate course was implemented for senior residents which included both theoretical and patient-based teaching, followed by an assessment exam. An effort was made to integrate a more interactive, problem based approach towards teaching, rather than the conventional didactic teaching.
In addition, a number of new assessment tools were implemented. These included an annual in-training mock exam based on the format of the postgraduate examination, as well as annual MCQ-based tests and OSCE (objective structured clinical examination). The number of chief residents was increased to two, since it was becoming increasingly difficult for a single chief resident to handle all the assigned academic, clinical and administrative responsibilities. FCPS Part I exam was made a requirement for entry into the residency program. We also introduced annual 'Residents’ Appreciation Evening' to reward outstanding performance and to motivate the trainees. All of these interventions were implemented over a two-year period, from 1998 to 20008.
We compared the pass rates on the FCPS and MRCP exit examinations of graduates before and after the interventions to determine their effectiveness. The study was approved by the Ethical Review Committee (ERC) at the Aga Khan University Hospital.
We performed descriptive analysis for demographic variables; results were reported as numbers with percentages for quantitative variables, and 95% Confidence Intervals (CI) for odds ratio were estimated to assess the strengths of association. We assessed statistical differences in proportions using Pearson Chi-square test. P-value < 0.05 was considered as statistically significant; all p-values were two sided. SPSS 15.0 (SPSS Inc., Chicago: IL, USA) was used for data entry and analysis.
A total of 115 residents graduated from the AKU Internal Medicine residency program between 1989 and 2007 and were candidates for postgraduate examination. Of these, 80 (69.6%) candidates passed their exit examinations. Most of the remaining either proceeded abroad for training, or pursued private practice. Table I shows a year-by-year distribution of the number of graduating residents, the number that passed each of the postgraduate exams, and the number of attempts required to pass the exam.
Following our interventions introduced in 1998, the first group of post-intervention residents graduated in 2001. Thus the intervention group was comprised of residents who graduated in and after 2001, who were 44 in number. The pre-intervention group was comprised of the remaining 71 residents who had graduated before 2001.
Table I: Yearly comparison of postgraduate exam success and number of attempts in passing the exam from 1989-2007
Postgraduate exam success in any number of attempts was 59.2% (42 of 71 graduates) before 2001, compared to 86.4% (38 of 44 graduates) after 2001 (p=0.002) (Table II). The number of candidates passing the examination in either first or second attempt after 2001 was 33 out of 38 (86.8%), compared to 17 out of 42 (40.5%) among graduates prior to 2001 (p≤0.001). The number of graduates passing the examination in three or more attempts after 2001 was 5 out of 38 (13.2%). On the other hand, 25 out of 42 (59.52%) candidates prior to 2001 required three or more attempts to pass the exam (p= < 0.001).
Table II: Comparative analysis among candidates before and after Educational Intervention (n=115)
Following the intervention, from 2001 through 2007 there were only five graduates who did not pass the exam within two tries, three candidates in 2001 and two candidates in 2004. We also analyzed the changes in pass rates and number of attempts required to pass before and after interventions specifically for the FCPS exam, which was the exam for a majority of our candidates, and we found the difference to be statistically significant. The results are depicted in Table III. Candidates appearing for MRCP made up only 11.5% of our sample, and so they were not separately analyzed.
Table III: Comparative analysis among candidates specifically appearing for FCPS-II before and after interventions. (n=104)
We used a process of curriculum improvement to assess and revise our Internal Medicine residency curriculum. Based on identified problems, we made a number of educational interventions. Such a coordinated institutional approach can provide the synergy essential for a large scale change, preferably at an institutional level, while fragmented approaches that focus on single areas are often frustrated by lack of interest or a lack of understanding among all individuals involved9. Hectic schedules and a lack of self-study time was an issue pointed out repeatedly by our residents and is a common issue, particularly in busy training programs. ACGME recommends residents’ work hours be limited to less than 80 hours per week averaged over 4 weeks10. Studies have shown that extended work hours increase the risk for medical errors11,12, personal injury13, and may even jeopardize patient safety14,15. As part of our improvement process, we have now implemented a work-hour limit for our residents of 80 hours per week, averaged over a three-month period.
We also restructured our training so as to ensure adequate exposure to all major subspecialties. Authors in the past have observed that medical residents are often not sufficiently exposed to Internal Medicine career options, leading them to make uninformed career choices16,17. We introduced a number of new educational activities and tried to base most of our teaching activities on interactive learning as it enhances the learning experience and maintains the interest of the learners. This is akin to a universal trend of increasing interest in interactive teaching in order to enhance learner participation and learning. Evidence shows that a shift from passive teaching at medical conferences to more interactive methods significantly improves physician learning18. In addition, we implemented different forms of summative and formative assessment tools to allow residents effective feedback regarding their performance. Effective and timely assessment and feedback plays an integral role in helping physicians identify and respond to their own learning needs, as well as find and generate new knowledge, and improve overall performance19.
Following implementation of these interventions, the proportion of residents achieving satisfactory knowledge competency as reflected in performance on several external examinations significantly improved. Implementation of teaching interventions and restructuring in a medical setup is itself a laborious task. Literature shows us a number of situations where professionals have attempted to implement a variety of changes in their respective residency curricula. Many of them were successful in bringing about improvements or enhancing satisfaction among their trainees, but a number failed to show any benefit. In one of those studies, Issenberg et al. demonstrated significant improvement in residents’ bedside cardiology skills after use of a review course in cardiology bedside skills20. Another study on 46 residents showed reduction in bleeding rates among the patients that received guideline based consultation21. In yet another, an improvement in residents’ attitudes towards patient care was seen after two months of psychiatry rotation22. However, Fischer and colleagues failed to show an improvement in the knowledge of residents after use of four weekly case-based, small group discussions on end of life care teaching material23. Lindberg et al. also failed to show improvement after use of four week intensive inpatient geriatric experience24. It is encouraging to note that a number of studies have actually shown that interventions can bring about both better education and improved patient care. However, there were no studies to our knowledge evaluating an internal medicine program’s ‘‘core’’ inpatient curriculum. In addition, very few of the studies performed direct assessment of resident competencies, and instead preferred use their satisfaction and self-assessment as a tool.
There can be a number of factors that make this process of change in education curriculum a difficult one, particularly in the setting of postgraduate medical training. Teachers often carry strong views that they have tested over years. The increasing workload in the context of limited work hours has led to excessive commitment to service over education for both faculty and residents3,4,25.
A number of studies have attempted to describe innovations and reforms that help shape an efficient redesigning of internal medicine curriculum, keeping in view the current challenges3,4,26,27. They provide a number of recommendations, including faculty development programs, funding and salary related reforms, and most importantly reforming of curriculum. Most of the recommended reforms focus on ensuring quality and relevance of the educational experience, satisfaction of the trainees, their adequate supervision, adapting to the expanding system of health care delivery and expanding body of knowledge, coupled with effective patient care4. Some view the first year of residency as an experience key to both education and future career decisions, and therefore an important stage for curriculum redesign26.
Being based on a single institution, external validity is a limitation of our study. We have combined the results of two different postgraduate exit examinations which may somewhat differ in their assessment of competencies and pass rates. The candidates appearing in MRCP form only a small fraction of the total candidates, and we also analyzed and compared the results of FCPS separately (Table III), which stand significant on their own. There is also a possibility that the pass rates of the examinations may have generally improved over the years due to factors apart from the curricular interventions, since no data regarding trend in pass rates is available. However, it is generally agreed upon that the difficulty level and pass rates in these exams have been fairly constant for the past many years.
Since we do not have detailed data on the residents’ medical school performances and backgrounds prior to residency, we cannot rule out the possibility that differences in quality or skills of residents could have accounted for the improved outcomes. The program duration was also increased from three years to four years during the study period in accordance with CPSP requirements, and this may have influenced the measured skills and competency of the residents.
Our study describes a number of educational interventions implemented at our institute that were successful in bringing about an improvement in knowledge, skills and attitudes in our residents. There is a growing consensus globally that the structure of Internal Medicine residency urgently needs redesign. Our study helps provide some important steps that may assist in creating such a redesign. The interventions we have highlighted and tested can potentially be applied to other departments at our institute as well as other residency programs the world over, although some variations may be essential based on institutional structure and preferences. Future research aimed at testing the effectiveness of these and similar interventions at other institutions can provide more concrete evidence of their effectiveness, and set a trend of education and curriculum improvement research in Pakistan which has, to this date, been almost non-existent in our country.
We wish to thank Dr William Burdick, (ECFMGs, Assistant Vice President of assessment services and FAIMERs Associate Vice president for Education and Co-Director) for reviewing the manuscript and providing valuable suggestions. We would also like to thank the faculty of the Department of Medicine at AKU for teaching of our residents, and in particular the Departmental Residency Committee for overseeing the residency program.
1. Holmboe ES, Bowen JL, Green M, Gregg J, DiFrancesco L, Reynolds E, Alguire P, Battinelli D, Lucey C, Duffy D. Reforming internal medicine residency training. A report from the Society of General Internal Medicine's task force for residency reform. Journal of general internal medicine. 2005; 20(12):1165-1172.
2. Bravata DM, Huot SJ, Abernathy HS, Skeff KM. The development and implementation of a curriculum to improve clinicians' self-directed learning skills: a pilot project. BMC medical education. 2003; 3:7.
3. Fitzgibbons JP, Bordley DR, Berkowitz LR, Miller BW, Henderson MC. Redesigning residency education in internal medicine: A position paper from the Association of Program Directors in Internal Medicine. Annals of internal medicine. 2006; 144(12):920-926.
4. Weinberger SE, Smith LG, Collier VU. Redesigning training for internal medicine. Annals of internal medicine. 2006; 144(12):927-932.
5. Boyatzis RE Cowen SS, Kolb DA. Innovation in professional education: Steps on a journey from teaching to learning. San Francisco: Jossey-Bass; 1995.
6. CPSP. College of Physicians and Surgeons Pakistan. 2011. Retrieved 2011; Available from: http://www.cpsp.edu.pk/
7. McManus IC, Elder AT, de Champlain A, Dacre JE, Mollon J, Chis L. Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC medicine. 2008; 6:5.
8. AKU. Aga Khan University - Internal Medicine Residency Programme. 2011. Retrieved 2011. Available from: http://www.aku.edu/collegesschoolsandinstitutes/medicine/pakistan/programmes/postgraduate/Pages/residencyinternalmedicine.aspx
9. Kolb A, Kolb D. Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education. Academy of Management Learning & Education. 2005; 4(2):193-212.
10. Harrison R, Allen E. Teaching internal medicine residents in the new era. Inpatient attending with duty-hour regulations. Journal of general internal medicine. 2006; 21(5):447-452.
11. Landrigan CP, Rothschild JM, Cronin JW, Kaushal R, Burdick E, Katz JT, Lilly CM, Stone PH, Lockley SW, Bates DW, Czeisler CA.. Effect of reducing interns' work hours on serious medical errors in intensive care units. The New England journal of medicine. 2004; 351(18):1838-1848.
12. Lockley SW, Cronin JW, Evans EE, Cade BE, Lee CJ, Landrigan CP, Rothschild JM, Katz JT, Lilly CM, Stone PH, Aeschbach D, Czeisler CA. Effect of reducing interns' weekly work hours on sleep and attentional failures. The New England journal of medicine. 2004; 351(18):1829-1837.
13. Barger LK, Cade BE, Ayas NT, Cronin JW, Rosner B, Speizer FE, Czeisler CA. Extended work shifts and the risk of motor vehicle crashes among interns. The New England journal of medicine. 2005; 352(2):125-134.
14. Philibert I, Friedmann P, Williams WT. New requirements for resident duty hours. The journal of the American Medical Association. 2002; 288(9):1112-1114.
15. Wallack MK, Chao L. Resident work hours: the evolution of a revolution. Archives of surgery. 2001;136(12):1426-1431; discussion 32.
16. Garibaldi RA, Popkave C, Bylsma W. Career plans for trainees in internal medicine residency programs. Academic medicine. 2005; 80(5):507-512.
17. NRMP. National Resident Matching Program Results and Data 1975-2005. 2005.
18. Ting J. A shift from passive teaching at medical conferences to more interactive methods improves physician learning. Medical teacher. 2007; 29(2-3):285.
19. Epstein RM. Assessment in medical education. The New England journal of medicine. 2007; 356(4):387-396.
20. Issenberg SB, McGaghie WC, Gordon DL, Symes S, Petrusa ER, Hart IR, Harden RM. Effectiveness of a cardiology review course for internal medicine residents using simulation technology and deliberate practice. Teaching and learning in medicine. 2002; 14(4):223-228.
21. Landefeld CS, Anderson PA. Guideline-based consultation to prevent anticoagulant-related bleeding. A randomized, controlled trial in a teaching hospital. Annals of internal medicine. 1992; 116(10):829-837.
22. Juergens SM, Rome JD, Ilstrup DM. Effect of inpatient psychiatry training on internal medicine residents. Results of a survey. General hospital psychiatry. 1990; 12(2):124-128.
23. Fischer SM, Gozansky WS, Kutner JS, Chomiak A, Kramer A. Palliative care education: an intervention to improve medical residents' knowledge and attitudes. Journal of palliative medicine. 2003; 6(3):391-399.
24. Lindberg MC, Sullivan GM. Effects of an inpatient geriatrics rotation on internal medicine residents' knowledge and attitudes. Journal of general internal medicine. 1996; 11(7):397-400.
25. Schroeder SA, Sox HC. Internal medicine training: putt or get off the green. Annals of internal medicine. 2006; 144(12):938-939.
26. Gustin W, Batra R, Amin A, Rucker L. Education first: reforming the first-year curriculum of the internal medicine residency. Academic medicine. 2009; 84(3):368-373.
27. Meyers FJ, Weinberger SE, Fitzgibbons JP, Glassroth J, Duffy FD, Clayton CP. Redesigning residency training in internal medicine: the consensus report of the Alliance for Academic Internal Medicine Education Redesign Task Force. Academic medicine. 2007; 82(12):1211-1219.