Print this page Email this page Users Online: 204 | Click here to view old website
Home About us Editorial Board Search Current Issue Archives Submit Article Author Instructions Contact Us Login 


 
 Table of Contents  
ORIGINAL RESEARCH ARTICLE
Year : 2016  |  Volume : 29  |  Issue : 2  |  Page : 68-74

Developing and pilot testing of a tool for "clinicosocial case study" assessment of community medicine residents


1 Department of Community Medicine, Pramukhswami Medical College, Karamsad, Gujarat, India
2 Central Research Services and Member of Human Research Ethics Committee, Charutar Arogya Mandal, Karamsad, Gujarat, India

Date of Web Publication19-Aug-2016

Correspondence Address:
Manisha Gohel
6, Punit Bunglows, Near La Casaa in Hotel, Ganesh Chokdi, Anand - 388 001, Gujarat
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/1357-6283.188684

  Abstract 

Background: Practical and clinical skills teaching should constitute a core part of the postgraduate curriculum of Community Medicine. The clinicosocial case study is a method to enhance learners' skills but there is no generally accepted organized system of formative assessment and structured feedback to guide students. A new tool based on the principles of mini-Clinical Evaluation Exercise (mini CEX) was developed and pilot tested as a 'clinicosocial case study' assessment of community medicine residents with feedback as a core component. Methods: Ten core domains of clinicosocial skills were identified after reviewing the relevant literature and input from local experts in community medicine and medical education. We pilot tested the tool with eight faculty members to assess five residents during clinicosocial case presentations on a variety of topics. Kappa statistic and Bland Altman plots were used to assess agreement between faculty members' average assessment scores. Cronbach's alpha was used to test the internal consistency with faculty members as domains. Results: All 95% confidence limits using the Bland-Altman method were within the predetermined limit of 2 points. The overall Kappa between two faculty members was fair ranging from 0.2 to 0.3. Qualitative feedback revealed that both faculty and residents were enthusiastic about the process but faculty suggested further standardization, while residents suggested streamlining of the process. Discussion: This new assessment tool is available for objective and unbiased assessment of residents through 'clinicosocial case study,' which enriches learning through comprehensive feedback. Further validation in different settings is needed.

Keywords: Assessment tool, clinicosocial case, mini-CEX, residents


How to cite this article:
Gohel M, Singh US, Bhanderi D, Phatak A. Developing and pilot testing of a tool for "clinicosocial case study" assessment of community medicine residents. Educ Health 2016;29:68-74

How to cite this URL:
Gohel M, Singh US, Bhanderi D, Phatak A. Developing and pilot testing of a tool for "clinicosocial case study" assessment of community medicine residents. Educ Health [serial online] 2016 [cited 2017 Feb 25];29:68-74. Available from: http://www.educationforhealth.net/text.asp?2016/29/2/68/188684


  Background Top


The development of the Indian healthcare system was guided by the Bhore committee report of 1946. This report emphasized healthcare for all, a preventive rather than curative approach, a focus on healthcare within rural areas and the training of 'social physicians.' [1] To include socio-cultural orientation in medical education, departments of Community Medicine-formerly known as 'Preventive and Social Medicine'-were established and have been updated from time to time since. [2] With its inclusive approach, in the early years of national independence the Indian healthcare system was able to develop pertinent healthcare delivery models without copying western curative models. The Indian healthcare system, at least in principle, was well ahead of the Alma Ata declaration for equity and focus on primary health care. But with the steady intrusion of the private sector and weakened implementation of public health delivery system, over time India has failed to address the health needs of rural population. [3]

The discipline of Community Medicine failed to sufficiently counter this curative approach of the private sector until recently, when the Government of India's National Rural Health Mission (NRHM) again recognized an urgent need to focus on rural health. [4]

The status of postgraduate teaching in Community Medicine in India varies from institution to institution. Since the training contents, methodologies and exposure vary across schools, and the desired student competencies observed among institutions vary significantly, the discipline does not attract sufficient numbers of medical graduates into its postgraduate training programs. [5]

The examination and assessment systems of Community Medicine residents vary across Indian medical training institutions. There is no organized system of formative assessment and structured feedback in many institutions of India. Typically residents are subjected to family case study in the form of clinicosocial case presentation in which residents demonstrate clinical skills through a case presentation. It is important to carry out postgraduate training in community settings for residents to demonstrate the skills they have learned in community diagnosis, needs assessment and resource mobilization. [5]

The objective of postgraduate medical education in Community Medicine is to create competent specialists in Public Health. The Task Force report on Medical Education for India's National Rural Health Mission addressed the desired attitudes and the mindset of physician graduates. [6] The Task Force has reemphasized the need to make medical training more practical and skills-based, which was previously identified and endorsed by the Medical Council of India (MCI). [6]

The mini-clinical evaluation exercise (mini-CEX) is a promising student assessment tool that can test a broader range of clinical situations than the traditional clinical evaluation exercise. To assess clinical skills in postgraduates of Community Medicine, most institutions in India use the traditional method of 'clinicosocial case examination.' This approach requires about one hour and is in the form of face to face formative assessment but does not require learners to demonstrate skills. The mini-CEX is a promising student training and assessment tool that can test cognitive, psychomotor and affective domains. Further, it can be conducted in as little as 15 to 20 minutes for each of several informed skill practice sessions. The Mini-CEX offers greater opportunity for observation and meaningful and comprehensive feedback by a team to trainers in real time. [7] Given the promising effects and measurable characteristics of the mini-CEX, the American Board of Internal Medicine has encouraged its use in conjunction with other student assessment methods.

The mini-CEX has an attractive format and is capable of broad formative assessment in which feedback is a core component and assessment events are used strategically to guide trainees' learning. [7] We designed the "Clinicosocial Case Study Assessment" for formative assessment of Community Medicine residents as a way to broaden the parameters we assess in residents to include more skills needed in a community setting. In this report, we present and test a clinicosocial case study resident assessment tool based on the mini-CEX.

The clinicosocial case study assessment

The Medical Council of India (MCI) regulations for the Community Medicine postgraduate course focus on social, economic, environmental, biological and emotional determinants of health. The regulations expect students to be able to demonstrate empathy and humane behaviour towards patients and exhibit effective interpersonal skills. This can be achieved by a well-organized family exercise or clinicosocial case study in the community. Although the learner assessment approach varies across institutions, the clinicosocial case is generally presented by residents and assessed by faculty through face to face formative assessment, where the cognitive domain is assessed but psychomotor and affective domains are rarely evaluated. Lal [5] lamented on the inability of the system to provide enough opportunity of family case study that will enable students to develop the characteristics expected of a Community Medicine resident. He further elaborated how these exercises are conducted as rituals rather than organized attempt to provide good learning opportunity.


  Methods Top


A total of ten areas were identified and finalized for clinicosocial case study assessment after carefully reviewing the MCI regulations, Task Force recommendations on medical education, [6] and other pertinent articles and their references [5],[6],[8],[9] [Appendix 1 [Additional file 1]]. A blueprint for a new assessment tool was prepared based on its objectives. The assessment tool was developed based on principal of the mini CEX which can provide assessment where feedback is a core component. The assessment tool also can assess all three domains (cognitive, psychomotor and affective) during the clinicosocial case presentation. The tool was discussed among faculty members and residents during the designing stage, and feedback on relevance and application of the assessment tool was incorporated before it was finalized.

The Community Medicine residents are posted at Rural Health Training Centre (RHTC) for six months, managed by the Community Medicine department. Faculty members of the department identified useful clinicosocial cases from patients visiting the RHTC or during community visits. Three cases were also identified from patients visiting the tertiary care hospital attached to our medical college. Cases were identified in such a way that together they constituted an appropriate mix of the communicable and non-communicable diseases prevalent in the community. For clinicosocial cases that occurred at patients' homes, residents presented the cases to faculty while in field. The three cases selected from hospital patients were presented to faculty while in the department. Verbal consent was obtained from patients for this process after explaining the objective and the process. All patients we approached willingly participated in the study. Evaluators (faculty members) discussed the finer nuances of the case and provided detailed feedback after the presentation. Both residents and faculty could talk to the patient directly for any clarification.

Five residents in the department were used during the pilot testing process. Two residents presented three cases each, and three each presented one case, for a total of nine clinicosocial case presentations. Eight department faculty members participated in the formative assessment, dictated by their availability. A total of 50 assessments were conducted by the faculty members for these 9 clinicosocial case studies. The themes of the case studies included varied topics in paediatrics (3), tuberculosis (2), antenatal care (2), diabetes (1) and HIV-AIDS (1).

For each case, faculty members recorded the date, the complexity of the patient's problem on a 3-point scale (low, moderate and high), the gender of the patient, the setting (ambulatory, inpatient, emergency department or other), the number of minutes spent observing the resident and the number of minutes spent giving feedback to the resident. Faculty also noted whether the focus of the case study was data gathering, diagnosis, treatment or counselling.

Using a 9-point scale (in which 1 to 3 were "unsatisfactory," 4 to 6 were "satisfactory," and 7 to 9 were "superior"), faculty rated the resident on medical interviewing skills, socioeconomic classification, environmental conditions, social history, family planning and immunization, nutritional assessment, physical examination skills, epidemiological findings, counselling skills and overall competence [Appendix 2 [Additional file 2]]. Faculty also rated their own level of satisfaction with the mini-CEX method as a valid and efficient assessment approach, using a 9-point scale (1 representing "dissatisfied" and 9 representing "very satisfied") Faculty could indicate "not observed" when appropriate.

This assessment was used in formative assessment only. After observation, the faculty provided constructive feedback to the residents on their performance. Faculty and residents judged their satisfaction with the assessment tool on a scale of 1 to 10 and also provided comments and suggestions on the tool and the process.

Bland-Altman plots were used to evaluate the agreement between assessors with respect to the total score. Weighted Kappa with quadratic weights was used to determine the agreement between faculty members after categorizing the scores. Cronbach's alpha was used to test the internal consistency across faculty member scores. The project was approved by the Institutional Ethics Committee.


  Results Top


Level of agreement between two faculty members on their average scores was in the "satisfactory" range. Most of the 95% confidence limits using Bland -Altman method were within the predetermined limit of 2 points, for example, -1.79 to 0.43 between faculty 1 and 2' -2.12 and 0.45 between faculty 1 and 3; etc. [Figure 1]. When faculty scores were categorized into unsatisfactory, satisfactory and superior ranges, the inter-observer agreement dropped significantly with the overall Kappa between two faculty members ranging from 0.2 to 0.3: 0.28 between faculty 2 and 3; 0.25 between 2 and 1; etc.
Figure 1: Bland-Altman plots depicting agreement between faculty members on total score

Click here to view


Further considering faculty members as domains, the internal consistency was excellent (Cronbach's alpha = 0.92). The hypothesized sigmoidal curve of the average score through time trend analysis could not be confirmed because of insufficient data.

Qualitative feedback

Faculty members as a group expressed satisfaction with the simplicity of assessment tool. However, they stressed the need for further standardization of the tool to avoid bias and discrepancies that may arise out of personal views about a subject and the resident during scoring process. Most faculty members also suggested forming a core team of three or four faculty members to evaluate each session to properly monitor the progress of the resident.

All residents expressed their satisfaction with the comprehensive feedback they received but expressed the need to streamline the process in terms of its organization and scheduling.


  Discussion Top


Community Medicine residents in India are typically subjected to clinicosocial case study to assess what they have learned in providing care in patient encounters. Acquiring practical and clinical skills is crucial aspect of learning by virtue of its importance in health care delivery and hence its precise assessment is required. Currently, community medicine skills are generally assessed through face to face formative assessments which assess only the cognitive domain, and the psychomotor and affective domains are seldom evaluated. Without a fuller assessment, the clinicosocial case study exercise fails to help the learner develop the full range of community medicine skills. [5]

In the traditional assessment method, an experienced faculty assesses the resident during his/her clinicosocial case study presentation. The faculty member provides substantive feedback following which the resident provides a written report for review. Feedback may or may not be offered in a structured way to the resident, as there is no structured system for doing so. Noel and colleagues reported the efficiency of using a structured format for evaluating clinical skills of internal medicine residents. [10]

The mini-CEX is a 'performance-based' assessment approach shown in the US to be valid, reliable, feasible, acceptable and fair in post-graduate and undergraduate assessment. [7] Durning and colleagues found the mini-CEX to be easy to use, reliable, valid and easy to use for residents in Internal Medicine. [11] Norcini reported its good inter-examiner reliability. [12] It has also been found to be feasible, reliable and valid in formative assessment with clinical clerkship in internal medicine. [13]

The aim of the formative assessment of learners is not only to assign grades but it is also an ongoing part of the instructional process to support and enhance learning. [14] The ability of faculty to accurately observe residents performing tasks and provide effective feedback is an important aspect of medical training. [15]

Using Bland Altman Plots to assess the agreement between faculty members on total scores in this pilot's mini-CEX exam of community medicine residents showed very good agreement. In feedback, some faculty suggested more standardization, but we feel it may be better to allow some amount of subjectivity for fuller comprehensive feedback to residents. Further, the internal consistency considering faculty members as domains was found to be excellent, which means a resident receiving high score by one faculty member will almost surely get high score by other faculty members. We expect that as long as the faculty are consistent in their own scoring, this tool will accurately depict performance of the resident over time. This tool might provide more comprehensive and real time structured feedback that will facilitate residents' learning. At the same time, this tool is proactive as it will keep track of the overall growth as well as learning challenges of the residents in real time. The expected change in total mean scores from repeated mini-CEX assessments repeated over time for a resident gaining skills would be expected to have a sigmoidal ("S") shape: Initially a resident's mean score curve will be low and flat, followed by steady rise with improving skills until a satisfactory level is reached, and then again flattening depicting maintenance of the satisfactory level. Any significant fluctuation from this trend can be considered a warning signal indicating a learning difficulty that might need intervention.

While most of the domains were assessed sufficiently by most faculty, domains of nutritional assessment, physical examination skills, epidemiological finding, counselling skills and overall competence were assessed by faculty only 8%, 38%, 24%, 24% and 6% of the time, respectively; hence, utility of mini-CEX cannot be judged for these domains. There might be few possible reasons why faculty did not provide assessments of residents for some domains. Perhaps residents were unable to demonstrate some domains for faculty to then assess them or faculty members may require more training in the use of the new assessment tool, or the tool itself may need modification.

This pilot study suggests that this new assessment tool is reliable for assessing the clinicosoical case study in Community Medicine. Content validity of the form is by foundation on a literature review input from experts in Community Medicine and medical education. Its face validity should be confirmed by others in future studies.

There are several key limitations. The concurrent and construct validity of the tool cannot be fully examined due to the small sample size. Also, the faculty did not assess all ten domains during each clinicosocial case, and all cases were not assessed in a community setting. As this is a single centre study, replicability of the tool in other settings is needed. It is difficult at this point to comment on whether the number of faculty members participating in the mini-CEX and providing feedback to residents or the number of cases each resident presents and receives feedback on has greater impact on residents' learning. However, previous work suggests that the number of encounters is more important than the number of evaluators. [16]


  Conclusions Top


This new assessment tool is available for objective and unbiased assessment of Community Medicine residents through a clinicosocial case study. Ideally, it will receive further validation in different settings with a sufficiently large learner sample before being broadly adopted.

Acknowledgements

We acknowledge support of Community Medicine department and Pramukhswami Medical College during the conception and implementation of this study. We would also like to acknowledge faculty members and residents of community medicine department for unflagging and voluntary support.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

 
  References Top

1.
Government of India, Health Survey and Development (Bhore) Committee, Report. Vol. I. Delhi: Publications Division; 1946.  Back to cited text no. 1
    
2.
Banerji D. Social orientation of medical education in India. Econ Polit Wkly 1975;8:135-42.  Back to cited text no. 2
    
3.
Banerji D. Politics of rural health in India. Indian J Public Health 2005;49:113-22.  Back to cited text no. 3
[PUBMED]  Medknow Journal  
4.
Government of India, National Rural Health Mission: Mission Document. New Delhi: MOHFW; 2005.  Back to cited text no. 4
    
5.
Lal S. Scenario of post graduate medical education in community medicine in India. Indian J Community Med 2004;29:56-61.  Back to cited text no. 5
  Medknow Journal  
6.
Task Force on Medical Education for the National Rural Health Mission. New Delhi: Ministry of Health & Family Welfare, Government of India; 2009.  Back to cited text no. 6
    
7.
Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med 1995;123:795-9.  Back to cited text no. 7
[PUBMED]    
8.
Azhar GS, Jilani AZ. Future of community medicine in India. Indian J Community Med 2009;34:266-7.  Back to cited text no. 8
[PUBMED]  Medknow Journal  
9.
Gupta SK, Nongkynrih B, Pandav CS. Postgraduate medical education in community medicine: The AIIMS model. Natl Med J India 2013;26:287-90.  Back to cited text no. 9
    
10.
Noel GL, Herbers JE Jr., Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med 1992;117:757-65.  Back to cited text no. 10
    
11.
Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med 2002;77:900-4.  Back to cited text no. 11
[PUBMED]    
12.
Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner differences in the mini-CEX. Adv Health Sci Educ Theory Pract 1997;2:27-33.  Back to cited text no. 12
[PUBMED]    
13.
Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. J Gen Intern Med 2004;19 (5 Pt 2):558-61.  Back to cited text no. 13
    
14.
Sphepard LA. The role of assessment in a learning culture. Educ Res 2000;29:4-14.  Back to cited text no. 14
    
15.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No 31. Med Teach 2007;29:855-71.  Back to cited text no. 15
[PUBMED]    
16.
Swanson DB, van der Vleuten CP. Assessment of clinical skills with standardized patients: State of the art revisited. Teach Learn Med 2013;25 Suppl 1:S17-25.  Back to cited text no. 16
[PUBMED]    


    Figures

  [Figure 1]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Background
Methods
Results
Discussion
Conclusions
References
Article Figures

 Article Access Statistics
    Viewed1515    
    Printed23    
    Emailed0    
    PDF Downloaded351    
    Comments [Add]    

Recommend this journal