|ORIGINAL RESEARCH ARTICLE
|Year : 2020 | Volume
| Issue : 3 | Page : 95-107
Validation of a Spanish version of a 360° feedback tool for residents' performance: A pilot study
Josefina Belén Parodi1, Alberto Alves de Lima2, Lucrecia Maria Burgos1, Ricardo J Gelpi3
1 Department of Cardiology, Instituto Cardiovascular de Buenos Aires, Argentina
2 Department of Cardiology, Instituto Cardiovascular de Buenos Aires; Department of Clinical Investigation, National Scientific and Technological Research Council (CONICET), Ciudad Autónoma de Buenos Aires, Argentina
3 Department of Clinical Investigation, National Scientific and Technological Research Council (CONICET), Ciudad Autónoma de Buenos Aires, Argentina
|Date of Submission||19-May-2019|
|Date of Decision||09-Aug-2020|
|Date of Acceptance||31-Aug-2020|
|Date of Web Publication||16-Mar-2021|
Josefina Belén Parodi
Av. del Libertador 6302, C1428ART, Ciudad Autonoma de Buenos Aires
Source of Support: None, Conflict of Interest: None
Background: The 360° feedback tool emerges as one of the most effective techniques for the assessment of humanistic qualities and communication skills of medical trainees, providing effective feedback. A valid Spanish version of this tool has not yet been published. The aim of this study was to evaluate the validity, reliability and feasibility rates of the Mini-peer Assessment Tool (Mini-PAT), a 360° feedback instrument, translated into Spanish applied on a cardiology residency program. Methods: We translated the Mini-PAT questionnaire into Spanish. The validation sample included all residents in our cardiology program (n = 19). Each resident was evaluated by 8 raters chosen by themselves, through a 4-point Likert scale. Validity was evaluated with factor analysis and reliability by analyzing internal consistency using the Cronbach's alpha coefficient. Feasibility was defined by a minimum of 80% of the raters responding the questionnaire. Results: The factor analysis clearly identified five item groupings, similar to the theoretical attributes predefined in the original questionnaire, providing evidence of the validity of the Spanish version. The Cronbach's alpha coefficient was 0.92, indicating high internal consistency of the items included. All the evaluators proposed completed the electronic form (152 surveys) demonstrating feasibility to implement. Discussion: This study provides evidence of reliability and validity of the Spanish version of the 360° feedback tool Mini-PAT performed in a cardiology residency program to assess global performance and humanistic qualities.
Keywords: Constructive feedback, Educational assessment, Professional competence, Self-evaluation programs, Training programs, Work performance
|How to cite this article:|
Parodi JB, de Lima AA, Burgos LM, Gelpi RJ. Validation of a Spanish version of a 360° feedback tool for residents' performance: A pilot study. Educ Health 2020;33:95-107
| Background|| |
The growing interest on improving the quality of medical care has led to include in the evaluation of trainees' performance not only theoretical knowledge but also humanistic qualities and communication skills, which cannot be easily evaluated with written examinations. It is mandatory for a residency program to implement assessment and feedback on residents' performance during their training period. However, a study by Day et al. documented that the vast majority of internal medicine residents were not observed in a patient encounter more than once by a staff physician. Without this observation, there is no opportunity to assess basic clinical skills and more importantly, to provide feedback to improve performance. This has led to efforts in creating quantitative tools and evaluation instruments that, through direct workplace-based assessment, provide opportunities to design an action plan based on the needs identified., Among different methods of evaluation, multisource feedback (MSF) or 360° feedback emerges as an effective instrument widely validated in the English-speaking medical environment.,, This method provides effective feedback to correct or strengthen actions and performance, allowing longitudinal monitoring. The 360° feedback assessment consists of a questionnaire where different dimensions such as clinical care, good medical practice, working with colleagues, relationship with patients, and leadership are analyzed. The items are rated on a Likert scale, with the particularity that the raters are people who continually interact with the residents in their daily practice, such as their peers, staff physicians, nurses, technicians, or any other medical personnel.
Since the eighties, different medical organizations have begun to include these aspects in the evaluation of medical trainees, though initially mostly based on global performance. Ramsey et al. were pioneers in demonstrating the reliability and feasibility of a peer-assessment tool consisting of 11 questions answered by a minimum number of 7–13 raters per participant.,,, However, the implementation of this instrument was slow, and it was not until 2005 that Archer et al. developed the Sheffield peer review assessment tool (SPRAT), consisting of 24 questions covering five dimensions. A minimum of 8 raters from the health-care system evaluated each physician. The results were so conclusive that the General Council of Medicine of London included this tool to define the standards of good medical practice, adapting it to a shortened version known as mini-peer assessment tool (PAT).
However, the value of feedback depends not only on the tool but mainly on who gives it. One of the many barriers teachers finds when willing to give feedback is the lack of instructions and training and the fear of providing negative feedback. Cantillon and Sargeant delivered recommendations in order to give a successful feedback. There are several techniques to impart it, but most importantly, feedback should be seen as an everyday component of the teacher–student relationship, so that both sides can expect it and manage its effects.
Although the validity and reliability of the 360° assessment was demonstrated, each work group used a variety of instruments over time, with questionnaires of different lengths, and without a clear consensus about the minimum number of raters needed. Moreover, a validation of the mini-PAT translated into Spanish with sufficient statistical power to be applied in medical trainees has not yet been published.
The aim of this study was to evaluate the validity, reliability, and feasibility of a modified version of the mini-PAT translated into Spanish applied on a cardiology residency program.
| Methods|| |
We conducted an analytical, cross-sectional study. We translated the mini-PAT questionnaire into Spanish. The instrument consists of 19 items distributed in 5 domains, rated using a 4-point Likert scale, where: (1) below expectations for the year of postgraduate training; (2) meets expectations; (3) above expectations; (4) unable to evaluate. The validation sample included all residents in our cardiology program. Each resident was evaluated by 8 raters, chosen by themselves among the staff physicians, chief residents, head of department, nurses, technicians, and any other health professionals. Each resident also completed a self-assessment questionnaire and by the end of the feedback process, a satisfaction survey.
The questionnaire was created using Google Drive platform forms, which allows to answer online meanwhile automatically stores information in a Google Drive database [Figure 1]. [Questionnaires 1 and 2 from Supplementary material].
The validity of our Spanish version of the questionnaire was evaluated using factor analysis of the variables observed. The principal component analysis approach was used for factor extraction and varimax rotation was used for interpretation. Then, we studied the possible relationship between items with high factorial load for the same component, to identify what this component represents as an attribute, and these attributes were compared with those prespecified in the original English questionnaire.
Reliability was evaluated by analyzing internal consistency using the Cronbach's alpha coefficient. Total and partial consistency were evaluated, analyzing each question within its corresponding dimension, and calculating Cronbach's alpha if the question was not part of that dimension.
The implementation feasibility was assessed by estimating that at least 80% of the evaluators would complete the electronic form.
The study protocol was reviewed and approved by the Institutional Ethics Committee. The investigation was conducted in accordance with the Helsinki Declaration. All the participants gave their consent to take part in the study.
| Results|| |
Nineteen trainees in the cardiology residency program were evaluated using the online 360° feedback tool; 57% were women and the mean age was 28.1 ± 3.9 years. A total of 152 evaluations were completed, as all the evaluators proposed completed the electronic form (8 for each resident). One hundred and twenty-five (82.2%) evaluations were fully complete; of the 27 remaining, at least one item was answered as “Unable to evaluate,” which were considered missing values.
Regarding to the reliability of the tool, the Cronbach's alpha coefficient was 0.92. Total and partial coefficients were close to 1, indicating that all items included in the instrument have high internal consistency, and thus demonstrating reliability of the translated tool.
As previously mentioned, 100% of the proposed evaluators completed the electronic form, demonstrating the tool was feasible to implement.
The factor analysis clearly identified five item groupings, which explained 73.6% of the variance: good clinical care, maintaining good medical practice, working with colleagues, relationship with patients, and leadership, teaching, and training. This items where conceptually similar to the theoretical attributes predefined in the original English questionnaire, providing evidence of the validity of the Spanish version of the questionnaire [Table 1].
|Table 1: Factor analysis. Items gathered in 5 components or dimensions, showing validity|
Click here to view
| Discussion|| |
This study provides evidence of reliability and validity of the Spanish version of the mini-PAT questionnaire performed in a group of cardiology residents. The implementation of the tool to assess the global performance was feasible, with high levels of internal consistency.
We decided to translate the mini-PAT tool instead of the original SPRAT because as Ramsey et al., previously reported, the response rate of the evaluators increases when the number of questions is reduced, maintaining the factorial structure of the original form. This was also proved with the factor analysis of our study.
To give feedback, we used the reflective feedback conversation. First, we discussed self-assessment conclusions with the learner, allowing him to recognize his own performance deficits and strengths. Then, we revealed the peer-assessment results, starting with their best outcomes and later on, the areas with poor performance or those that could be improved. Finally, we defined an appropriate plan of action to overcome these difficulties. This technique encourages the development of the learners' ability to self-assess and how he plans to improve, more than only receiving the counterpart´s opinion. Although the aim of our study was to analyze the feedback tool and not the technique to inform it, we understand that one cannot succeed without the other so we took special interest and thoughtful planning of the feedback deliver, according to international recommendations [Figure 2].
The final aim of the MSF tool is to identify areas requiring improvement through the assessment of global performance, and thus develop a plan of action to achieve the desired learning outcomes and continuously improve residents' training program. Both the ISO 9001-2008 quality standards for health-care delivery and the Argentine Ministry of Health, following the recommendations of the International Medical Councils, encourage the residency programs to apply standardized evaluation methods like the 360° feedback instrument, which are less dependent on subjective factors. However, there are no such tools translated into Spanish so far. Therefore, we consider that this could represent a pilot study in order to achieve the international recommendations, as it is the first study able to demonstrate the validity, reliability, and feasibility of implementing the mini-PAT translated from its original English version into Spanish, in a group of medical trainees from a residency program in Argentina. Nevertheless, we are aware of the limitations and possible bias of the results when only including a single and private health-care center.
Other limitation of our study was including only 19 residents in a single subspecialty program. Although the findings agree with the original version in terms of factors identified, repeated testing with other cohorts and in different centers is needed to gather additional evidence of the validity of this tool.
The absence of unified methods to assess residents' performance represents one of the most important barriers to measure it periodically. This pilot study provides evidence of a valid translated tool that could be used by another residency program in Spanish speaking countries, along with written examinations and mini-CEX evaluations, to help narrowing the gap between actual and desired performance in order to achieve quality care.
| Conclusion|| |
The Spanish version of the mini-PAT, a type of 360° feedback tool, used during a cardiology residency program to assess the humanistic qualities and communication skills is valid, reliable, and feasible to apply.
Financial support and sponsorship
We declare we have no affiliations with or involvement in any organization or entity with any financial interest in the subject discussed in this manuscript.
Conflicts of interest
We have no conflicts of interest to disclose.
| References|| |
Langsley DG. Medical competence and performance assessment. A new Era. JAMA 1991;266:977-80.
Day SC, Grosso LJ, Norcini JJ Jr., Blank LL, Swanson DB, Horne MH. Residents' perception of evaluation procedures used by their training program. J Gen Intern Med 1990;5:421-6.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-71.
Miller A, Archer J. Impact of workplace based assessment on doctors' education and performance: A systematic review. BMJ 2010;341:c5064.
Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269:1655-60.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med 1989;110:719-26.
Wenrich MD, Carline JD, Giles LM, Ramsey PG. Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med 1993;68:680-7.
Cañadas I, Sánchez Bruno A. Categorías de respuesta en escalas tipo Likert. Psicothema 1998;10:623-31.
Carline JD, Paauw DS, Thiede KW, Ramsey PG. Factors affecting the reliability of ratings of students' clinical skills in a medicine clerkship. J Gen Intern Med 1992;7:506-10.
Archer J, Norcini J, Davies H. Peer review of paediatricians in training using SPRAT. Br Med J 2005;330:1251-3.
General Medical Council. Good Medical Practice. [Internet]. UK: General Medical Council; c2013 [cited 2019 Jan 30]. Available from: https://www.gmc-uk.org/
Dudek NL, Marks MB, Regehr G. Failure to fail: The perspectives of clinical supervisors. Acad Med 2005;80:S84-7.
Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ 2008;337:a1961.
Alves de Lima AE. Constructive feedback. A strategy to enhance learning. Medicina (B Aires) 2008;68:88-92.
[Figure 1], [Figure 2]