Print this page Email this page Users Online: 317 | Click here to view old website
Home About us Editorial Board Search Current Issue Archives Submit Article Author Instructions Contact Us Login 


 
 Table of Contents  
ORIGINAL RESEARCH ARTICLE
Year : 2020  |  Volume : 33  |  Issue : 3  |  Page : 95-107

Validation of a Spanish version of a 360° feedback tool for residents' performance: A pilot study


1 Department of Cardiology, Instituto Cardiovascular de Buenos Aires, Argentina
2 Department of Cardiology, Instituto Cardiovascular de Buenos Aires; Department of Clinical Investigation, National Scientific and Technological Research Council (CONICET), Ciudad Autónoma de Buenos Aires, Argentina
3 Department of Clinical Investigation, National Scientific and Technological Research Council (CONICET), Ciudad Autónoma de Buenos Aires, Argentina

Date of Submission19-May-2019
Date of Decision09-Aug-2020
Date of Acceptance31-Aug-2020
Date of Web Publication16-Mar-2021

Correspondence Address:
Josefina Belén Parodi
Av. del Libertador 6302, C1428ART, Ciudad Autonoma de Buenos Aires
Argentina
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/efh.EfH_119_19

  Abstract 


Background: The 360° feedback tool emerges as one of the most effective techniques for the assessment of humanistic qualities and communication skills of medical trainees, providing effective feedback. A valid Spanish version of this tool has not yet been published. The aim of this study was to evaluate the validity, reliability and feasibility rates of the Mini-peer Assessment Tool (Mini-PAT), a 360° feedback instrument, translated into Spanish applied on a cardiology residency program. Methods: We translated the Mini-PAT questionnaire into Spanish. The validation sample included all residents in our cardiology program (n = 19). Each resident was evaluated by 8 raters chosen by themselves, through a 4-point Likert scale. Validity was evaluated with factor analysis and reliability by analyzing internal consistency using the Cronbach's alpha coefficient. Feasibility was defined by a minimum of 80% of the raters responding the questionnaire. Results: The factor analysis clearly identified five item groupings, similar to the theoretical attributes predefined in the original questionnaire, providing evidence of the validity of the Spanish version. The Cronbach's alpha coefficient was 0.92, indicating high internal consistency of the items included. All the evaluators proposed completed the electronic form (152 surveys) demonstrating feasibility to implement. Discussion: This study provides evidence of reliability and validity of the Spanish version of the 360° feedback tool Mini-PAT performed in a cardiology residency program to assess global performance and humanistic qualities.

Keywords: Constructive feedback, Educational assessment, Professional competence, Self-evaluation programs, Training programs, Work performance


How to cite this article:
Parodi JB, de Lima AA, Burgos LM, Gelpi RJ. Validation of a Spanish version of a 360° feedback tool for residents' performance: A pilot study. Educ Health 2020;33:95-107

How to cite this URL:
Parodi JB, de Lima AA, Burgos LM, Gelpi RJ. Validation of a Spanish version of a 360° feedback tool for residents' performance: A pilot study. Educ Health [serial online] 2020 [cited 2021 Sep 28];33:95-107. Available from: https://www.educationforhealth.net/text.asp?2020/33/3/95/311315




  Background Top


The growing interest on improving the quality of medical care has led to include in the evaluation of trainees' performance not only theoretical knowledge but also humanistic qualities and communication skills, which cannot be easily evaluated with written examinations.[1] It is mandatory for a residency program to implement assessment and feedback on residents' performance during their training period. However, a study by Day et al. documented that the vast majority of internal medicine residents were not observed in a patient encounter more than once by a staff physician.[2] Without this observation, there is no opportunity to assess basic clinical skills and more importantly, to provide feedback to improve performance. This has led to efforts in creating quantitative tools and evaluation instruments that, through direct workplace-based assessment, provide opportunities to design an action plan based on the needs identified.[3],[4] Among different methods of evaluation, multisource feedback (MSF) or 360° feedback emerges as an effective instrument widely validated in the English-speaking medical environment.[5],[6],[7] This method provides effective feedback to correct or strengthen actions and performance,[4] allowing longitudinal monitoring. The 360° feedback assessment consists of a questionnaire where different dimensions such as clinical care, good medical practice, working with colleagues, relationship with patients, and leadership are analyzed. The items are rated on a Likert scale,[8] with the particularity that the raters are people who continually interact with the residents in their daily practice, such as their peers, staff physicians, nurses, technicians, or any other medical personnel.

Since the eighties, different medical organizations have begun to include these aspects in the evaluation of medical trainees, though initially mostly based on global performance. Ramsey et al. were pioneers in demonstrating the reliability and feasibility of a peer-assessment tool consisting of 11 questions answered by a minimum number of 7–13 raters per participant.[5],[6],[7],[9] However, the implementation of this instrument was slow, and it was not until 2005 that Archer et al.[10] developed the Sheffield peer review assessment tool (SPRAT), consisting of 24 questions covering five dimensions. A minimum of 8 raters from the health-care system evaluated each physician. The results were so conclusive that the General Council of Medicine of London included this tool to define the standards of good medical practice, adapting it to a shortened version known as mini-peer assessment tool (PAT).[11]

However, the value of feedback depends not only on the tool but mainly on who gives it. One of the many barriers teachers finds when willing to give feedback is the lack of instructions and training and the fear of providing negative feedback.[12] Cantillon and Sargeant delivered recommendations in order to give a successful feedback.[13] There are several techniques to impart it,[14] but most importantly, feedback should be seen as an everyday component of the teacher–student relationship, so that both sides can expect it and manage its effects.[13]

Although the validity and reliability of the 360° assessment was demonstrated, each work group used a variety of instruments over time, with questionnaires of different lengths, and without a clear consensus about the minimum number of raters needed. Moreover, a validation of the mini-PAT translated into Spanish with sufficient statistical power to be applied in medical trainees has not yet been published.

The aim of this study was to evaluate the validity, reliability, and feasibility of a modified version of the mini-PAT translated into Spanish applied on a cardiology residency program.


  Methods Top


We conducted an analytical, cross-sectional study. We translated the mini-PAT[3] questionnaire into Spanish. The instrument consists of 19 items distributed in 5 domains, rated using a 4-point Likert scale, where: (1) below expectations for the year of postgraduate training; (2) meets expectations; (3) above expectations; (4) unable to evaluate. The validation sample included all residents in our cardiology program. Each resident was evaluated by 8 raters, chosen by themselves among the staff physicians, chief residents, head of department, nurses, technicians, and any other health professionals. Each resident also completed a self-assessment questionnaire and by the end of the feedback process, a satisfaction survey.

The questionnaire was created using Google Drive platform forms, which allows to answer online meanwhile automatically stores information in a Google Drive database [Figure 1]. [Questionnaires 1 and 2 from Supplementary material].
Figure 1: Electronic questionnaire preview (first page)

Click here to view


The validity of our Spanish version of the questionnaire was evaluated using factor analysis of the variables observed. The principal component analysis approach was used for factor extraction and varimax rotation was used for interpretation. Then, we studied the possible relationship between items with high factorial load for the same component, to identify what this component represents as an attribute, and these attributes were compared with those prespecified in the original English questionnaire.

Reliability was evaluated by analyzing internal consistency using the Cronbach's alpha coefficient. Total and partial consistency were evaluated, analyzing each question within its corresponding dimension, and calculating Cronbach's alpha if the question was not part of that dimension.

The implementation feasibility was assessed by estimating that at least 80% of the evaluators would complete the electronic form.

The study protocol was reviewed and approved by the Institutional Ethics Committee. The investigation was conducted in accordance with the Helsinki Declaration. All the participants gave their consent to take part in the study.


  Results Top


Nineteen trainees in the cardiology residency program were evaluated using the online 360° feedback tool; 57% were women and the mean age was 28.1 ± 3.9 years. A total of 152 evaluations were completed, as all the evaluators proposed completed the electronic form (8 for each resident). One hundred and twenty-five (82.2%) evaluations were fully complete; of the 27 remaining, at least one item was answered as “Unable to evaluate,” which were considered missing values.

Regarding to the reliability of the tool, the Cronbach's alpha coefficient was 0.92. Total and partial coefficients were close to 1, indicating that all items included in the instrument have high internal consistency, and thus demonstrating reliability of the translated tool.

As previously mentioned, 100% of the proposed evaluators completed the electronic form, demonstrating the tool was feasible to implement.

The factor analysis clearly identified five item groupings, which explained 73.6% of the variance: good clinical care, maintaining good medical practice, working with colleagues, relationship with patients, and leadership, teaching, and training. This items where conceptually similar to the theoretical attributes predefined in the original English questionnaire, providing evidence of the validity of the Spanish version of the questionnaire [Table 1].
Table 1: Factor analysis. Items gathered in 5 components or dimensions, showing validity

Click here to view



  Discussion Top


This study provides evidence of reliability and validity of the Spanish version of the mini-PAT questionnaire performed in a group of cardiology residents. The implementation of the tool to assess the global performance was feasible, with high levels of internal consistency.

We decided to translate the mini-PAT tool instead of the original SPRAT because as Ramsey et al.[5],[6] previously reported, the response rate of the evaluators increases when the number of questions is reduced, maintaining the factorial structure of the original form. This was also proved with the factor analysis of our study.

To give feedback, we used the reflective feedback conversation.[14] First, we discussed self-assessment conclusions with the learner, allowing him to recognize his own performance deficits and strengths. Then, we revealed the peer-assessment results, starting with their best outcomes and later on, the areas with poor performance or those that could be improved. Finally, we defined an appropriate plan of action to overcome these difficulties. This technique encourages the development of the learners' ability to self-assess and how he plans to improve, more than only receiving the counterpart´s opinion.[13] Although the aim of our study was to analyze the feedback tool and not the technique to inform it, we understand that one cannot succeed without the other so we took special interest and thoughtful planning of the feedback deliver, according to international recommendations [Figure 2].
Figure 2: Example of feedback chart returned to each resident

Click here to view


The final aim of the MSF tool is to identify areas requiring improvement through the assessment of global performance, and thus develop a plan of action to achieve the desired learning outcomes and continuously improve residents' training program. Both the ISO 9001-2008 quality standards for health-care delivery and the Argentine Ministry of Health, following the recommendations of the International Medical Councils, encourage the residency programs to apply standardized evaluation methods like the 360° feedback instrument, which are less dependent on subjective factors.[15] However, there are no such tools translated into Spanish so far. Therefore, we consider that this could represent a pilot study in order to achieve the international recommendations, as it is the first study able to demonstrate the validity, reliability, and feasibility of implementing the mini-PAT translated from its original English version into Spanish, in a group of medical trainees from a residency program in Argentina. Nevertheless, we are aware of the limitations and possible bias of the results when only including a single and private health-care center.

Other limitation of our study was including only 19 residents in a single subspecialty program. Although the findings agree with the original version in terms of factors identified, repeated testing with other cohorts and in different centers is needed to gather additional evidence of the validity of this tool.

The absence of unified methods to assess residents' performance represents one of the most important barriers to measure it periodically. This pilot study provides evidence of a valid translated tool that could be used by another residency program in Spanish speaking countries, along with written examinations and mini-CEX evaluations,[15] to help narrowing the gap between actual and desired performance in order to achieve quality care.


  Conclusion Top


The Spanish version of the mini-PAT, a type of 360° feedback tool, used during a cardiology residency program to assess the humanistic qualities and communication skills is valid, reliable, and feasible to apply.

Financial support and sponsorship

We declare we have no affiliations with or involvement in any organization or entity with any financial interest in the subject discussed in this manuscript.

Conflicts of interest

We have no conflicts of interest to disclose.





 
  References Top

1.
Langsley DG. Medical competence and performance assessment. A new Era. JAMA 1991;266:977-80.  Back to cited text no. 1
    
2.
Day SC, Grosso LJ, Norcini JJ Jr., Blank LL, Swanson DB, Horne MH. Residents' perception of evaluation procedures used by their training program. J Gen Intern Med 1990;5:421-6.  Back to cited text no. 2
    
3.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-71.  Back to cited text no. 3
    
4.
Miller A, Archer J. Impact of workplace based assessment on doctors' education and performance: A systematic review. BMJ 2010;341:c5064.  Back to cited text no. 4
    
5.
Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269:1655-60.  Back to cited text no. 5
    
6.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med 1989;110:719-26.  Back to cited text no. 6
    
7.
Wenrich MD, Carline JD, Giles LM, Ramsey PG. Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med 1993;68:680-7.  Back to cited text no. 7
    
8.
Cañadas I, Sánchez Bruno A. Categorías de respuesta en escalas tipo Likert. Psicothema 1998;10:623-31.  Back to cited text no. 8
    
9.
Carline JD, Paauw DS, Thiede KW, Ramsey PG. Factors affecting the reliability of ratings of students' clinical skills in a medicine clerkship. J Gen Intern Med 1992;7:506-10.  Back to cited text no. 9
    
10.
Archer J, Norcini J, Davies H. Peer review of paediatricians in training using SPRAT. Br Med J 2005;330:1251-3.  Back to cited text no. 10
    
11.
General Medical Council. Good Medical Practice. [Internet]. UK: General Medical Council; c2013 [cited 2019 Jan 30]. Available from: https://www.gmc-uk.org/.  Back to cited text no. 11
    
12.
Dudek NL, Marks MB, Regehr G. Failure to fail: The perspectives of clinical supervisors. Acad Med 2005;80:S84-7.  Back to cited text no. 12
    
13.
Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ 2008;337:a1961.  Back to cited text no. 13
    
14.
Alves de Lima AE. Constructive feedback. A strategy to enhance learning. Medicina (B Aires) 2008;68:88-92.  Back to cited text no. 14
    
15.
Davini C, Rodríguez G. Guidebook to Develop Performance Measurement in Healthcare Residency Programs. [Internet]: Argentina's Ministry of Health; c 2015 [cited 2019 Jan 30]Available from: https://www.argentina.gob.ar/sites/default/files/guia-desarrollo-de-la-evaluacion-del-desempenio-residencias-del-equipo-salud.pdf.  Back to cited text no. 15
    


    Figures

  [Figure 1], [Figure 2]
 
 
    Tables

  [Table 1]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Background
Methods
Results
Discussion
Conclusion
References
Article Figures
Article Tables

 Article Access Statistics
    Viewed1737    
    Printed18    
    Emailed0    
    PDF Downloaded276    
    Comments [Add]    

Recommend this journal