|ORIGINAL RESEARCH PAPER
|Year : 2009 | Volume
| Issue : 1 | Page : 112
Measuring Students' Perceptions of the Educational Climate of the New Curriculum at the Pontificia Universidad Catσlica de Chile: Performance of the Spanish Translation of the Dundee Ready Education Environment Measure (DREEM)
A Riquelme1, M Oporto2, J Oporto2, JI Méndez2, P Viviani3, F Salech2, J Chianale1, R Moreno2, I Sánchez2
1 Department of Gastroenterology, Pontificia Universidad Católica de Chile, Santiago, Chile
2 Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile
3 Faculty of Public Health, Pontificia Universidad Católica de Chile, Santiago, Chile
|Date of Submission||03-Sep-2007|
|Date of Acceptance||13-Feb-2009|
|Date of Web Publication||08-May-2009|
Marcoleta 367, Casilla 114-D, Santiago
Source of Support: None, Conflict of Interest: None
Context: During the last decade a major curriculum reform was carried out at the Pontificia Universidad Católica de Chile Medical School. The process included changes in curriculum development, staff development and in the infrastructure. However, it is not known how students perceived the climate of their education within the new model.
Objectives: To measure students' perceptions of the educational environment of the new curriculum and to evaluate the internal consistency of the 50-item Dundee Ready Education Environment Measure (DREEM) Spanish version questionnaire.
Methods: The DREEM Spanish version questionnaire was administered to undergraduate medical students in training years 3, 4 and 5. Internal consistency of the instrument and its subscales were measured with the method described by Cronbach, and the results were expressed with alpha coefficient ranging from 0 to 1.
Findings: Responses were received from 297 out of 328 students (90.5%). The 50-item DREEM Spanish version was found highly reliable with an alpha coefficient of 0.91. The subscale with the highest mean score was "Academic Self-Perceptions", which indicates students' perceptions of their academic achievements. Mean score of this subscale was 22.3 ± 4.1 corresponding to 69.7% of the maximum score. The lowest mean score was for the Students' Perceptions of their Social Environment: 15.9 ± 4.0 (56.8%). The overall mean score for the 50 items was 127.5 ± 20.9 (63.8% of maximum). Scores observed in students in year 5 were significantly lower for several subscales, including Students' Perceptions of Learning, Students' Perceptions of Teachers, Students' Perceptions of the Learning Atmosphere and Students' Perceptions of the Social Environment, and also lower for the overall mean score (119.3 ± 20.2) compared to scores in years 3 and 4 (128.8 ± 21 and 132.5 ± 19.7, respectively; p<0.001).
Conclusions: The school's educational climate was generally perceived positively by students, although they viewed the school's social environment less favorably. Specific areas identified by students as needing improvement included an overloaded curriculum and inadequate student supports. The DREEM Spanish version proved generally reliable, by internal consistency scores based on ratings by Chilean undergraduate medical students; it should be a useful tool for assessing students' perceptions of the educational environments of other Latin American medical schools.
Keywords: Educational climate, undergraduate medical students, Chile, DREEM questionnaire
|How to cite this article:|
Riquelme A, Oporto M, Oporto J, Méndez J I, Viviani P, Salech F, Chianale J, Moreno R, Sánchez I. Measuring Students' Perceptions of the Educational Climate of the New Curriculum at the Pontificia Universidad Catσlica de Chile: Performance of the Spanish Translation of the Dundee Ready Education Environment Measure (DREEM). Educ Health 2009;22:112
|How to cite this URL:|
Riquelme A, Oporto M, Oporto J, Méndez J I, Viviani P, Salech F, Chianale J, Moreno R, Sánchez I. Measuring Students' Perceptions of the Educational Climate of the New Curriculum at the Pontificia Universidad Catσlica de Chile: Performance of the Spanish Translation of the Dundee Ready Education Environment Measure (DREEM). Educ Health [serial online] 2009 [cited 2020 Nov 29];22:112. Available from: https://www.educationforhealth.net/text.asp?2009/22/1/112/101566
In the 1980s, the undergraduate curriculum of the Pontificia Universidad Católica de Chile Medical School (PUCMS) was a traditional one, as defined by the General Medical Council (GMC) in the United Kingdom (UK) (1993). The curriculum was teacher-centred, discipline-based and hospital-based, without many options for students or elective modules. Despite the general perception that the educational climate of the traditional curriculum was uncomfortable, there was no measurement of the educational environment in the PUCMS at that time.
The faculty initiated a reform of the undergraduate curriculum in 1993 (Moreno & Velasco, 1994; Rosso et al., 1997), addressing the educational aspects highlighted by the World Federation of Medical Education (1993) and targeting the characteristics desired of the tomorrow’s physicians as developed by the Association of American Medical Colleges (AAMC) (1984) and the GMC in the UK (1993).
Curriculum changes included a reduction in the number of curriculum core contents, expansion in the number of electives, vertical and horizontal integration of the curriculum, modernization of teaching methods through use of small groups, problem-oriented teaching and more ambulatory care exposure (Sanchez et al., 2007). Six education strategies have been identified (Harden et al., 1984) relating to the curriculum in a medical school. Each issue can be represented as a spectrum or continuum: student-centred / teacher-centred, problem-based / information gathering, integrated /discipline-based, community-based / hospital-based, elective / uniform and systematic /apprenticeship –based. The SPICES (student-centred, problem-based, integrated, community-based, electives and systematic) model of curriculum strategy analysis can be used in curriculum planning or review. Newer schools tend to be more to the left on the continuum and if we analysed the new curriculum of the PUCMS, it incorporated most of the educational strategies highlighted by Harden et al. (1984) in the SPICES model. In order to meet the needs of the new curriculum, a new school of medicine building was inaugurated in 2004, which included a biomedical library with Information Communication Technology (ICT) services.
Academic performance among medical students improved after the curricular reform, with lower student failure rates, fewer promotion delays, fewer dropouts from the school and improved grades, all without changing the standards for passing within the PUCMS (Bitran et al., 2002). Moreover, during the period 2003-2005 graduates from the PUCMS obtained the highest average scores among graduates of all Chilean medical schools on the National Medical Examination (Enriquez & Mena, 2005), (National Medical Examination, Chile, 2005).
There is well recognized connection between schools’ educational environment and the important outcomes of students’ achievement, satisfaction and success (Genn, 2001). The World Federation for Medical Education (1998) considers the learning environment as one of the areas that should be targeted when evaluating medical education programmes. In our school, students’ perceptions of the educational environment within this new curriculum model were unknown.
It is important to understand students’ perceptions of the learning environment of their schools because this information, gained through questionnaires, interviews and focus groups, can be used to enhance the strengths and address the weaknesses of the institution (Roff, 2005). In 1986, Genn & Harden (1986) reviewed instruments that had been developed to measure the environment and climate of higher education institutions. Several instruments were then available for assessing the environment of medical schools (Hutchins, 1961; Rothman & Ayoade, 1970; Marshall, 1978; Feletti & Clarke, 1981). These questionnaires are now out of date in that they do not address recent imperatives for curriculum changes and newer educational strategies (Hutchinson, 2003; Roff, 2005).
More recently, Roff et al. (1997) developed the Dundee Ready Education Environment Measure (DREEM) to assess the educational environment of a Scottish medical school as perceived by undergraduate medical students. Deza translated the 58-item preliminary DREEM questionnaire into Spanish and found it was reliable, with an internal consistency of 0.91 (Roff et al., 1997). The questionnaire was subsequently refined by Roff et al. (1997) and the number of items was reduced from 58 to 50; the reliability of the 50-item DREEM questionnaire (Spanish version) has not been tested.
(1) To measure students’ perceptions of the educational environment of the new undergraduate curriculum of the PUCMS, and to see how perceptions varied according to students’ year of training. (2) To evaluate the internal consistency of the 50-item DREEM Spanish version questionnaire.
Material and Methods
The development and validation of the 50-item DREEM has been reported elsewhere (Roff et al., 1997). Respondents score each item from 0 to 4 (4 = Strongly agree, 3 = Agree, 2 = Unsure, 1 = Disagree and 0 = Strongly disagree). Some items (4, 8, 9, 17, 25, 35, 39, 48 and 50) contain negative statements and are therefore reverse-coded when incorporated into scales. According to Roff et al. (1997) individual items with a mean score of 3 or greater reflect a positive educational environment and are considered areas of strength for a school; values between 2 and 3 reflect areas that are neither strengths nor weaknesses but identify areas that could be enhanced; and items with a mean score below 2 are considered areas of weaknesses for a school of medicine. The DREEM yields a global score of up to 200 with its 50 items combined and has five subscales proposed by Roff et al. (1997): (1) Students’ Perceptions of Learning (items 1, 7, 13, 16, 20, 22, 24, 25, 38, 44, 47 and 48), which addresses students’ views of aspects of the teaching activities, such as whether they receive clear course objectives, and whether learning is student-focused and encourages active learning rather than being teacher-centred and stresses factual learning; (2) Students’ Perceptions of Teachers (items 2, 6, 8, 9, 18, 29, 32, 37, 39, 40 and 50), which address students’ views of the qualities of teachers, including their communication skills, whether they provide feedback to students and patients, their level of knowledge and their level preparation for classes; (3) Students’ Perceptions of their Academic Skills (items 5, 10, 21, 26, 27, 31, 41 and 45), which includes students’ views of the learning strategies and problem-solving skills they have developed to prepare themselves for their profession; (4) Students’ Perceptions of the Learning Atmosphere (items 11, 12, 17, 23, 30, 33, 34, 35, 36, 42, 43 and 49), which includes items addressing how relaxed the atmosphere is during lectures and ward teaching, whether teaching activities are motivating for students, and whether there are opportunities for students to develop interpersonal skills; and (5) Students’ Perceptions of the Social Environment (items 3, 4, 14, 15, 19, 28 and 46), which addresses students’ views of the support systems available to those who become stressed, the school’s accommodations for students, the quality of campus social life, and whether students’ are able to find friends at school.
The survey of the educational environment at the PUCMS was carried out in December 2005. The questionnaire was administered to undergraduate medical students of training years 3, 4 and 5. The DREEM was administered in paper form to students in years 3 and 4. Year 5 medical students were on various clinical rotations and for this reason some questionnaires were administered on paper and others through an on-line questionnaire. The questionnaires were answered anonymously by students. The study was approved by the human subjects review committee of the PUCMS.
The variables were described using means and standard deviations (SD). Comparisons of students of the various years were analysed using t tests. Variance analysis was carried out for years 3, 4 and 5 using ANOVA. Differences were considered statistically significant with a p value <0.05. The method described by Cronbach (1951) was used to measure scale reliability. The internal consistencies of the inventory and its subscales were calculated using the Statistical Package for Social Science (SPSS 11) and the results were expressed as alpha coefficients ranging from 0 to 1.
Completed questionnaires were received from 108 out of 110 students (98.2%) in year 3, 106 out of 108 students (98.1%) in year 4 and 83 out of 110 students (75.5%) in year 5. The overall response rate was 90.5%. Among respondents, 159 (53.5%) were males and 138 (46.5%) were females.
Individual item scores
The mean scores of individual items are shown in table 1. Scores for 9 out of 50 items (18%) were below 2, indicating areas of weaknesses of the PUCMS. Items 3, 17, 25, 27 and 46 had the lowest mean scores of the 50 items. The values for the majority of the items (60%) fell into the range that indicated aspects of the environment that would benefit from improvement (mean scores between 2 and 3). Eleven items (22%) had scores of 3 or greater and identified areas perceived as contributing to a good educational environment by students: items 2, 10, 15, 40 and 45 were the five that received the highest mean scores.
Mean response values for 30 (60%) of the individual items were lower for students of year 5 than for students of years 3 or 4 (Table 2).
Table 1. Mean responses to DREEM items for medical students of the various clinical years
Subscales and overall mean scores
The 5 subscales and overall mean scores are shown in Table 2. The subscale with the highest mean score was students' perceptions of their academic skills: 22.3 ± 4.1 out of 32, corresponding to 69.7% of the maximum score. The lowest mean score was for the students' perceptions of the school’s social environment: 15.9 ± 4.0 (56.8% of the maximum score). Scores observed for year 5 students were statistically significantly lower in subscales 1,2,4 and 5 compared to years 3 and 4 (p<0.001).
The overall mean score for the 50 items was 127.5 ± 20.9 (63.8%). Students of year 4 had the highest mean score with 132.5 ± 19.7 and the lowest mean score was found for students in year 5 with 119.3 ± 20.2 (p<0.0001).
Table 2. A comparison of subscales and full DREEM inventory among undergraduate medical students
Internal consistency findings are shown in Table 3. The 50-item DREEM Spanish version was found highly reliable, with an alpha coefficient of 0.91 in year 3, 0.9 in year 4 and 0.91 in year 5 (p=NS for difference in alphas across years). The internal consistency of the inventory among all 297 respondents was 0.91. Internal consistencies of the five subscales were found to reflect adequate reliability, although the coefficient for Perceptions of the Social Environment was lower than that of the other subscales indicating its poorer reliability (Table 3).
Table 3. Cronbach’s alpha coefficients for subscales and full DREEM inventory
The DREEM questionnaire was found to be an internally reliable instrument for measuring students’ views of the educational climate of a Chilean university. The 50-item DREEM Spanish version was found to have a similar reliability coefficient (Cronbach alpha) to that obtained by Deza in the first Spanish version (58-item DREEM), administered in Argentina (Roff et al., 1997). However, the alpha coefficients of each subscale were lower than the overall reliability, ranging from 0.58 to 0.75. The lowest reliability was obtained by the Students’ Social Self-Perceptions. Mayya & Roff (2004) obtained very similar alpha coefficients for the subscales and full inventory in India. Psychometric analyses of the instrument were carried out by de Oliveira Filho et al. (2005) in postgraduate training programmes in Brasil. They found the DREEM inventory valid, with high discriminant and concurrent validities and high reliability with an alpha coefficient of 0.93 and generalizability coefficient of 0.95.
The individual items’ mean scores offered an opportunity to improve certain areas of the PUCMS educational environment that were perceived by students as weaknesses, including the support systems for students who become stressed, cheating as a problem in the school of medicine, teaching that over-emphasized factual learning, and poor accommodations for students. Analysis of individual items also proved to be a useful tool for recognizing areas where the educational environment was good, such as the quality of the school’s teachers in terms of their medical knowledge and teaching abilities.
The analysis of the subscales showed a high mean score for students’ perceptions of their teachers. This finding is consistent with the item analysis and may reflect the successes of the staff development efforts of the Centre for Medical Education. The other subscales that reflected a positive educational environment were students’ perceptions of their academic skills and their perceptions of the learning atmosphere. On the other hand, students’ perceptions of learning and of the social environment revealed domains of the educational environment that could stand improvement. The new medical school building and the biomedical library were evaluated highly by the students. However, the students ratings challenged the medical school to provide better social activities and accommodations for students, which could be related to infrastructure deficiencies.
Four of the five subscale mean scores of students in year 5 were lower than the mean scores for students in years 3 and 4. This may be due to differences in the physical environments, proportion of clinical activities and proportion of mature students or validity of the instrument used to measure the educational environment (Wangsaturaka, 2005; Roff et al., 2005). Al-Hazimi et al. (2004) analysed the educational environment at three schools of medicine: two with traditional curricula in Saudi Arabia and the Republic of Yemen, and a third, Scottish medical school with an innovative curriculum as defined by the GMC in the UK (1993). The Scottish medical school obtained an overall DREEM score of 139, significantly higher than the scores obtained by the traditional universities, which 100 and 107. However, an innovative curriculum design is not the only factor related to a good educational environment. Curriculum changes are usually undertaken in order to improve the overall learning environment for students; the process, however, is often stressful for both students and faculty (Roff et al, 1997; McAleer et al., 1998).
The PUCMS obtained an overall mean score of 127.5, which is considered an indication of a generally positive educational environment. The DREEM scores obtained by students in year 3 and 4 were comparable with DREEM scores reported by UK medical schools with curricula aligned with the recommendations suggested by the GMC in Tomorrow’s Doctors (Roff et al., 1997; Al-Hazimi et al., 2004; Varma et al., 2005). Roff (2005) pointed out that “It remains to be established if the type of curriculum offered by a given school can be ‘detected’ by the DREEM”. However, a higher DREEM score indicates a more student-centred curriculum, and schools with traditional curricula commonly score less than 120 (Al-Hazimi et al., 2004; Varma et al., 2005).
DREEM has proven to be a useful tool for identifying the strengths and weaknesses of the PUCMS’s curriculum. In 2006-2007 the core curriculum was reduced to avoid curriculum overload and the proportion of the curriculum devoted to ambulatory clinical practice was increased. In the integrated clinical course in year 4, the assessment system was improved to assess higher cognitive levels in its multiple-choice question examinations, expanded in its use of Objective Structured Clinical Examinations (OSCE) and it introduced portfolios in order to minimize assessments of students’ factual recall of information. A focus group of students and teachers of the PUCMS was conducted in 2007 in the context of the school’s accreditation process. The information provided by the DREEM questionnaire was analysed in depth. Focus groups were used to clarify the underlying causes for areas with poor scores, which varied by student-year group. Stress resulting from experiences on clinical placements was highlighted by some students and cheating in written examinations was confirmed as a general problem, although not for OSCE or portfolios. The qualitative data have substantially enhanced questionnaire interpretation, allowing us to undertake remedies to address common causes for student dissatisfaction. This approach of combining DREEM results and qualitative analysis using focus groups and open ended questions has been recommended by Whittle et al. (2007) as an approach to improving the educational environment and the overall quality of students’ education. In the future, the DREEM questionnaire could again be administered in our school to measure the impact of current and future changes in the curriculum on students’ perceptions of their educational environment.
Based on this study’s findings, we recommend the 50-item DREEM Spanish version as a reliable instrument for other Chilean and Latin American schools of medicine with Spanish-speaking students. It can be used to identify an institution’s strengths and weaknesses, make comparisons of students’ perceptions of educational environments within an institution (like identifing changes in perceptions of over time) and between students of different medical schools. It can also be used to assess the correlation of the overall mean score of the DREEM questionnaire with students’ academic performance reflected in their grade point averages, and serve as a tool to identify students who are likely to be academic achievers and those who are at risk of poor academic performance (Roff, 2005).
AL-HAZIMI, A., ZAINI, R., AL-HYIANI, A., HASSAN, N., GUNAID, A., PONNAMPERUMA, G., KARUNATHILAKE, I., ROFF, S., McALEER, S. & DAVIS, M. (2004). Educational environment in traditional and innovative medical schools: a study in four undergraduate medical schools. Education for Health, 17(2), 192-203.
ASSOCIATION OF AMERICAN MEDICAL COLLEGES (1984). Physicians for the twenty first century. Report of the Project Panel on the General Professional Education of the Physician and College Preparation for Medicine, Journal of Medical Education, 59(11 Pt 2), 1-208.
BITRAN, M., WRIGHT, A., ZUNIGA, D., MENA, B., VELASCO, N. & MORENO, R. (2002). Improvement of medical student's academic performances in times of curricular reform. Revista Médica de Chile, 130(4), 437-445.
CRONBACH, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
DE OLIVEIRA FILHO, G.R., VIERA, J.E. & SCHONHORST, L. (2005). Psychometric properties of the Dundee Ready Educational Environment (DREEM) applied to medical residents. Medical Teacher, 27(4), 343-347.
ENRIQUEZ, L.O. & MENA, C.B. (2005). Professional qualification. Conditions for the quality assurance of medical education and the conditions for reciprocal trust. The experience and vision of the Association of Medical Schools in Chile (ASOFAMECH). Revista Médica de Chile, 133(4), 483-494.
FELETTI, G.I. & CLARKE, R.M. (1981). Review of psychometric features of the medical school learning survey. Medical Education, 15(2), 92-96.
GENERAL MEDICAL COUNCIL (1993). Tomorrow’s Doctors. Recommendations on Undergraduate Medical Education, London, General Medical Council.
GENN, J. (2001). AMEE Medical Education Guide No 23 (Part I): Curriculum, environment, climate, quality and change in medical education-a unifying perspective. Medical Teacher, 23(4), 337-344.
GENN, J. & HARDEN, R.M. (1986). What is medical education here really like? Suggestions for action research studies of climates of medical education environments. Medical Teacher, 8(2), 111-124.
HARDEN, R.M., SOWDEN, S. & DUNN, W.R. (1984). Educational strategies in curriculum development: the SPICES Model. Medical Education, 18(4), 284-297.
HUTCHINS, E.B. (1961). The 1960 medical school graduate; his perception of his faculty, peers and environment. Journal of Medical Education, 36, 322-329.
HUTCHINSON, L. (2003). The ABC of learning and teaching: educational environment. British Medical Journal, 326(7393), 810-812.
MARSHALL, R.E. (1978). Measuring the medical school learning environment. Journal of Medical Education, 53(2), 98-104.
MAYYA, S. & ROFF, S. (2004). Students’ perceptions of educational environment: a comparison of academic achievers and under-achievers at Kasturba Medical College, India. Education for Health, 17(3), 280-291.
McALEER, S., ROFF, S., HARDEN, R.M., AL-QAHTANI, M., UDDIN, A.A., DEZA, H. & GROENEN, G. (1998). The medical education environment measure; a diagnostic tool. Medical Education, 32(2), 217.
MORENO, R. & VELASCO, N. (1994). Curricular changes in the School of Medicine of the Pontificia Universidad Católica de Chile. Revista Chilena de Cirugía, 46, 333-336.
NATIONAL MEDICAL EXAMINATION, CHILE (2005). Resultados Generales del Examen Medico Nacional 2005. Asociación de Facultades de Medicina de Chile (ASOFAMECh), Santiago. Retrieved August 25, 2007 from http://www.emn.cl/resultados.html
ROFF, S., McALEER, S., HARDEN, R.M., AL-QAHTANI, M., UDDIN, A.A., DEZA, H., GROENEN, G. & PRIMPARYON, P. (1997). Development and Validation of the Dundee Ready Education Environment Measure (DREEM). Medical Teacher, 19(4), 295-299.
ROFF, S. (2005) The Dundee Ready Educational Measurement (DREEM)-a generic instrument for measuring students’ perceptions of undergraduate health professions curricula. Medical Teacher, 27(4), 322-325.
ROFF, S., McALEER, S. & SKINNER, A. (2005). Development and validation of an instrument to measure postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Medical Teacher, 27(4), 326-331.
ROSSO, P., VELASCO, N. & MORENO, R. (1997). Undergraduate curriculum reform at the Pontifical Catholic University Medical School: aims, methodology and advance status. Revista Médica de Chile, 125(7), 796-807.
ROTHMAN, A.I. & AYOADE, F. (1970). The development of learning environment: a questionnaire for use in curriculum evaluation. Journal of Medical Education, 45(10), 754-759.
SANCHEZ, I., RIQUELME, A., MORENO, R., MENA, B., DAGNINO, J. & GREBE, G. (2007). Revitalising medical education: the School of Medicine at the Pontificia Universidad Católica de Chile. The Clinical Teacher, 5, 1-5.
VARMA, R., TIYAGI, E. & GUPTA, J.K. (2005). Determining the quality of educational climate across multiple undergraduate teaching sites using the DREEM inventory. BioMed Central Medical Education, 5(1), 8.
WORLD FEDERATION OF MEDICAL EDUCATION (1993). The changing medical profession: implications for medical education. Medical Education, 27(3), 291-296.
WORLD FEDERATION OF MEDICAL EDUCATION (1998). International standards in medical education: assessment and accreditation of medical schools' - educational programmes: a WFME position paper. Medical Education, 32(5), 549 - 558.
WANGSATURAKA, D. (2005). Development of Learning Climate Measures for Thai Medical Education. PhD thesis, Faculty of Medicine, Dentistry and Nursing, University of Dundee.
WHITTLE, S.R., WHELAN, B. & MURDOCH-EATON, D.G. (2007). DREEM and beyond; studies of the educational environment as a means for its enhancement. Education for Health, 20(1):7.