|ORIGINAL RESEARCH ARTICLE
|Year : 2014 | Volume
| Issue : 1 | Page : 28-33
Exploring the trustworthiness and reliability of focus groups for obtaining useful feedback for evaluation of academic programs
Ayesha Rauf1, Lubna Baig2, Tara Jaffery3, Riffat Shafi4
1 Department of Health Professions Education, Shifa College of Medicine, Islamabad, Pakistan
2 Department of Medical Education, King Saud Bin Abdulaziz University of Health Sciences, Riyadh, Saudi Arabia
3 Department of Medicine, Shifa College of Medicine, Islamabad, Pakistan
4 Department of Physiology, Shifa College of Medicine, Islamabad, Pakistan
|Date of Web Publication||11-Jun-2014|
Dr. Ayesha Rauf
Department of Health Professions Education, Shifa College of Medicine, H8/4 Pitras Bukhari Road, Islamabad
Source of Support: None, Conflict of Interest: None
Background: At Shifa College of Medicine, evaluation is an integral part of the curriculum. We used focus groups as a tool to obtain in-depth information regarding students' experience with the new integrated, system- based curriculum. The purpose of this study was to explore the usefulness of focus groups in identifying important issues for curriculum improvement and to explore the trustworthiness and representativeness of data obtained through this strategy. Methods: In 2012, we used focus groups to seek feedback from students regarding their experience with the integrated curriculum. One course of each of the three preclinical years was selected for this purpose. Three parallel focus groups were conducted for each selected course. Each focus group was audio recorded, and the moderator and a volunteer student took additional notes during the session. The audio recordings were transcribed and data obtained from the three sources were analyzed, coded, and categorized independently by three investigators. Both manifest and latent themes were identified, using an inductive approach. Final agreement on themes was reached by comparison of the independently done thematic analysis by the three researchers. Reliability of data was established by comparing responses from the three parallel focus groups of each course. Trustworthiness of inferences was ensured by multiple coding, audit trail and member checking with focus group participants who reviewed the themes for validity. Results: Most of the data on students' perceptions of their courses from each of the parallel groups were in agreement: Similar themes were seen within groups of the same class as well as across the three preclinical years. Conclusion: Focus groups can be a useful tool for collecting trustworthy and reliable information through a process that promotes interaction among student participants. They can support quantitative data from students and be used to support curriculum reform.
Keywords: Course evaluation, focus groups, qualitative studies
|How to cite this article:|
Rauf A, Baig L, Jaffery T, Shafi R. Exploring the trustworthiness and reliability of focus groups for obtaining useful feedback for evaluation of academic programs. Educ Health 2014;27:28-33
|How to cite this URL:|
Rauf A, Baig L, Jaffery T, Shafi R. Exploring the trustworthiness and reliability of focus groups for obtaining useful feedback for evaluation of academic programs. Educ Health [serial online] 2014 [cited 2019 Dec 5];27:28-33. Available from: http://www.educationforhealth.net/text.asp?2014/27/1/28/134303
| Background|| |
Accrediting bodies have mandated continuous monitoring and evaluation of programs using a variety of strategies, collecting multiple types and sources of data including student and teacher feedback, and then closing the loop through improvements made for the benefit of the students and the stakeholders based on the data. ,
It has been recommended that evaluation should be an inherent part of teaching and learning and should be etched in the philosophy of curriculum development. , The changes made should be evident and the stakeholders including those providing feedback should be assured that their opinion is valued.
In this context, student feedback is a valuable source of information for taking measures regarding curricular reform, faculty promotions, resource allocations, and improvement in the teaching-learning cycle.  This feedback is very important for reflective teaching as it helps to refine teaching and make teachers better professionals. ,
The most common method used in obtaining students' feedback has been structured questionnaires.  If an end-of-module questionnaire is able to capture all important points, then this approach is adequate; however, if the information gathered through questionnaires does not serve the purpose, the institute should use appropriate tools that are useful in their own school context.  Qualitative data can be useful for promoting positive and desirable improvements in the curriculum. 
Focus groups are a popular method for collecting qualitative data, as the method capitalizes on the interaction of the group members, , a feature that differentiates it from other methods of qualitative data collection.  It provides an opportunity for the participants to reflect upon their responses hence laying emphasis on analysis of information rather than only data gathering. , The information gathered in a focus group is subject to the phenomenon of 'groupthink' wherein interactions within the group also become the subject of analysis.  The questions are not addressed to each individual, and the data comes from the group interaction.  The earliest use of focus groups was made by Robert Merton, a sociologist and has been used excessively in market research,  whereas only later in the 90s, its efficacy was realized in health sciences research.
Curriculum evaluation, through student and faculty feedback is an integral part of our curriculum at Shifa College of Medicine. Initially a closed-ended questionnaire based on Likert scores was used to obtain feedback. Later, this was changed to open-ended questions, based on broad themes addressing essential aspects of the curriculum module. We recognized a need for qualitative data from students to clarify their issues. Focus groups were identified because of their proven worth in collecting qualitative data. Focus group discussions with students were formally initiated at the end of each module as a method for obtaining more comprehensive feedback related to each module.
Through this study, we aim to explore the value of focus groups for obtaining data for evaluating academic programs and suggesting improvements in the curriculum while it is 'fluid'.  We also aim to establish the trustworthiness of the data obtained from focus groups.
SCM introduced the Integrated Contextual Modular Curriculum in 2008. The curriculum at SCM is modular, integrated, and divided into three spirals. Spiral one includes academic years I and II. Spiral two is in year III and spiral three falls in clerkship years IV and V.
| Methods|| |
This was an exploratory descriptive study to evaluate the usefulness of focus groups as a strategy to understand students' perceptions of their academic courses. Three courses, one from each year of study, were identified from which to collect student feedback through focus groups: Year I, Cell and Molecular Biology; Year II, Maxillofacial/Special Senses; and Year III. Gastrointestinal tract.
In addition to the principal investigator, two other faculty members were identified as coresearchers as well as moderators. These three faculty members were familiar with the process of conducting focus groups and were not part of the courses that were included in this study.
We grounded our guiding questions for the focus groups based on data available from open and closed ended questionnaires that were previously used for module feedback in our school.
Three parallel focus groups of 15 students each were conducted at the end of each of the three courses (nine groups in total). To minimize variability in facilitator style, the same three moderators conducted the focus groups. From a class of 100 students, 45 participants were randomly selected using Excel software and divided into three groups. The focus groups were conducted simultaneously at the end of each course at three different venues by the three researchers/moderators.
Participants were informed of the purpose of the study at the beginning of each focus group. The facilitators took verbal consent from the participants and they assured them of total anonymity and asked to express themselves freely.  The facilitator used the predefined guiding questions to conduct the focus group. The average time planned for the focus groups was 50 min.
Qualitative data from each group was collected by three methods: Notes were taken by the facilitator and by a student volunteer from the group, and audio recording were made and later transcribed.
All data was made anonymous and any names mentioned were edited during transcription. The three sources were filed together for multiple coding later.
Trustworthiness of the data was ensured through multiple coding.  At a scheduled meeting, each facilitator reviewed the three sources of information from their focus groups. Common themes from the three documents were independently identified by each facilitator. These were then discussed and a final document was compiled for each class, representing common themes from the three focus groups for the class. Any disagreements in the themes were discussed and final consensus was achieved.
The first author compiled all the common themes that were identified. With the consensus of all three, perceptions relevant to curricular reform were selected and tabulated along with selected student quotes thus making a structured categorization matrix for common themes from the three parallel groups of one module. In addition, the feedback or perceptions common to all the three academic years with regard to the specific categories were plotted as subthemes in another categorization matrix.
The discussion was set to encourage naturalistic data that was cross-checked through note taking by three different individuals, specifically the student and moderator to identify latent themes through recordings of their impressions and transcription by coresearcher. The thematic categorization that emerged very closely reflected the guiding questions for the focus groups. Member checking was done with students participating in the focus groups. Saturation of data was sought by constituting three parallel focus groups of each module and striving for adequate sample sizes in each group.
| Results|| |
Attendance in focus groups has been a matter of concern,  something which was also identified in our study. Ten to thirteen students attended the focus groups for the first two student years and five to six students attended focus groups for year III.
Our primary objective was to establish the usefulness of focus groups as a strategy to obtain qualitative data for evaluation of academic programs. We aimed to explore students' perceptions to identify important aspects of the academic program for modifications and improvement. The themes identified from the focus group discussions through the investigators' analyses included module organization, learning strategies, facilitation, and learning resources.
Results of focus group for cell and molecular biology module year I
[Table 1] gives the details of the parallel focus groups. In general, there was consensus among all three groups regarding module organization, the learning strategies and need for formative assessments. The students appreciated small groups but some felt that facilitators should be given workshops to improve their expertise. The general note of the session was of anxiety and discussions demonstrated an apprehension toward the absence of proper logistics, extracurricular activities, and free time for students. All concerns highlighted students' state of uncertainty and transition from high school to medical school.
|Table 1: Categorization matrix for common themes of cell and molecular biology module- year I|
Click here to view
Results of focus group for maxillofacial and special senses modules year II
[Table 2] demonstrates that there was consensus among the three groups regarding most of the identified themes. The data collected reflected that the students gave more importance to learning resources than to the scheduling. They also valued senior faculty support. It was clear that there was a general dissatisfaction with some faculty members and the students demanded that the faculty should respect students. Their search for professional identity was evident.
|Table 2: Categorization matrix for common themes of Maxillofacial and Special Senses Module Year II|
Click here to view
In general, the students were vocal and forthcoming with their feedback and supported each other while discussing sensitive areas of dissatisfaction with some faculty members.
Results of focus group for gastro intestinal system module year III
The third year class was different from the first two classes, because they experienced additional unexpected changes in curriculum during their third year. These changes related to the change in organization of the pathology content imposed by examining bodies.
[Table 3] reflects consensus between all three focus groups regarding all issues related to pathology. They were vocal and complained as a group. The lack of participation by most students reflected their cynicism. They were focused on their need for authenticity of information provided to them and placed all responsibility of their learning on the faculty.
|Table 3: Categorization matrix for common themes of Gastrointestinal Tract Module- Year III|
Click here to view
Commonality across all three preclinical years
It was interesting to discover, through comparing the content/subthemes across years, that the basic expectations of students across the three clinical years were the same. Their perceptions of the quality of the learning strategies, availability of resources, and proper guidance and support regarding assessments were similar.
Similar themes were identified across all three parallel focus groups of each class, which demonstrated saturation of data. In addition, [Table 4] demonstrates the commonalities regarding themes ac ross all three preclinical classes.
|Table 4: Structured categorization matrix of themes common to all preclinical years|
Click here to view
| Discussion|| |
The purpose of this research at SCM was to explore the usefulness of focus groups in identifying important issues for curriculum improvement and to explore trustworthiness and representativeness of data. Focus groups have been defined as 'a set of central tendencies with many useful variations that can be matched to a diversity of research purposes'.  In our study, our purpose was to identify areas for improvement in the courses. Huxman et al. compared various methods of collecting student feedback. These included questionnaires, rapid fire questions, and focus groups. It was concluded that the basic difference and richness of one data over the other was due to the questions being open or closed ended.  At SCM, the move from questionnaires to focus groups for student feedback was in search of this rich qualitative data. However, because focus groups are a relatively uncommon strategy for academic program evaluation, the authenticity and trustworthiness of the data needed to be established.
We noted that the emerging conformity of views was a result of group interaction and not individual perceptions and opinions.  This is probably the reason for the similar feedback received from all three groups of the same academic year and the general feedback of students across classes as well. This phenomenon of 'group thinking' makes the focus group a useful strategy to obtain in-depth feedback regarding academic programs. Our research design based on having three focus groups for each module helped ensure the reliability of our data.  There was agreement on most themes identified by the three focus groups for each module. This supports the representativeness of the information for program evaluation gathered through focus groups.
Our groups had students of both genders. There was no issue of dominance observed in our groups and the feeling of consensus that developed within each group demonstrated positive group dynamics. Each guiding question, as it was presented before the group, initiated a discussion and the interaction generated a consensus of opinion, which was then confirmed and documented. This conformity of consensus was an emergent property of the group context and was not an aggregation of individual participants' opinions. 
The descriptive validity of the inferences drawn in this study is strengthened by the presence of an audit trail, cross validation of data from each of the focus groups for one course through the notes of the moderator, the volunteer student, and the transcription from audio recordings by a second faculty member.  Iterative thematic analysis, multiple coding, and member check were carried out to preserve the interpretative validity of our inferences. 
As with other similar studies, we realize that these results depended upon the skills, insights, critical analysis, and signature style of the researchers. 
Students' perceptions regarding the importance given to students' feedback regarding academic programs has been explored at SCM.  Focus groups have been used as an integral component of program evaluation to identify issues related to curriculum planning and implementation.  Though laborious and intensive, the focus groups have proved to be a useful tool to be used along with other tools for program evaluation. 
Our study, similarly identified issues related to each class as well as common issues across all three preclinical classes. The in-depth inquiry inherent to the focus group strategy provided answers and clarifications for concerns identified in previously conducted student questionnaires. Latent themes identified by the moderators also gave us a feel regarding concerns not voiced explicitly by the participants.
Closing the loop through emphasizing action taken after the process of evaluation is essential  and we achieved this by disseminating the data collected from the focus groups to the course directors and the teams for curricular modifications wherever feasible.
Strengths and Weaknesses of Our Study
This study addressed an ongoing issue and helped to give credibility to an ongoing activity at our workplace. Because of the homogeneity of our focus groups, which included students from the same level of education, social background, and level of expertise - they belonged to the same class, they were confident in voicing their opinions. An effect of polarization was seen, which was pronounced because of this homogeneity, the prevalent viewpoint was exaggerated and the divergent opinion was suppressed. This was a concern that was addressed in our study through conducting three focus groups from the same class simultaneously. This added to the reliability of the data as it diluted the interactions resulting in this polarization by a different level of interaction within the other groups. 
Our student feedback of the chosen courses was collected through a single strategy. If we had used other strategies in addition to focus group, the data could be triangulated to strengthen the claim that all important themes were accurately identified.
Feedback from individual classes provided useful insight into the preferences of that year, which were similar across the three focus groups for that year. The consensus among the participants within and across groups increased as they moved to senior classes showing a greater degree of cohesion in comparison to the first year. It was interesting to note irrespective of the learners' level, certain preferences and perceptions about quality of curriculum were similar, a fact that would help curriculum developers work toward making reforms across the 5-year curriculum.
The process of organizing, conducting, and analyzing data from focus groups is resource intensive and laborious and may exceed the resources of some schools. However, the richness and usefulness of that data that it gathers makes it a worthwhile tool for information gathering. Focus groups can be used in conjunction with quantitative methods and useful for 'telling the story' behind the numbers as in a mixed method design. 
| Conclusion|| |
The data gathered from the focus groups helped to identify students' preferences and areas for improvement. The observations made provided insight into perceptions that were not explicit. Some perceptions were found to be common among students irrespective of their academic level and these preferences can be used across all levels of education and made an integral part of the curriculum.
This study gives empirical evidence that focus groups with a random selection of student from a population can be broadly representative of students and the data obtained can be used along with information from other sources for program evaluation and curriculum changes.
The richness of data obtained through the use of focus groups for student feedback further validates this approach for program evaluation. Owing to its demands on the administration, it should be used judiciously to identify sensitive issues otherwise not obvious from regular means of feedback or to corroborate other quantitative data.
| Acknowledgments|| |
Ms Humera Yasin Khan for her support and facilitation of all activities regarding this project.
Ethical approval: Taken from the official body Internal Review Board of the institute.
| References|| |
|1.||Gibson KA, Boyle P, Black DA, Cunningham M, Grimm MC, McNeil HP. Enhancing Evaluation in an Undergraduate medical education program. Acad Med 2008;83:787-93. |
|2.||Vassar M, Wheeler DL, Davison M, Franklin J. Program evaluation in medical education: An overview of the utilization-focused approach. J Educ Eval Health Prof 2010;7:1. |
|3.||Shodokshi O, Sukkar MY. An approach to medical curriculum evaluation. Med Educ 1988;22:426-32. |
|4.||Iqbal M, Khizar B. Medical students perceptions of teaching evaluations. Clin Teach 2009;6:69-72. |
|5.||Huxham M, Laybourn P, Cairncross S, Gray M, Brown N, Goldfinch J, et al. Collecting student feedback: A comparison of questionnaire and other methods. Assess Eval Higher Educ 2008;33:675-86. |
|6.||Archer JC. State of the science in health professional education: Effective feedback. Med Educ 2010;44:101-8. |
|7.||Sim J. Collecting and analyzing qualitative data: Issues raised by the focus group. J Adv Nurs 1998;28:345-52. |
|8.||Rothwell E. Analyzing focus group data: Content and interaction. J Spec Pediatr Nurs 2010;15:176-80. |
|9.||McLafferty I. Focus group interviews as a data collecting strategy. J Adv Nurs 2004;48:187-94. |
|10.||Barbour RS. Making sense of focus group. Med Educ 2005;39:742-50. |
|11.||Chioncel NE, Van Der Veen RG, Wildemeersch D, Jarvis P. The validity and reliability of focus groups as a research method in adult education. Intern J Lifelong Educ 2003;22:495-517. |
|12.||Loriz LM, Foster PH. Focus groups: Powerful adjuncts for program evaluation. Nurs Forum 2001;36:31-6. |
|13.||Lingard L, Kennedy TJ. Qualitative reseach methods in medical education. In: Swanwick T, editor. Understanding medical education: Evidence, theory and practice. West Sussex: Wiley Blackwell; 2010. p. 323-35. |
|14.||Webb C, Kevern J. Focus groups as a research method: A critique of some aspects of their use in nursing research. J Adv Nurs 2001;33:798-805. |
|15.||Elo S, Kyngas H. The qualitative analysis process. J Adv Nurs 2008;62:107-15. |
|16.||Longitudinal Evaluation Project-University of New Mexico School of Medicine. Focus groups on curriculum and program evaluation. Acad Med 1996;71:519. |
|17.||Nabors LA, Ramos V, Weist MD. Use of focus groups for evaluating programs for children and families. J Educ Psychol Consult 2001;12:243-56. |
[Table 1], [Table 2], [Table 3], [Table 4]
|This article has been cited by|
||Using Resident and Faculty Focus Groups to Obtain Stakeholder Input during the ACGME Self-study
| ||Kathryn M. Huggins,Angelina R. Sprewell,Dominique C. Elmore,Meagan W. Shepherd,Tracy L. LeGrow,Marie D. Frazier,Susan L. Flesher |
| ||Pediatric Quality and Safety. 2019; 4(4): e186 |
|[Pubmed] | [DOI]|
||Ethical challenges in primary care: a focus group study with general practitioners, nurses and informal caregivers
| ||Ildikó Gágyor,Arndt Heßling,Susanne Heim,Andreas Frewer,Friedemann Nauck,Wolfgang Himmel |
| ||Family Practice. 2018; |
|[Pubmed] | [DOI]|
||Developing an online health literacy curriculum for two German universities: a key stakeholder approach
| ||Sandra Vamos,Paul Yeung,Steffen Schaal,Kirsten Schlüter |
| ||Global Health Promotion. 2018; 25(3): 43 |
|[Pubmed] | [DOI]|
||Sickle cell anaemia and the experiences of young people living with the condition
| ||Nicole Foster,Michelle Ellis |
| ||Nursing Children and Young People. 2018; |
|[Pubmed] | [DOI]|