|ORIGINAL RESEARCH ARTICLE
|Year : 2018 | Volume
| Issue : 2 | Page : 95-102
Teaching and evaluating smartphone applications: The effectiveness of a curriculum expansion
Susan G Rodder1, Tiffany B Kindratt2, Chunyun Xiao1, Venetia Orcutt2, Courtney Koch2, Kelley Mcilvaine2, Mary Alice Neville2
1 Department of Clinical Nutrition, UT Southwestern School of Health Professions, UT Southwestern Medical Center, Dallas, Texas, USA
2 Department of Physician Assistant Studies, UT Southwestern School of Health Professions, UT Southwestern Medical Center, Dallas, Texas, USA
|Date of Web Publication||30-Nov-2018|
Susan G Rodder
Department of Clinical Nutrition, UT Southwestern School of Health Professions, UT Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, Texas
Source of Support: None, Conflict of Interest: None
Background: Mobile health (mHealth) technology is increasingly utilized to support lifestyle recommendations through nutrient and blood pressure tracking. As patients pose questions regarding the validity and use of this technology, curriculum targeting mHealth technology is essential for the future health professionals. This study evaluated the effectiveness of a curriculum expansion which addressed mHealth technology provided to physician assistant (PA) and clinical nutrition (CN) students enrolled in an academic health center. In addition, the validity of the mobile application (app), MyNetDiary, was determined. Methods: A smartphone application appraisal tool, based on scientific recommendations, was developed. Students were taught how to use this tool to evaluate mobile apps. Students received instruction on providing patient education on mobile apps used to track calories and nutrients and mobile medical apps to measure blood pressure. Pre-/post-surveys and objectively structured clinical examinations measured students' confidence and abilities in teaching patients to use MyNetDiary and Withings Health Mate apps. Wilcoxon rank sum tests evaluated statistical significance. Validity of nutrient estimates was determined using Spearman correlations. Results: Confidence levels improved significantly on all items measured for both PA and CN students (P < 0.001). During the objectively structured clinical examination, all students demonstrated effective communication skills with 98.4% successfully demonstrating of how to enter foods into the MyNetDiary app and 94.3% connecting the blood pressure cuff with the withings app. Significant correlations were found when comparing MyNetDiary to SuperTracker (all P < 0.001). Discussion: This study investigated and demonstrated the effectiveness of an expanded curriculum designed to enhance students' confidence and skills in providing lifestyle counseling incorporating the use of mHealth technology.
Keywords: Lifestyle counseling, mobile applications, mobile health, mobile health technology, mobile medical applications, nutrient estimates, nutrition education, objective structured clinical exam, smartphone applications, students
|How to cite this article:|
Rodder SG, Kindratt TB, Xiao C, Orcutt V, Koch C, Mcilvaine K, Neville MA. Teaching and evaluating smartphone applications: The effectiveness of a curriculum expansion. Educ Health 2018;31:95-102
|How to cite this URL:|
Rodder SG, Kindratt TB, Xiao C, Orcutt V, Koch C, Mcilvaine K, Neville MA. Teaching and evaluating smartphone applications: The effectiveness of a curriculum expansion. Educ Health [serial online] 2018 [cited 2020 May 30];31:95-102. Available from: http://www.educationforhealth.net/text.asp?2018/31/2/95/246752
| Background|| |
Primary care physicians have limited time for lifestyle counseling in the management of weight loss, diabetes, and heart disease. The use of mobile health (mHealth) technology, or mHealth, is increasingly utilized to support lifestyle recommendations through the measurement and tracking of calories, daily steps, glucose, and blood pressure., Currently, over 165,000 health-related mobile applications (apps) are available and approximately one-third of physicians recommend their use to patients. However, physicians and other health-care providers have little guidance for the critical evaluation of mobile apps before their recommendation. In addition, physician assistant (PA) and clinical nutrition (CN) students who may provide the majority of lifestyle counseling once in practice have little or no instruction targeting the delivery of effective patient education on the use of this technology.
mHealth includes the use of mobile medical apps (MMAs) and mobile apps in medical care. Both types of apps are software programs that run on smartphones or other mobile communication devices. A MMA can be either an accessory to a regulated device or an app which transforms a mobile platform into a regulated device. An example of a MMA which is an accessory to a regulated device is an app which supports a blood pressure cuff. Whereas, an example of a MMA that transforms a mobile platform into a regulated device is an app intended to analyze and interpret electrocardiogram waves to detect heart rhythm irregularities. Beginning in 1997, the Food and Drug Administration (FDA) initiated an approval process for MMAs that meet the regulatory definition of a device.,
Unlike MMAs, mobile apps are not evaluated or approved by the FDA. Mobile apps are designed for general patient education to facilitate access to commonly used reference information without the intended use in the diagnosis or treatment of disease. Examples of mobile apps include apps used to monitor calorie intake and or physical activity. As the use of mobile apps has expanded, the quality of evidence used to create them has come into question.,,
The lack of regulation of mobile apps by the FDA or certifying agencies makes it challenging for health-care professionals to recommend specific mobile apps; however, patients continue to use these mobile apps to track calories, daily steps, and blood pressure and they seek reassurance that the mobile apps they are using are appropriate. In addition, inquiries from clinicians on which mobile app to recommend calls for the development of a critical appraisal tool to evaluate the mobile apps. The use of an appraisal tool must be further supported through the development of curriculum for clinicians regarding effective patient education on how to use this technology.
The CN and PA programs at our institution have provided an interprofessional nutrition (Life Habits) curriculum since 1990. The Life Habits curriculum has evolved over time to reflect interprofessional information targeting weight management, diabetes, and cardiovascular health. Initially, Life Habits included didactic lectures only; however, the practical application of patient education as experiential learning activities was added to enhance student knowledge and skills and to prepare them for working in the clinical outpatient setting. Current experiential learning activities include: (1) a health fair to improve student knowledge and skills in providing patient education regarding the implementation of dietary recommendations, (2) the provision of a heart-healthy meal to present foods which promote the lowering of cardiac risk factors, and (3) the completion of a 2-day food record by PA students with subsequent nutrient analysis by CN students to demonstrate how patients can be educated to make dietary improvements. PA food records, similar to patient food records, have been traditionally recorded on paper and analyzed by software programs, but with the increasing use of mobile apps, the Life Habits curriculum was expanded to educate students on the use of this technology to accomplish the same end-point. This expansion included instruction on (1) the critical appraisal of mobile apps, (2) the use of two apps (MyNetDiary, Withings Health Mate), and (3) the delivery of effective patient education regarding the use of this technology.
The primary aims of this study were to develop and expand the existing interprofessional nutrition curriculum to include mHealth and to investigate changes in PA and CN students' confidence in their ability to use MMAs and mobile apps to provide lifestyle counseling in weight loss, diabetes, and heart disease. A secondary aim was to determine the validity of MyNetDiary as compared to the gold standard nutritional estimation tool, SuperTracker.
| Methods|| |
Setting and subjects
We conducted a mixed-methods quasi-experimental study with PA (n = 173) and CN (n = 78) students during the Spring semesters in 2015 through 2018.
The “Smart-Life Habits Pilot,” conducted in Spring 2015, was added to the existing nutrition curriculum to address the use of mHealth, specifically the use of the mobile app MyNetDiary. In this pilot study, PA (n = 35) and CN (n = 20) students received instruction on the use of the mobile app MyNetDiary to track food intake. As in the previous curriculum, PA students completed a 2-day food record that was then provided to their assigned CN student for entry into the software program NutriBase. Subsequently, the PA students downloaded the mobile app MyNetDiary and entered their 2-day food record intake, for comparison of nutrient estimates between MyNetDiary and NutriBase.
Following this pilot study, the Smart-Life Habits curriculum was further expanded in 2016 through 2018 to teach PA (n = 138) and CN (n = 58) students how to (1) critically evaluate mobile apps and (2) effectively instruct patients how to use this technology. To critically evaluate mobile apps, a smartphone application appraisal tool (SAAT) was developed to reflect the Life Habits curriculum content, specifically the medical nutrition therapy recommendations for the management of weight, diabetes, and heart disease.
The SAAT appraised the mobile apps through a series of close-ended questions using parameters outlined in the 2013 AHA/ACC/TOS for the Management of Overweight and Obesity in Adults, the diabetes prevention program, and modeling expert opinions on food healthfulness: A nutrition metric. In addition, several questions pertaining to the resources used for the derivation of recommended micronutrients (i.e., Vitamin C) and the number of calories expended during various forms of physical activity were included to reflect established scientific recommendations and institutions, specifically the Institute of Medicine and the compendium of physical activities. The PA and CN students used the SAAT to evaluate if a given mobile app reflected scientifically based evidence for the management of weight, diabetes, and heart disease. The SAAT is presented in [Appendix 1].
Following instruction on critical evaluation of mobile apps, all students were taught how to effectively instruct a patient how to use the mobile app MyNetDiary to track food intake and how to use the MMA Health Mate with the Withings blood pressure measurement cuff to measure blood pressure. After skilled practice sessions with each other, the students' skill acquisition was evaluated using two objectively structured examinations (OSCEs). [Table 1] provides an overview of the expanded Life Habits curriculum and the OSCE stations.
Student confidence levels
In the expanded curriculum (2016 through 2018), PA students completed pre- and post-didactic curriculum surveys evaluating changes in confidence in their skills related to the nutrition needs of patients (survey measures PA: 1–7) [Table 2] and evaluating changes in confidence in their skills related to educating patients on utilization of mHealth (survey measures PA: 8–14) [Table 2]. CN students completed pre- and post-didactic curriculum surveys evaluating changes in confidence in their skills specifically related to educating patients on utilization of mHealth (survey measures CN: 1–9) [Table 2].
|Table 2: Physician assistant and nutrition students' comparison of Life Habits pre- and post-instruction confidence level surveys, 3-year combined|
Click here to view
Objectively structured examination station scores
During the two OSCEs, standardized patients evaluated whether students demonstrated measurable skills using a rating scale of: 0 = wrong, 1 = some wrong, and 2 = correct. The specific measurable skills and corresponding OSCE scores are reported in [Table 3]a and [Table 3]b for PA and CN students, respectively.
Validation of MyNetDiary
During the pilot study, PA students entered their dietary intake into MyNetDiary and matching data were entered into the software program NutriBase by CN students to determine the validity and level of agreement between the technologies. Spearman correlations were run to assess the relationship between MyNetDiary and NutriBase (n = 35) for the following measures: calories, fat, protein, sodium, Vitamin C, calcium, iron, magnesium, and potassium. In the expanded curriculum (2016 through 2018), PA students again utilized MyNetDiary for data entry while matching data was entered into SuperTracker (n = 137) by CN students. All previously evaluated dietary measures were evaluated; fiber and Vitamins A, D were additionally evaluated. [Table 4] details the estimated nutrients and the validity of MyNetDiary compared to SuperTracker.
|Table 4: Validity of MyNetDiary with SuperTracker, 2016, 2017, 2018, and 3.year combined correlations results|
Click here to view
Data analysis was performed using STATA version 13 from StataCorp, USA. Medians and interquartile ranges were used to report instruction confidence levels for the pilot study (2015) and the further expanded curriculum (2016 through 2018) data. Wilcoxon signed rank tests were used to determine statistically significant differences between pre- and post-instruction confidence levels. Frequencies and percentages of correct scores on the OSCE stations were also determined. Spearman correlations were run to determine correlation coefficients and statistical significance for MyNetDiary validations.
| Results|| |
Pre- and post-instruction confidence levels
In the expanded curriculum (2016 through 2018), confidence levels improved significantly for all survey measures, for both PA and CN students (P ≤ 0.001), as shown in [Table 2]. Among PA students, confidence in their skills related to educating patients on utilization of mHealth improved most on the following survey measures: (11) how to use a food tracking mobile app; (12) how to track daily macro and micronutrients; and (13) how to measure blood pressure using a MMA (all P < 0.001). Among CN students, confidence in their skills related to educating patients on utilization of mHealth improved most on the following survey measures: (6) explaining how to use a mobile app to determine nutrient estimates of a recipe, how to use a photo scanner to enter a food, or enter a custom food; (7) educating a patient on how to measure blood pressure using a MMA; and (8) identifying when blood pressure readings require a referral to a PA (all P < 0.001).
Objectively structured examination station scores
[Table 3]a and [Table 3]b report the OSCE scores (frequency counts and percentages of correctly performed patient measures) for Station 1 (MyNetDiary App) and Station 2 (Withings Health Mate MMA), as scored by standardized patients for PA and CN students, respectively. In this results section, the average percentage for combined PA and CN students is discussed; separate year-by-year details for each of PA and CN students are in [Table 3]a and [Table 3]b. During Station 1, results for Basic Provider Communication Skills indicated that almost all students (98.9%) introduced himself/herself to the standardized patient, a majority of students (88.7%) used the patient's name, and almost all students (97.9%) asked the patient if they had any questions during the mock encounter. Results for Communication Skills with Apps indicated that almost all students (96.4%) demonstrated how to download MyNetDiary, 98.4% demonstrated how to enter food in the app, and 90.3% discussed the benefits of using MyNetDiary for food tracking. Results for Special Communication Skills indicated that a majority of students (76.0%) used the teach-back method by asking the standardized patient to teach back to them how to enter food in MyNetDiary; however, only 33.7% of students asked the standardized patient to verbalize the benefits of using the app at the end of the mock encounter. During Station 2, results for Basic Provider Communication Skills indicated that almost all students (98.9%) introduced himself/herself to the standardized patient, 90.8% used the patient's name, and a majority of students (88.7%) and asked the patient if they had any questions. Results for Communication Skills with Apps indicated that a majority of students (83.1%) demonstrated how to download the app, almost all students (94.3%) demonstrated how to hook up the BP cuff, a majority of students (72.9%) discussed the benefits of using the Withings Health Mate app for measuring blood pressure, and a majority of students (78%) told the standardized patients their BP results. (It is noted that the percentage of students who told the standardized patients their BP results increased from 51% (2016) to higher values of 95% and 86%, for 2016 and 2017, respectively.) However, results for Special Communication Skills indicated that only 35.2% of students used the teach-back method by asking the standardized patient to teach back to them how to hook up the blood pressure cuff, and only 11.7% of students asked the standardized patient to verbalize the benefits of using the BP cuff and Withings Health Mate app at the end of the mock encounter.
Validation of MyNetDiary
[Table 4] reports the correlation coefficients and P values for validation results for MyNetDiary compared to SuperTracker. Significant correlations were found when comparing MyNetDiary and SuperTracker for all nutrients (all P < 0.001). There were strong positive correlations between results from MyNetDiary and SuperTracker for sodium (rs = 0.879), calories (rs = 0.835), carbohydrate (rs = 0.802), protein (rs = 0.774), Vitamin C (rs = 0.744), and fats (rs = 0.730). Moderate positive correlations were found between results for fiber (rs = 0.687), calcium (rs = 0.648), potassium (rs = 0.586), Vitamin A (rs = 0.552), and Vitamin D (rs = 0.548). A low positive correlation was found between results for iron (rs = 0.471) and magnesium intake (rs = 0.450).
| Discussion|| |
The primary aims of this study were to develop and expand an existing curriculum to include mHealth and to investigate changes in PA and CN students' confidence in their ability to use MMAs and mobile apps to provide lifestyle counseling in weight loss, diabetes, and heart disease. Overall, we found that the expanded curriculum significantly increased both PA and CN students' confidence levels and enhanced their ability to teach mHealth to standardized patients as measured by pre- and post-didactic curriculum surveys, and OSCE scores, respectively. To the best of our knowledge, these results represent the findings of the first study investigating the effectiveness of an expanded curriculum designed to increase students' confidence in their ability to use MMAs and mobile apps to provide lifestyle counseling. It is noted that although other researchers discuss the use of mHealth to train students, for example, in areas of laboratory skills and diagnosis, they do not address approaches to teach students how to teach patients lifestyle counseling supported by mHealth.,, The OSCE scores clearly indicated that both PA and CN students effectively taught standardized patients to use mobile apps to track food intake and to use MMAs to measure blood pressure. Our study fills a large gap in the OSCE literature as one of the first studies evaluating students' ability to effectively communicate with and train standardized patients how to download and appropriately use critically appraised mobile apps.
A secondary aim was to determine the validity of MyNetDiary as compared to the gold standard nutritional estimation tool, for example, SuperTracker. This study found that MyNetDiary was as effective as SuperTracker for key estimation of sodium, calories, and macronutrients. Other researchers have similarly reported equivalence in nutrient estimates when comparing an analysis of food records by Tap and Track for the Apple iPod Touch with data entered by a research dietitian into a traditional nutrient analysis program. Evaluation of mobile apps against the gold standard nutrition estimation tools must continue to ensure accuracy.
A strength of this study is the demonstration of the effectiveness of an expanded curriculum to increase students' confidence in their ability to use MMAs and mobile apps to provide lifestyle counseling in weight loss, diabetes, and heart disease. A further strength of this study is the development of the evidence-based SAAT which was then subsequently readily utilized by students to make evidence-based decisions about which mobile app to recommend to their patients.
Utilization of the SAAT for critical appraisal of a wide range of mobile apps remains to be performed. Although biases may have affected changes in students' confidence levels as measured by pre- and post-didactic curriculum surveys, their actual abilities were measured by trained standardized patients, that is, OSCE scores, so as to have additional measures without bias. As with any introduction of new technology into educational settings, initial challenges did arise in 2016, for example, poor wireless synching of the Withings blood pressure cuff with the MMA during the OSCE Station 2; this challenge was fully addressed in subsequent years of this study. Finally, since this study was limited to one university albeit with two health professions, the effectiveness of a similar expanded curriculum remains to be verified.
| Conclusion|| |
With the use of mHealth being increasingly utilized to support lifestyle recommendations, and with patients increasingly seeking reassurance that the mobile apps they are using are appropriate, health professions education should provide the knowledge and skills to appraise and recommend mobile apps as part of lifestyle recommendations. This study demonstrated that an expanded curriculum focused on mHealth can improve skills and students' confidence in their use of MMAs and mobile apps to provide lifestyle counseling. The development of the SAAT, based on established scientific recommendations, provided the means to critically evaluate mobile apps designed for the management of weight, diabetes, and cardiovascular disease. It is expected that the expanded curriculum in this study is easily transferable to other health-care professions and would be ideal for nursing programs and medical school curriculum. Future research should examine how such educational experiences translate into better coordination of care and improved patient outcome.
The authors would like to thank the University of Texas Kenneth I. Shine, M. D., Academy of Health Science Education Grants Program for funding. This work was executed at the UT School of Health Professions, UT Southwestern Medical Center.
Financial support and sponsorship
The authors would like to thank the University of Texas Kenneth I. Shine, M. D., Academy of Health Science Education Grants Program for funding.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Whitehead L, Seaton P. The effectiveness of self-management mobile phone and tablet apps in long-term condition management: A systematic review. J Med Internet Res 2016;18:e97.
Jimoh F, Lund EK, Harvey LJ, Frost C, Lay WJ, Roe MA, et al.
Comparing diet and exercise monitoring using smartphone app and paper diary: A two-phase intervention study. JMIR Mhealth Uhealth 2018;6:e17.
Ventola CL. Mobile devices and apps for health care professionals: Uses and benefits. P T 2014;39:356-64.
Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-based strategies in weight-loss mobile apps. Am J Prev Med 2013;45:576-82.
Boland P. The emerging role of cell phone technology in ambulatory care. J Ambul Care Manage 2007;30:126-33.
Jensen MD, Ryan DH, Apovian CM, Ard JD, Comuzzie AG, Donato KA, et al
. 2013 AHA/ACC/TOS guideline for the management of overweight and obesity in adults: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and the Obesity Society. Circulation 2014;129:S102-38.
Diabetes Prevention Program Research Group, Knowler WC, Fowler SE, Hamman RF, Christophi CA, Hoffman HJ, et al
. 10-year follow-up of diabetes incidence and weight loss in the diabetes prevention program outcomes study. Lancet 2009;374:1677-86.
Martin JM, Beshears J, Milkman KL, Bazerman MH, Sutherland LA. Modeling expert opinions on food healthfulness: A nutrition metric. J Am Diet Assoc 2009;109:1088-91.
Ainsworth BE, Haskell WL, Leon AS, Jacobs DR Jr., Montoye HJ, Sallis JF, et al.
Compendium of physical activities: Classification of energy costs of human physical activities. Med Sci Sports Exerc 1993;25:71-80.
StataCorp. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP; 2013.
Masters K, Ellaway RH, Topps D, Archibald D, Hogue RJ. Mobile technologies in medical education: AMEE guide no 105. Med Teach 2016;38:537-49.
O'Connor S, Andrews T. Smartphones and mobile applications (apps) in clinical nursing education: A student perspective. Nurse Educ Today 2018;69:172-8.
Sayedalamin Z, Alshuaibi A, Almutairi O, Baghaffar M, Jameel T, Baig M, et al.
Utilization of smart phones related medical applications among medical students at king Abdulaziz University, Jeddah: A cross-sectional study. J Infect Public Health 2016;9:691-7.
Raatz SK, Scheett AJ, Johnson LK, Jahns L. Validity of electronic diet recording nutrient estimates compared to dietitian analysis of diet records: Randomized controlled trial. J Med Internet Res 2015;17:e21.
[Table 1], [Table 2], [Table 3], [Table 4]