|ORIGINAL RESEARCH PAPER
|Year : 2011 | Volume
| Issue : 1 | Page : 540
Mixed Methods Evaluation of an International Internet-based Continuing Medical Education Course for Pediatric HIV Providers in Pune, India
R Ramanathan1, R Aldis2, S Gupta3, M Desai4, RC Bollinger5, VA Reed6
1 Department of Surgery, Virginia Commonwealth University Health System, Richmond, VA, USA
2 Department of Psychiatry; Cambridge Health Alliance, Harvard Medical School, Boston, MA, USA
3 Department of Surgery, UCSF East Bay Program, CA, USA
4 London School of Hygiene and Tropical Medicine, Camden, London, United Kingdom
5 Global Health Education, Baltimore, Maryland, USA
6 Dartmouth Medical School, Hanover, NH, USA
|Date of Submission||28-Aug-2010|
|Date of Acceptance||24-Mar-2011|
|Date of Web Publication||29-Apr-2011|
Department of Surgery, Virginia Commonwealth University Health System, Richmond VA
Source of Support: None, Conflict of Interest: None
Context: Studies of HIV care in Pune, a high-HIV-prevalence city in India, have shown that a significant proportion of
practitioners were not adhering to national guidelines due to inadequate awareness and understanding.
Objectives: This study examined the effectiveness of a pilot Internet-based continuing medical education course in increasing knowledge of pediatric HIV diagnosis and treatment among providers in Pune. The study also explored perceived factors limiting the effectiveness of the pilot course.
Methods: The mixed methods evaluation design consisted of quantitative pre- and post-course knowledge assessments, and qualitative focus groups and in-depth interviews conducted on site with healthcare providers with experience treating HIV to explore the barriers to optimal course utilization, the applicability of the course content, and the systemic barriers to the implementation of physician knowledge.
Findings: There were significant increases (p<0.05) in mean knowledge scores on the global knowledge assessment and for two of five individual course modules. Perceived barriers to optimal course utilization were identified as being related to Internet access. The course content was reported to be generally useful, although certain guidelines and information were described as not congruent with local resource availability. Participants reported that the major barriers to implementing their knowledge were stigma that prevented patients from seeking care and financial resource limitations affecting physician practice.
Discussion and Conclusions: This course resulted in a modest increase in pediatric HIV knowledge among Pune healthcare providers. Identification of perceived factors limiting the effectiveness of the course provides guidance for improving future Internet-based courses.
Keywords: Continuing medical education, distance education, HIV, Internet
|How to cite this article:|
Ramanathan R, Aldis R, Gupta S, Desai M, Bollinger R C, Reed V A. Mixed Methods Evaluation of an International Internet-based Continuing Medical Education Course for Pediatric HIV Providers in Pune, India. Educ Health 2011;24:540
|How to cite this URL:|
Ramanathan R, Aldis R, Gupta S, Desai M, Bollinger R C, Reed V A. Mixed Methods Evaluation of an International Internet-based Continuing Medical Education Course for Pediatric HIV Providers in Pune, India. Educ Health [serial online] 2011 [cited 2022 Oct 4];24:540. Available from: https://educationforhealth.net//text.asp?2011/24/1/540/101452
In 2008, India had an estimated 2.4 million persons living with HIV, trailing only South Africa and Nigeria in prevalence1. Between 2002 and 2006, the proportion of all HIV infected individuals in India who were children increased from 3.0 to 3.8%2. This translated into 70,000 children infected with HIV in 2006, with an estimated 21,000 additional cases being added annually3. In response, the Indian government’s National AIDS Control Organization (NACO), which initially focused on education and prevention among high-risk populations, began a program that delivered free HIV care and antiretroviral therapy (ART) to nearly 96,000 patients at 127 HIV centers located in the six highest HIV-prevalence states2,4. Focusing on efforts to decrease the number of children with HIV, including reducing the transmission of HIV from mother to child, provides a major opportunity to change the trajectory of HIV infection5.
Pune, located along a major trucking route in the western state of Maharashtra, is considered a high-HIV-prevalence city6,7 in a NACO-recognized high-HIV-prevalence state8. Studies of HIV care in Pune have shown that a significant proportion of private practitioners, including those providing pediatric HIV care, were not adhering to national guidelines by not obtaining proper informed consent, providing pre- and post-test counseling or performing confirmatory testing9,10. This lack of adherence was due in part to inadequate provider awareness and understanding9,10 and, combined with ineffective doctor-patient communication, has contributed to fear and stigma surrounding HIV/AIDS9-13. Such gaps in education and awareness have prompted calls for increased continuing medical education (CME) for general and HIV providers in India, echoing the appeals by global agencies for practical HIV management guidance in limited resource settings14-16.
India has 58 physicians per 100,000 population17. This is significantly below what the World Health Organization (WHO) has defined as a critical threshold of about 228 physicians per 100,000 population18. The majority of physicians in India are concentrated in cities19, and few have specialist training in HIV9. While India lacks national mandatory CME requirements15, programs aimed at improving HIV care have been delivered in multiple ways, ranging from traditional one- and two-day face-to-face didactic workshops on antiretroviral therapy20 to Internet-based modules on chlamydia screening education21. Internet-based CME has been shown to be an effective medium for increasing knowledge and positively influencing physician practice22-25. Studies have indicated that in the short-term, Internet-based CME is comparable to live interactive CME workshops, and have suggested that long-term, it may produce more behavioral changes than conventional live CME26. Internet-based international distance education can be a cost-effective way to increase global access to medical knowledge27-29.
Understanding the perceived personal, interpersonal and systemic barriers to Internet-based education has been
advocated as a necessary step to improve the effectiveness of
Internet-based education programs,30-32. The use of mixed methods – the collection and use of both qualitative and quantitative data in the same study33 – is specifically aimed at better understanding both program impact and these perceived barriers. Thus, the goal of this mixed methods study was to examine the effectiveness of a pilot Internet-based CME course in increasing knowledge of pediatric HIV diagnosis and treatment among physicians in Pune, India. Given the pilot nature of the CME course, a secondary goal was to explore factors limiting the effectiveness of the course through investigating perceived barriers to optimal utilization of the course resources, applicability of the course content and systemic barriers to the implementation of the knowledge learned34,35.
The course, designed by the Johns Hopkins University Center for Clinical Global Health Education (JHU-CCGHE), was introduced in 2007. In a needs assessment conducted prior to the course development, six physicians recruited by JHU-CCGHE’s partners in Pune identified knowledge gaps in HIV epidemiology, tuberculosis (TB) co-infection and antiretroviral therapy (ART). Based on those results and the 2006 WHO publication on ART for children in resource-limited settings36, seven course objectives were developed and organized in five course modules. Information covered included: HIV epidemiology and prevention of maternal-child transmission; management and monitoring of HIV-infected infants and children; HIV-infected children; families and communities; antiretrovirals; and organ system illness and assessment, as shown in Figure 1. WHO guidelines37,38 formed the framework for module design. Each module consisted of three to four units that included online lectures, transcripts, references and resources. The six-week course contained 29 online streaming video lectures given by experts in pediatric HIV at the Johns Hopkins University School of Medicine. These lectures made use of graphics, case examples and outlines of WHO guidance. The course also utilized weekly case discussion sessions between the course participants and the Johns Hopkins course faculty via videoconferencing at a central facility in Pune. A few modules made use of questions posed and answers given.
Figure 1: Learning objectives for a 6-week Internet-based continuing medical education (CME) course on pediatric HIV diagnosis and treatment in Pune, India
Physicians and other healthcare providers from both the public and private sector were recruited for the pilot course through purposive sampling by our local partners at the B.J. Medical College in Pune, the National AIDS Research Institute (NARI) in Pune and the Indian Institute of Technology (IIT) in Kharagpur in late 2006. Participation in the course was free and voluntary. Forty-two participants with experience treating pediatric HIV/AIDS completed the online course and attended the weekly videoconferencing sessions. Thirty-five of the 42 participants recruited (83%) were physicians from the public and private sectors. The other participants were nurses and other health professionals. Nearly two-thirds of participants were male and 90% specialized in pediatrics (64.3%) or medicine (26.2%).
Mixed Methods Program Evaluation
We designed a mixed methods study to assess the effectiveness of the course and explore factors perceived to limit its effectiveness. The Johns Hopkins School of Medicine Institutional Review Boards (IRB) approved this study and local permission was obtained from the hospital management in Pune.
We selected an expansion framework with its goal of 'increasing the scope of inquiry by selecting the methods most appropriate for multiple inquiry components'39. This approach utilized quantitative and qualitative information to assess program outcomes and processes, respectively39.
The effectiveness of the course in increasing knowledge of pediatric HIV diagnosis and treatment was measured through online assessments administered before and after the CME course. A database of 80 multiple-choice questions (MCQ), categorized by learning objective, was prepared by the course faculty and used as the source of knowledge questions. Ten questions covering all seven learning objectives were randomly selected by computer for the global pre-and post-course knowledge assessments. In addition to global tests of knowledge, each of the five course modules was tested with pre- and post-module knowledge tests. There was no defined pass mark; since the MCQs tested core knowledge in each module, it was hoped that participants could achieve as close to 100% as possible. The instrument did not draw on any specific validated tool, but was based on commonly used MCQ CME format. Results of the knowledge assessments were analyzed using paired student t-tests and 95% confidence intervals.
The global post-course assessment also included questions about the usefulness of the content and educational approaches, knowledge acquired, and accessibility of the course and educational materials. Lastly, participants were asked to provide recommendations to improve the course content and structure.
Factors limiting the effectiveness of the course were explored through focus group discussions and in-depth interviews (described in Figure 2). To explore perceived barriers to optimal utilization of the course, participants were asked about any technical issues encountered, difficulties with Internet access and the utility of videoconferencing. The applicability of the course was assessed by asking participants to identify useful aspects of the course, suggest improvements and comment on what would constitute a successful course. Implementation barriers were explored through questions about helpful and hindering aspects of the HIV care system in Pune, participants’ descriptions of the HIV care process for patients and our observations of participants’ practices. Observation of the practices consisted of one day at each of the practices shadowing the participants as they interacted with their patients, other medical staff and their respective healthcare systems.
Figure 2: Focus group discussion and in-depth interview questions used to explore the factors perceived to limit Internet-based continuing medical education (CME) course effectiveness
For both focus groups and individual interviews, participants were informed of the fact that focus group leaders and interviewers were not involved with the creation of the course. They were also informed of confidentiality procedures, and verbal consent was obtained prior to the start of each session. Each focus group consisted of three to seven participants, at least one scribe, one facilitator and one observer. Focus group participants were grouped by their clinical practice location; individual interviews took place at the offices of the participants. Sessions were tape-recorded and the investigators independently took notes and reviewed them at the end of each session. Three focus group sessions with 14 participants and 9 in-depth interviews were conducted. Participants were also offered the opportunity to complete an anonymous survey on the applicability of the content and usability of the web interface.
Three separate investigators independently identified and clustered individual statements from the focus group discussions and interviews according to codes inferred from the data. A second round of sorting identified some codes as subsets of others and resulted in consolidation of the initial codes into themes pertaining to the three factors identified as limiting the course effectiveness.
As previously described, we used an expansion model of mixed methods39. The quantitative and qualitative data were collected simultaneously and integrated in the analysis.
Demographic characteristics of the course participants are described in Table 1. All 42 participants were offered the opportunity to participate in focus groups and in-depth interviews. Twenty-four participants took part in three focus groups. Of these, nine participated in in-depth interviews. Focus group and interview participants did not differ from other course participants in terms of gender and specialty representation, however there were fewer public sector providers participating in the focus groups – 33.3% as compared to 57.1% in the course.
Table I: Characteristics of Study Participants
Effect on physician knowledge
As shown in Table 2, mean post-test scores for the global knowledge assessment and two of the five course modules increased significantly from pre-test scores40. Modules for which scores improved addressed the social impact of HIV and opportunistic infection diagnosis and treatment.
Table II: Results of multiple-choice knowledge assessments administered to participants before and after the course and each module
Barriers to optimal utilization of course resources
Participants reported that issues related to Internet access limited them from taking full advantage of the audio and video components of the course. Nearly 60% of participants reported that their primary source of Internet access was at home and/or at work, while 36% of participants relied on public Internet cafés for Internet access. Nearly half of the participants reported daily access to the Internet while one-fifth had access only a few times a week.
The weekly videoconference sessions with the course faculty were well received: attendance exceeded 90%, with 81% of participants reporting that the sessions were helpful to their learning. However, nearly 40% of participants reported they had trouble traveling to the sessions at the scheduled time and location. Although all participants were proficient in English, they stated that noisy audio connections, differences in accents and the relatively fast rate of speech of the Johns Hopkins faculty limited their comprehension at times.
Applicability of course content
Overall, 77% of the participants reported the lectures to be of high quality and 88% felt that the material improved their ability to treat HIV. Several participants reported having received minimal education on HIV diagnosis and treatment during their medical training. One young pediatrician commented:
'The doctors who are one generation ahead hardly have any knowledge of HIV. They have not updated their knowledge. We don’t have any systematic updating of knowledge in India at the present. […] In college we had some information in our textbooks, but nothing on treatment.'
The content areas that were described as most informative were those that address perinatal diagnosis, diet and infant feeding, and vaccinations in HIV+ individuals. However, many participants commented that the guidelines on opportunistic illnesses and HIV diagnostic testing frequently specified tests that were unavailable or unaffordable for patients. One internist at the NACO center noted:
'Guidelines on using second-line ART was good for personal knowledge, but was not useful since no one can get them, not even at NARI (National Aids Research Institute).'
Several participants requested alternate best-practice guidelines for situations when the prescribed diagnostic test was not available. A number of participants also requested example-based instruction on counseling since most physicians performed counseling without any formal training. As noted by a private practitioner and one of the participants from the municipal hospital:
'We ourselves do the counseling also. In big hospitals like Sassoon [central government hospital in Pune] they have counselors, but here we do it only.'
'There is no teaching for counseling, I just do [it] by telling them the important details. How it is done other places I don’t know. With the video, if they show us how they do in [the] United States, we can see.'
Perceived barriers to implementation of knowledge
Access to resources
Participants described access to resources as one of the primary barriers limiting their implementation of medical knowledge. Even though most diagnostics and treatments are available in Pune’s fee-for-service system, patients often cannot afford them, thereby forcing private practitioners to make clinical decisions based on limited information. One private practice dermatologist commented:
'In private [practice], we only do [laboratory tests] if necessary, often after they have failed empiric therapy. This is to minimize costs for patients.'
Participants noted that restrictive costs have been mitigated somewhat by local non-governmental organizations (NGOs) and international organization subsidies.
Another major implementation barrier identified was stigma against HIV+ individuals. One participant noted that government-sponsored media campaigns have worked to decrease such stigma:
'The stigma is decreasing - there is now many signs in buses about HIV, on TV people talk about HIV. It is becoming more accepted.'
However many patients continued to hide their HIV status due to fears of being ostracized, losing their job or being unable to find a partner.
Prior to taking the course, many care providers had taken steps to reduce the magnitude of stigma their patients faced. A number of participants in our study had founded various programs to decrease social stigma, ranging from informational telephone hotlines about HIV/AIDS to orphanages for HIV+ orphans and NGOs organizing vocational programs for HIV+ individuals41. Additionally, participants working at the local municipal hospital had devised innovative ways of maintaining patient privacy, such as using euphemisms for HIV on the crowded ward floors and providing lactation suppressants for mothers in restrictive households who would otherwise face opposition to formula-feeding their infants:
'[In the government pediatrics ward] we call it ICS – immunocompromised state – so that the other people don’t know they are HIV+…'
'When a new HIV+ mother gives birth, we tell her she should not breastfeed if she cannot take the medicines at home. So we give a different medicine to stop her milk if she wants.'
Discussion and Conclusions
Ultimately, the goals of CME are to increase knowledge, positively impact healthcare provider behavior and improve patient outcomes42,43. Although the course was successful in increasing knowledge in certain areas of pediatric HIV, there is room for improvement. While knowledge alone is not sufficient to change provider behavior44,45 , addressing knowledge gaps, particularly those identified locally46,47, is an important step.
While studies have indicated that Internet-based CME may produce more behavioral changes than conventional live CME26, it does have some shortcomings. The pilot course was designed to address some common limitations of Internet-based education48-50. Social isolation was addressed through video lectures and group videoconferencing sessions with the faculty. Inflexibility was addressed by allowing participants to complete the course at their desired pace at any Internet-capable location, and the course was free to encourage and maximize participation. In our evaluation, we identified additional opportunities for improving future courses through a qualitative exploration of the perceived barriers to optimal utilization of course resources, the applicability of the course content and the systemic barriers to the implementation of the knowledge learned.
The main perceived barriers to optimal utilization of the course resources were Internet access and quality. While Internet-based education offers exciting possibilities to reach urban and rural providers in a cost effective way, the technology utilized must be appropriate. In future courses, providing participants with the option of a low-bandwidth version of a course without streaming audio or video would help ensure that participants on slower Internet connections could also access the course. Additionally, increased text resources that participants could print may decrease reliance on Internet connections. Such trade-offs must be weighed against the decreased interaction that would result45. Videoconferencing, while technologically intensive, was successful in this course because of the availability of a reliable high-speed Internet connection and necessary technology at the central videoconferencing site in Pune.
We found that the perceived local systemic constraints most affecting provider practice and the applicability of the course were social stigma faced by patients and resource limitations faced by healthcare providers because patients could not afford certain diagnostics and therapeutics. Although resource limitations and access cannot be modified by education alone, their impact can be lessened through educational interventions such as providing alternate best-practice guidelines and recommendations that utilize more affordable tests. Providing guidelines in future courses that rely less on expensive tests like polymerase chain reaction (PCR), and instead rely more on patterns in clinical examination and basic laboratory tests will allow healthcare professionals in low-resource settings to provide evidence-based care in a resource-sensitive manner. Similarly, promoting networking between local physicians through online forums or live group meetings can modulate the effect of social stigma faced by patients. Increased networking can be powerful given that many providers had devised innovative ways to combat or minimize stigma through low-cost strategies including euphemisms and lactation-suppressants, and many providers were involved with NGOs providing HIV services. Empowering them with a platform through which to collaborate would augment their efforts at minimizing the stigma patients endure and hopefully encourage additional initiatives.
In terms of the content in the course, 88% of participants felt that their ability to treat HIV was improved as a result of the course. They described the most useful topics as perinatal diagnosis, diet and infant feeding and vaccinations in HIV+ individuals. This is consistent with other feedback suggesting that knowledge and guidelines requiring minimal additional resources to implement are the most applicable. As suggested by a number of participants, guidelines that rely less on expensive tests and more on clinical examination and basic laboratory test patterns would be more applicable in low-resource settings. Many of the participants worked in small practices where they counseled patients themselves without any formal training. Utilizing streaming video in future courses to provide example-based instruction on counseling would provide training without requiring additional resources from the practitioner. In addition to addressing specific knowledge gaps, comments by some younger providers about the paucity of CME and of the overall inadequacy of HIV/AIDS knowledge among older providers reiterates the need for such targeted clinically relevant courses.
The most prominent criticism and challenge of the course noted by participants was in tailoring recommendations and guidelines to the local resource availabilities of the providers. While knowledge assessments inform us of changes in the fund of knowledge of the participants, they do not shed light on the social factors that will ultimately influence practice and patient care. Thus, an emphasis on mixed methods evaluation that combines quantitative reports of change in pre/post-knowledge with qualitative information - focus groups, interviews or other comparable methods - to elicit direct feedback from participants should be an important component of assessment of Internet-based education. Based on course feedback, certain technical issues, such as the speed of speech, can be readily addressed. The challenge for Western developers of CME courses is in better understanding the local cultural and resource constraints of the target audience.
Given that this was an evaluation of a pilot course, this study is subject to a number of limitations. One is the relatively small sample size available. A related limitation is the potential of selection bias stemming from the fact that participants were a purposive sample recruited by our local partners, and they were urban physicians who were required to have Internet access and likely had more educational resources than the average physician providing pediatric HIV care in Pune. Despite these limitations, there were significant increases in knowledge in two modules and the global knowledge assessment. While generalizability is limited, it is likely that a similar course with a larger and more representative physician population would show greater knowledge gains. The relative socio-economic advantages of the participants, as noted above, mean that they likely would have benefited from greater access to current knowledge and guidelines than many of their rural counterparts51. This study was performed soon after the conclusion of the course and thus evaluated short-term changes in knowledge. Future studies should focus on longer-term follow up, as well as on assessment of changes in provider behavior, attitudes, and patient outcomes as additional endpoints.
An additional limitation of this study is that the focus groups and interviews were led by investigators who, while not involved directly with course development and participant evaluation, were nonetheless affiliated with the course. As a result, some participants may have been reluctant to provide criticism. We attempted to minimize this by providing participants with both a focus group setting and a more personal interview setting in which to provide feedback, criticism and suggestions.
Since the completion of this pilot study, additional courses incorporating its findings and recommendations have been developed and launched by the Johns Hopkins University CCGHE in India, Ethiopia and Zambia. By investigating the common factors that limit the effectiveness of such courses and developing solutions to minimize these limitations, Internet-based CME programs have the potential to improve the quality of HIV care across the globe.
We would like to thank Dr. Stephen Plume and Dr. Gautham Suresh for their assistance in planning this project. Additionally, Ms. Lisa Wolf, Dr. Bharath Purandare and Mr. Nikhil Gupte were integral to the coordination and data collection. This work was supported financially by the Dartmouth International Health Group.
1. UNAIDS. Report on the global HIV/AIDS epidemic 2008. Joint United Nations Programme on HIV/AIDS. Geneva; 2008.
2. NACO. Annual Sentinel Surveillance Country Report. New Delhi: National Institute of Health and Family Welfare; 2006.
3. NACO. Policy Framework for Children and AIDS, India. New Delhi; 2007. Available from: http://www.nacoonline.org/upload/Publication/Treatment%20Care%20and%20support/PolicyFramework%20Aug_31.pdf
4. Solomon S, Chakraborty A, Yepthomi RD. A review of the HIV epidemic in India. AIDS Education and Prevention. 2004; 16(3 Suppl A):155-169.
5. World Health Organization. Preventing mother-to-child transmission Strategic Vision 2010-2015: Preventing mother-to-child transmission of HIV to reach the UNGASS and Millennium Development Goals. Geneva, Switzerland: World Health Organization; 2010.
6. Paranjape RS, Tripathy SP, Menon PA, Mehendale SM, Khatavkar P, Joshi DR, Patil U, Gadkari DA, Rodrigues JJ. Increasing trend of HIV seroprevalence among pulmonary tuberculosis patients in Pune, India. Indian Journal of Medical Research. 1997; 106:207-211.
7. Mehendale SM, Shepherd ME, Divekar AD, Gangakhedkar RR, Kamble SS, Menon PA, et al. Evidence for high prevalence & rapid transmission of HIV among individuals attending STD clinics in Pune, India. Indian Journal of Medical Research. 1996\; 104:327-335.
8. NACO. Annual Report 2009-2010. New Delhi: Department of AIDS Control, Ministry of Health & Family Welfare; 2010.
9. Kielmann K, Deshmukh D, Deshpande S, Datye V, Porter J, Rangan S. Managing uncertainty around HIV/AIDS in an urban setting: private medical providers and their patients in Pune, India. Social Science and Medicine. 2005; 61(7):1540-1550.
10. Sheikh K, Rangan S, Kielmann K, Deshpande S, Datye V, Porter J. Private providers and HIV testing in Pune, India: challenges and opportunities. AIDS Care. 2005; 17(6):757-766.
11. Datye V, Kielmann K, Sheikh K, Deshmukh D, Deshpande S, Porter J, Rangan S. Private practitioners' communications with patients around HIV testing in Pune, India. Health Policy and Planning. 2006; 21(5):343-352.
12. Pisal H, Sutar S, Sastry J, Kapadia-Kundu N, Joshi A, Joshi M, Leslie J, Scotti L, Bharucha K, Suryavanshi N, Phadke M, Bollinger R, Shankar AV. Nurses' health education program in India increases HIV knowledge and reduces fear. Journal of the Association of Nurses in AIDS Care. 2007; 18(6):32-43.
13. Suryavanshi N, Jonnalagadda S, Erande AS, Sastry J, Pisal H, Bharucha KE, Shrotri A, Bulakh PM, Phadke MA, Bollinger RC, Shankar AV. Infant feeding practices of HIV-positive mothers in India. Journal of Nutrition. 2003; 133(5):1326-1331.
14. Bhat R. Characteristics of private medical practice in India: a provider perspective. Health Policy and Planning. 1999; 14(1):26-37.
15. de Sarkar B, Kumar S. Delays in legislation slow the progress of CME in India. Bulletin of the World Health Organization. 2004; 82(2):154-155.
16. Bennett DE, Bertagnolio S, Sutherland D, Gilks CF. The World Health Organization's global strategy for prevention and assessment of HIV drug resistance. Antiviral Therapy. 2008; 13 Suppl 2:1-13.
17. World Health Organization. India: health profile. Geneva: World Health Organization; 2010. Available from: http://www.who.int/gho/countries/ind.pdf
18. World Health Organization. The World Health Report 2006: Working together for health. Geneva: World Health Organization; 2006.
19. World Health Organization, Country Office for India. Not enough here...Too many there. Health Workforce in India. New Delhi: World Health Organization; 2007.
20. Kabra SK, Mukherjee A, Vani SA, Sinha S, Sharma SK, Mitsuyasu R, Fahey JL. Continuing medical education on antiretroviral therapy in HIV/AIDS in India: needs assessment and impact on clinicians and allied health personnel. National Medical Journal of India. 2009; 22(5):257-260.
21. Casebeer L, Allison J, Spettell CM. Designing tailored Web-based instruction to improve practicing physicians' chlamydial screening rates. Academic Medicine. 2002; 77(9):929.
22. Casebeer L, Kristofco RE, Strasser S, Reilly M, Krishnamoorthy P, Rabin A, Zheng S, Karp S, Myers L. Standardizing evaluation of on-line continuing medical education: physician knowledge, attitudes, and reflection on practice. Journal of Continuing Education in the Health Professions. 2004; 24(2):68-75.
23. Curran V, Lockyer J, Sargeant J, Fleet L. Evaluation of learning outcomes in Web-based continuing medical education. Academic Medicine. 2006; 81(10 Suppl):S30-34.
24. Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Medical Education. 2005; 39(6):561-567.
25. Umble KE, Cervero RM, Yang B, Atkinson WL. Effects of traditional classroom and distance continuing education: a theory-driven evaluation of a vaccine-preventable diseases course. American Journal of Public Health. 2000; 90(8):1218-1224.
26. Fordis M, King JE, Ballantyne CM, Jones PH, Schneider KH, Spann SJ, Greenberg SB, Greisinger AJ. Comparison of the instructional efficacy of Internet-based CME with live interactive CME workshops: a randomized controlled trial. Journal of the American Medical Association. 2005; 294(9):1043-1051.
27. Karimova S, Laporte R, Shubnikov E, Linkov F. Maternal and child health supercourse for the former Soviet Union countries. Maternal and Child Health Journal. 2007; 11(6):628-633.
28. Flores R, Robles J, Burkhalter BR. Distance education with tutoring improves diarrhea case management in Guatemala. International Journal for Quality in Health Care. 2002; 14 Suppl 1:47-56.
29. Kiviat AD, Geary MC, Sunpath H, Moosa MY, Wilson D, Narasimhan P, Ghandi RT. HIV Online Provider Education (HOPE): the internet as a tool for training in HIV medicine. Journal of Infectious Diseases. 2007; 196 Suppl 3:S512-515.
30. Goodyear-Smith F, Janes R. New Zealand rural primary health care workforce in 2005: more than just a doctor shortage. Australian Journal of Rural Health. 2008; 16(1):40-46.
31. Grant J. Learning needs assessment: assessing the need. BMJ. 2002; 324(7330):156-159.
32. Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Medical Journal of Australia. 2004; 180(6 Suppl):S57-60.
33. Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications; 2007.
34. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003; 362(9391):1225-1230.
35. Haines A, Kuruvilla S, Borchert M. Bridging the implementation gap between knowledge and action for health. Bulletin of the World Health Organization. 2004; 82(10):724-731; discussion 732.
36. Jordan MR, Bennett DE, Bertagnolio S, Gilks CF, Sutherland D. World Health Organization surveys to monitor HIV drug resistance prevention and associated factors in sentinel antiretroviral treatment sites. Antiviral Therapy. 2008; 13 Suppl 2:15-23.
37. Gilks CF, Crowley S, Ekpini R, Gove S, Perriens J, Souteyrand Y, Sutherland D, Vitoria M, Guerma T, De Cock K. The WHO public-health approach to antiretroviral treatment against HIV in resource-limited settings. Lancet. 2006; 368(9534):505-510.
38. World Health Organization. Antiretroviral Drugs for Treating Pregnant Women and Preventing HIV Infection in Infants: Guidelines on Care, Treatment and Support for Women Living with HIV/AIDS and their Children in Resource-constrained Settings. Department of HIV/AIDS DoRHaR. Geneva: World Health Organization; 2004.
39. Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis. 1989;11(3):255-274.
40. Sheiner LB, Beal SL. Some suggestions for measuring predictive performance. Journal of Pharmacokinetics and Biopharmaceutics. 1981;9(4):501-512.
41. Ramanathan R. Filling the gaps: Saluting those making life with HIV in India an accessible reality. Global Pulse 2009 5(1) Retrieved January 12, 2011 from: http://www.globalpulsejournal.com/2009_ramanathan_rajesh_filling_the_gaps.html.
42. Lockyer J, Ward R, Toews J. Twelve tips for effective short course design. Medical Teacher. 2005; 27(5):392-395.
43. Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. International Journal of Technology Assessment in Health Care. 2005; 21(3):380-385.
44. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993; 342(8883):1317-1322.
45. Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. Canadian Medical Association Journal. 1997;157(4):408-416.
46. Siddiqi K, Robinson M. Getting evidence into practice in developing countries. Evidence- Based Cardiovascular Medicine. 2006; 10(1):5-7.
47. Siddiqi K, Newell JN. Putting evidence into practice in low-resource settings. Bulletin of the World Health Organization. 2005; 83(12):882.
48. Cook DA. Web-based learning: pros, cons and controversies. Clinical Medicine. 2007; 7(1):37-42.
49. Flynn D, Gregory P, Makki H, Gabbay M. Expectations and experiences of eHealth in primary care: a qualitative practice-based investigation. International Journal of Medical Informatics. 2009; 78(9):588-604.
50. Ratka JR. Journey to Oz: the yellow brick road to a blended learning environment. Critical Care Nursing Quarterly. 2010; 33:35-43.
51. Maniar JK. Health care systems in transition III. India, Part II. The current status of HIV-AIDS in India. Journal of Public Health Medicine. 2000; 22(1):33-37.