Improving the efficiency and standards of a National Immunization Program Review: lessons learnt from United Republic of Tanzania

Improving the efficiency and standards of a National Immunization Program Review: lessons learnt from United Republic of Tanzania

Dafrossa Lyimo¹, Christopher Kamugisha², Emmanuel Yohana¹, Messeret Eshetu³, Aaron Wallace⁴, Kirsten Ward⁴, Carsten Mantel⁵, Karen Hennessey^5,&

 

¹Ministry of Health & Social Welfare, Dar Es Salaam, Tanzania, ²World Health Organization, Tanzania Country Office, Dar Es Salaam, Tanzania, ³World Health Organization, Intercountry Support Office for Eastern and Southern Africa, Harare, Zimbabwe, ⁴US Centers for Disease Control & Prevention, Atlanta, GA, USA, ⁵World Health Organization, Headquarters, Geneva, Switzerland

 

 

^&Corresponding author
Karen Hennessey, World Health Organization, Headquarters, Geneva, Switzerland

 

 

 

 

National immunization program (NIP) Reviews are the core of a country’s monitoring and evaluation activities; they guide implementation of immunization services nationwide and help track progress towards goals of the national programme and of the Global Vaccine Action Plan 2016-2020 [1,2].

 

The World Health Organization (WHO) recommends that Member States conduct a comprehensive NIP Review every five years to guide the development of a new comprehensive multiyear plan for immunization (cMYP) [2]. NIP Review teams include global and local experts who collect, observe and synthesize program information across all levels (national to health facility) over a 2-3 week period.

 

The WHO Regions have developed Review guidelines [3,4]; however, reviews have grown increasingly complex, with concerns raised about variable quality across reviews. Additionally, increasing numbers of special-topic NIP assessments have been recommended, including post-vaccine introduction evaluations (PIE), vaccine-preventable disease surveillance reviews, vaccine supply management evaluations, data quality reviews, and further donor-specific evaluations [3-9]. These activities often have substantial overlap with NIP reviews, raising concern about duplication and NIP staff time spent conducting assessments [6].

 

Tanzania’s immunization program is one of the best performing in Africa with >90% of children completing three doses of Diphtheria-Tetanus-Pertussis-Hepatitis B- Haemophilus influenza type b (DTP-HepB-Hib) vaccine during 2010-2014. The country continues to strive to improve the program, especially to reach high coverage in all districts (12% of districts had <80% coverage with three doses of DTP-HepB-Hibvaccine in 2014) and to increase measles 2^nd dose coverage (29% in 2014). A NIP Review was planned for 2015 in preparation for developing the cMYP for 2016-2020.

 

This report outlines lessons learned from applying innovative strategies to the 2015 NIP Review in Tanzania.

 

 

To improve the quality and efficiency of the NIP Review, Tanzania MoHSW and WHO Tanzania implemented innovative Review strategies and assessed their feasibility and acceptability. Five such strategies used were:

 

Extensive integration of assessments: this Review integrated the measles-rubella (MR) PIE, an assessment of the human papillomavirus (HPV) vaccine demonstration project, and assessments of data systems and quality, VPD surveillance and immunization financing.

 

Modified desk review: new timing and tasks of the desk review were tested. It is generally recommended [4] to conduct a desk review several months in advance and submit a report of findings. In this Review, the desk review was started 6 weeks before the field activities by a team of two people (one local staff and one external consultant) and was completed two weeks prior to field activities. Additional tasks were to immediately integrate desk review findings into tool development and Review team training.

 

Selection of topic-leads: this Review nominated topic-leads responsible for facilitating discussion and drafting conclusions and recommendations for their respective topic areas. Topic-leads were identified before field deployment by reviewing the expertise of all external Review members and making the best match of skill sets with topic areas.

 

Interactive review team training: the training included practical sessions on using hand-held tablets and mock interviews to familiarize the teams with the data collection tools and country context. A trainer initially guided each team through the data collection tools and use of the handheld tablets. Afterwards, each individuals within each team conducted mock interviews with one another to familiarize themselves with each question in the data collection tool and with use of the handheld tablets. Feedback was immediately collected from each team to identify any problems encountered with either the tablet or the data collection tool.

 

Streamlined data collection tools and use of hand-held tablets: as part of a regional effort to develop a standardized methodology for conducting Reviews in the African region, A team of experts from global agencies (WHO Headquarters, WHO African Regional Office, Centers for Disease Control and UNICEF) reviewed tools from previous reviews to identify core variables. The core variables were provided to the team Topic Leads conducting the Tanzania Review and these variables were enhanced based on the desk review findings and objectives of the other assessments being integrated. Topic leads reviewed the respective segment of the questionnaires to ensure desk review findings and local context were taken into account. Field teams used a tablet containing the data collection tools programmed with Zegeba software [10].

 

Evaluation of review strategies: the following methods were used to evaluate the review strategies.

 

Post-review survey: to assess the acceptability and feasibility of strategies, all reviewers were asked to complete a standard post-review evaluation form.

 

Post-review group interview: an interactive group session was held at the end of the Review to obtain feedback on process and practical experiences from the Review.

 

Written feedback: the review process was documented through reports written by the desk review consultant and the review-lead.

 

These evaluation findings are presented according to the five NIP Review phases: 1) concept development, 2) preparation, 3) implementation, 4) synthesis of conclusions and recommendations, and 5) translating recommendations to action [2].

 

 

The Tanzania Review was conducted from 8^th to 24^thJuly 2015 and included 22 national and 10 international participants. The post-Review survey was completed by 22(66%) participants. Of these, over half (55%) had previously participated in at least one NIP Review. The post-Review interactive group session was attended by 25 persons, of whom over 90% participated as field team reviewers.

 

Phase 1: concept development

 

The concept note was developed six months prior to the review and was used to secure national participation and help partner agencies nominate external reviewers with the required expertise as topic leads.

 

Phase 2: preparation

 

An international consultant was hired for four weeks to conduct the desk review and assist with tool development and team training. A second international consultant was hired as the overall Review-lead, responsible for Review team training, oversight, debriefing and report writing. Additionally, eight external expert reviewers served as topic-leads on: NIP management and financing, vaccine management, service delivery, monitoring and data quality, surveillance, community demand, measles-rubella and HPV vaccine introduction.

 

The desk review summarized the status of recommendations from six large assessments during 2010-2014 including the 2010 NIP Review, three PIEs, a VPD surveillance review and a vaccine management assessment. Immunization stakeholders were interviewed to help assess status of recommendations from these previous assessments and identify priority areas for the Review.

 

Phase 3: review implementation

 

The NIP Review questionnaires were shortened by at least 30% (based on number of questions) compared to questionnaires used in other recent NIP Reviews. Ten of 30 regions in Tanzania were purposefully selected to be representative of the geographic variations in the country; within each region, one high-performing district and one low-performing district were randomly selected from a high-performing and low-performing sampling frame (performance based on DTP3 vaccination coverage) (Figure 1). Within each district, two health facilities were randomly selected. Ten teams each covered one region and in total visited 10 regional and 20 district health offices and 40 health facilities, observed 33 immunization sessions and 123 children being vaccinated, and interviewed 126 caregivers of these children (three children had two caregivers with them and both caregivers were included in interviews). Each team spent approximately five days within each region visiting their assigned locations for the Review. In addition, two international reviewers conducted the national-level aspect of the Review by having 21 key informant discussions with government officials and immunization partners.

 

Among the surveyed Review participants, 60% found integration of multiple assessments acceptable and 65% found the length of the tools acceptable (Table 1). Participants who did not find integration of multiple assessments acceptable expressed that it may only be feasible with good country capacity and support to do so. Participants who did not find tools acceptable felt that queries were missing in specific topic areas namely surveillance or new vaccine introduction. The tablets used for data collection were well accepted by 100% of the surveyed participants, with approximately half believing the tablets helped increase accuracy and decrease workload. Participants reported that tablet-based data were available in a timelier manner for analysis compared to paper-based tools.

 

Phases 4 & 5: synthesis of conclusions and recommendations and translation into action

 

Field teams reconvened and presented findings from the region they visited, while topic-leads consolidated the respective topic-area findings from all teams into a topic-area report and presentation. Immediately afterwards, topic-leads led breakout groups to discuss and draft cross-teamconclusions and recommendations. Back in plenary, topic-specific conclusions and recommendations were presented to ensure agreement. Linking findings from the national and field teams was limited due to intense focus on the field team findings. Of the surveyed reviewers, 86% reported having sufficient time to synthesize findings. The Review concluded with a presentation to MOHSW and key NIP partners; Review findings and recommendations that were endorsed by MOHSW and NIP partners were used to develop the 2016-2020 cMYP.

 

 

Tanzania was able to successfully implement a high quality integrated NIP Review. The findings and recommendations of the Review were summarized and translated into action plans during a subsequent workshop to develop the 2016-2020 cMYP. Although a number of these innovations may have been utilized on an ad-hoc basis in prior reviews, this Review is among the first to use a systematic approach to implementing multiple innovations together as a single package with the goal of streamlining the Review while also ensuring this Review provides sufficient information to answer questions that would normally be answered through conducting several separate reviews. Additionally, the Tanzania Review provided an opportunity to document lessons learned that were later incorporated into new integrated NIP review guidelines which are designed to ensure countries need only conduct a single activity to answer the many questions which originally had to be answered through the conduct of several separate activities. These lessons are summarized in Table 2.

 

The approach to conducting an integrated review as a means to efficiently address several different objectives simultaneously (i.e. introduction of a new vaccine, evaluation of vaccine supply chain management, general program evaluation etc) has gained traction in recent years, with examples of countries where the Review leads choose a more integrated approach, including in Ghana and Rwanda. In these latter two reviews, the general aim was to incorporate a traditional NIP review with a post-vaccine introduction evaluation (PIE) rather than conducting two separate activities. In contrast, for Tanzania, our regional team of immunization experts aimed to design this integrated review to address the needs of a traditional NIP review, PIE, financial records review, supply chain management review, and surveillance systems review. Additionally, the integrated review leads shared this aim through the review concept note with all immunization program stakeholders to ensure that their needs would be met by the single review and that they would not mandate an additional separate review. Additionally, we incorporated a framework of efficiency measures, including using topic leads to consolidate the field review findings by topic, and using electronic data collection to accelerate data management and analysis. Lastly, we documented the feedback of all review participants in regards to both the integrated design of the review and the success of the efficiency measures as a means to feed into a draft set of best practices for conducting integrated NIP reviews. This best practices document will be disseminated soon on the WHO website for all countries to incorporate into their future NIP reviews.

 

Here we document lessons for future NIP Reviews which may also be relevant for other nationwide program evaluations. First, adetailed NIP Review concept note developed six months prior to the field review helped secure both national and international participation. Second, timely engagement of an experienced desk review consultant helped tailor the Review to meet country needs. Third, the majority of surveyed participants found that integrating multiple assessments was feasible and NIP management found it highly desirable for efficiency reasons. Fourth, reviewers recommended tablet use for future reviews with a major benefit being that quantitative data were available in time to support conclusions and recommendations. However, a staff member with specialized technical knowledge was required to support tablet use. Lastly, identifying experienced topic-leads allowed for highly technical and focused discussions on various topics to take place concurrently during the field teams’ debriefing sessions. In future Reviews, linking of national and field findings may be improved by engaging topic-leads earlier in the process and having explicit guidance for them to synthesize the two levels of findings.

 

Limitations and challenges exist in the innovations used in the Tanzania Review. Although we used electronic data collection to great success per the feedback provided by participants, the ability of sharing data rapidly with all participants requires cellular or wireless network availability which could be limited in a number of rural settings globally. Additionally, a specialized skillset is required to manage and troubleshoot computing devices; this skillset may not be available in all reviews, which could hamper use of electronic data collection. Another challenge is ensuring that topic leads with sufficient applicable experience are recruited to a review; generally a key aspect of these reviews is inclusion of an external reviewer who is likely to provide a less-biased perspective on the performance of the program, but sufficiently experienced external participants may not always be available for all reviews.

 

 

Many reviews have been conducted in the WHO African Region and it is common for reviewers to report that topics are not fully assessed even when the long version of the tools are used. This is the nature of a comprehensive evaluation covering a broad range of technical areas. The finding that the majority of participants found integration of multiple assessments and streamlined data collection tools acceptable was considered a favorable outcome, especially given that data collection queries were reduced by at least 30%. This approach was made successful by having a thorough desk review and training that provided the rationale for eliminating or expanding certain topics.

 

 

The authors declare no competing interests.

 

 

All authors contributed to developing the concept for the manuscript. DL, CK, AW and KY wrote the first draft; all authors revised successive versions. All authors have read and agreed to the final version of the manuscript.

 

 

We are grateful for the extensive contributions of the following persons to the success of the Tanzania comprehensive NIP review and for their respective work described in this article: Dr. RufaroChatora (World Health Organization Representative, Tanzania); Samantha Dolan and Andrea Carcelen (US Centers for Disease Control and Prevention); David Oh and Jan Grevendonk (World Health Organization, Geneva) and Jethro Chakauya (World Health Organization, Intercountry Support office for Eastern and Southern Africa).

 

 

Table 1: results from a survey completed by 22 (66%) reviewers participating in a comprehensive national immunization program review, United Republic of Tanzania, July 2015

Table 2: summary of approaches and lessons for improving the quality and efficiency of the national immunization program (NIP) review and lessons learnt, United Republic of Tanzania, July 2015

Figure 1: the 10 regions and 20 districts selected for field assessment in the 2015 National Immunization Program (NIP) Review in Tanzania

 

 

1. WHO. Immunization, Vaccines and Biologicals. WHO-UNICEF guidelines for developing a comprehensive multi-year plan (cMYP). Geneva, 2013. accessed 17 December 2015


2. WHO. Immunization, Vaccines and Biologicals. Global Vaccine Action Plan 2011-2020. Geneva, 2012. Accessed 17 December 2015


3. WHO. Pan American Health Organization. Methodology for the international evaluation of the expanded program on immunization. Washington DC, 2013. Accessed 17 December 2015


4. World Health Organization. African Regional Office. African Region EPI Review Guidelines. Brazzaville, 2012. Accessed 17 December 2015


5. WHO. Immunization, Vaccines and Biologicals. New Vaccine Post Introduction Evaluation (PIE) Tool. Geneva, 2010. Accessed 17 December 2015


6. WHO. UNICEF. Global Immunization Meeting Report. Geneva, 2015. Accessed 17 December 2015


7. GAVI. Joint appraisals. Geneva, 2015. Accessed 17 December 2015


8. USAID. Government of Malawi. In-depth National Surveillance Review Report May, 2012. Accessed 17 December 2015 Google Scholar


9. WHO. Immunization, Vaccines and Biologicals. Effective Vaccine Management (EVM) Initiative. Geneva, 2015. Accessed 17 December 2015


10. Zegeba. Data Capture Software. Gaseid, Norway. Accessed 17 December 2015