A pilot study of a novel, incentivised mHealth technology to monitor the vaccine supply chain in rural Zambia

A pilot study of a novel, incentivised mHealth technology to monitor the vaccine supply chain in rural Zambia

Camillo Lamanna^1,&, Lauren Byrne²

 

¹Division of Emergency Medicine, University of Cape Town, South Africa, ²Department of Emergency Medicine, University of Sydney, Australia

 

 

^&Corresponding author
Camillo Lamanna, Division of Emergency Medicine, University of Cape Town, South Africa

 

 

 

 

In the past decade, mobile phone ownership and mobile network infrastructure have rapidly expanded across sub-Saharan Africa. In tandem with this, there have been numerous initiatives to use new mobile health (mHealth) technologies to improve healthcare in the region. These have ranged from reminders [1], through to delivery of care [2], diagnostic pathways [3] and surveillance programs [4]. While the World Health Organization estimates that up to 50% of vaccines are wasted [5], only a minority of mHealth programs in Africa have been directed at vaccine supply chain optimisation. However, there have been successful pilots using mHealth technology to monitor supply chains for other medications [6, 7]. Indeed, with the introduction of newer, more expensive vaccines, the need for improved supply chains and logistics systems is greater than ever [8]. In Zambia, prior to 2015, vaccine stock levels were recorded manually using paper registries. There was no timely co-ordination of these individual registries at national level; information was often out-of-date by the time it had been received at the central warehouse in Lusaka [9]. In response to this situation, the Zambian Ministry of Health launched in 2015 a pilot using mHealth technology to enable real-time reporting of vaccine stocks. Over the course of 2016-7, the pilot was expanded and successfully rolled out across all provincial and district health facilities [10]. However, the project faced two main challenges.

 

Firstly, the mHealth intervention required Java- or smart-phones which could connect to mobile data networks. While the prevalence of mobile phones in Zambia is high and always increasing, rural smartphone penetration is low and access to mobile data in rural areas is highly variable [11]. This is a challenge for all mHealth supply chain management systems to operate effectively over the "last mile." Secondly, the reporting rate over the period 2016-7 was between 50-70%. Low uptake was felt to be multifactorial in nature, with proposed causes including a lack of sufficient training and understanding, and insufficient use of prompts or "nudges" to remind users to engage [12]. In response to these challenges, we designed an mHealth solution (called "Tessellate") dependent only on short message service (SMS) technology in order for workers in rural health centres (RHCs) to be able to report vaccine stock levels. In order to improve engagement, we incentivised users quasi-financially, by rewarding timely stock updates with free airtime, as well as prompting users to send stock updates by sending weekly reminder SMS messages. The objective of this study was to evaluate the potential of the Tessellate technology to offer a viable stock-tracking solution for RHCs in rural Zambia. This would be evaluated via a primary outcome measure of percentage-of-doses-tracked. The secondary objectives of the study were to establish if user engagement could be improved with increased airtime incentives or with increased frequency of SMS prompts.

 

 

Ethics statement: this study was approved by the Zambian Ministry for Health.

 

Study setting: the study was conducted in the Kazungula District in the Southern Province. This area is served by 21 public RHCs which cover a population of over 105,000 people. The distance between the these RHCs and the district pharmacy in Livingstone varied between 17-287km [13].

 

Study procedure: prior to commencement, the study team visited each of the 21 sites and conducted a 30-minute training session with the community health workers (CHWs) regarding the Tessellate system. They were given written materials and a poster to keep explaining the study, and a study telephone number which they could contact at any time for further information. For the duration of the study, between July and November 2017, identifying labels were placed on every box of pentavalent DTP vaccines at the district vaccine store in Livingstone. Each label displayed a unique 5-digit code. CHWs were asked to send this 5-digit code as an SMS to the project mobile phone number (printed on the vaccine box and on the written materials given during the site visit) when they collected the vaccines from the vaccine store; this "pick-up" message was the only unprompted message required. An automated reply was sent back, asking to which RHC the vaccines were being taken. Upon reply to this, an automated message was sent with a code for airtime. All users who had communicated a "pick-up" message were sent an automated message once a week asking how many doses remained in the box they had collected. If a user replied to this reminder message, they would receive a further airtime code. See Figure 1 for an example interaction between a user and the automated system. All messages received by the project mobile phone were automatically time-stamped and aggregated by a custom-written web platform which displayed recorded levels of vaccine stocks across all 21 RHCs in the district.

 

The primary outcome measure (percentage-of-doses-tracked) was defined as the total number of vaccine doses accounted for by the online platform at the end of the pilot period divided by the doses of vaccine which were collected from the district pharmacy during the same period. If no stock update message had been received regarding a particular box of vaccines in the past two weeks, or if no pick-up message had been sent by the CHW upon collection of the box, all remaining doses within the box were considered untracked. In order to achieve the secondary objective, namely, to establish if the response rate from users was affected by airtime incentive and/or reminder frequency, a 2 x 2 full factorial design was used, with airtime incentive (ZMK 4 vs ZMK 2, equivalent to US$0.40 and US$0.20 respectively) and reminder frequency (daily vs weekly) as factors. Importantly, the vaccine box labels did not display the airtime incentive amount or reminder frequency, and therefore these factors can have had no impact on whether or not CHWs sent an initial "pick-up" message. Therefore, for the secondary objective we measured median-time-to-reply rather than percentage-of-doses-tracked in order to measure any difference in messaging behaviour between groups. In boxes with "daily" reminder frequency, if the user did not respond to the weekly message asking for a stock update, then they were sent daily reminders until a reply was received. There were no such follow-up messages in the "weekly" reminder group. For this part of the statistical analysis, a 2-way analysis of variance (ANOVA) was used.

 

 

Of the 21 RHCs, responses were received from users from 10 facilities (48%); there were 13 unique users. Over the study period, a total of 7,900 doses of DTP vaccine were collected from the district pharmacy. Of these, complete tracking data were collected for 3,150 doses (39.9%). If an initial message "pick-up" message was sent to the platform upon collection of the vaccines, complete tracking data were collected for 93.8% of doses. Overall, there was no significant difference in average time-to-reply between the standard airtime incentive and double airtime incentive groups (13.3 hours (95% CI 2-24.5 hours) vs 7.5 hours (95% CI 3.6-11.5 hours) respectively, p = 0.42), nor was there a significant difference in average time-to-reply between the standard reminder and daily follow-up reminder group (16.4 hours (95% CI 4.3-28.5 hours) vs 4.0 hours (95% CI 1.9-6.1 hours) respectively, p = 0.08); although there was a trend for more timely replies in the daily follow-up reminder group, this did not reach statistical significance. There was no significant interaction between reminder frequency and airtime incentive on response time (F = 1.28(1, 68), p = 0.26). The mean airtime compensation sent per vaccine dose was ZMK 0.19 (USD$0.02) in the standard incentive group and ZMK 0.28 (USD$0.03) in the double incentive group. The collected data demonstrated that there was no consistent pattern with regard to the rate of DTP vaccine usage across facilities: the average time to use fifty doses (the number of doses in one box of DTP vaccine) was 10.5 weeks, however this demonstrated significant variability (Figure 2). Due to the low engagement rate, a post-hoc structured questionnaire was developed in order to identify the barriers to uptake from the 11 RHCs which did not send any update data to the project. All 11 RHCs were visited in order to administer the questionnaire and therefore the response rate was 100%. The majority (64%) of respondents cited inadequate understanding of the project as the main reason for their lack of engagement. On further questioning, it often transpired that key staff members had not been present on the training visits prior to commencement of the study period and therefore were not able to ensure that more junior staff bought into the project. A significant minority (27%) of respondents identified poor mobile phone signal as their reason for non-engagement.

 

 

The overall engagement rate found in this study - with just under half of RHCs choosing to participate - is broadly in keeping with existing evidence concerning the rate of engagement with stock tracking in Zambia [12]. Among those users who did choose to participate, however, the response rate was remarkably high, with most CHWs replying to stock update reminder messages in under a day. Within this active subgroup, near-real-time stock data were captured for 93.8% of doses, which is significantly above usual levels of reporting. It was unsurprising that there was a tendency towards more timely replies in the group who received daily reminder messages; this did not reach statistical significance in part because those in the weekly reminder group also replied promptly to their reminder messages. This study did not demonstrate any significant difference in engagement between standard and double remuneration groups. Previous studies in sub-Saharan Africa have shown that SMS-mediated financial incentives may increase patient adherence to medication [14], however there is ongoing and lively debate as to whether financial "bonuses" improve CHW performance in low-income settings [15, 16]. Research conducted in Zambia suggests that while conditional financial incentives may improve overall job satisfaction for CHWs, they do not significantly increase motivation [17]; this is consistent with findings in other sub-Saharan African countries [18, 19]. Finally, in similar settings it has been found that airtime is perceived to be less motivating than a cash transfer [20]; the findings from the present study would support the notion that non-monetary rewards do not significantly improve motivation in Zambian CHWs.

 

Strengths and limitations: one strength of this study was that it demonstrated the feasibility of using CHWs' own mobile telephones in order to record vaccine stock data. Previous pilot studies in which (smart)phones are distributed to CHWs are subject to the cost of phones as well as their loss, repair and theft - this significantly limits the scalability of such projects [21]. In the present study, the cost of incentivising users with airtime amounted to only USD$0.02 per dose. Moreover, as CHWs in sub-Saharan Africa often use their personal mobile phones for work-related communication [22], it seems sensible to take advantage of this existing resource. A further strength of the study was that its use of airtime rewards enabled a rapid positive feedback loop: allowing for lags in mobile communication, CHWs received their reward for stock updates typically within under ten seconds. Studies have shown that incentive schemes are frequently hampered by delays between positive behaviour and rewards [23, 24]; in the psychological literature, it is generally accepted that the more immediate a reward, the stronger the effect of positive reinforcement [25]. It is plausible, therefore, that the very high retention rate of users was in part due to the immediate receipt of the airtime reward.

 

One challenge for implementation included limited technological understanding of an automated SMS service - although clear instructions were given on the box, in written materials, and in the training session that users needed to send only the five-digit code to the project mobile phone number upon collection, a number of users sent lengthy natural-language messages explaining which vaccines they had collected and where they were being taken. It is possible that this limited understanding was a factor in the 52% of RHCs which did not send an initial "pick-up" message when they collected their vaccine stock. Multiple other studies conducted in Africa have also found that the feasibility of mHealth interventions in rural areas may be limited by technical barriers [26-29]. While a more comprehensive training course to explain the use of the SMS tracking system may mitigate this effect, it would also limit the scalability of the intervention. A second limitation of this study was that it did not capture the effect of vaccine tracking on rates and causes of vaccine wastage. This feature could be incorporated into the next iteration of the Tessellate technology: users could receive automated messages asking if any doses/vials from the boxes they collected have been wasted and, if so, for what reason. While this would undoubtedly gather useful information, there are a number of reasons why it may also be counterproductive. There is evidence to suggest that users of mHealth systems experience "message fatigue" [30, 31], asking users to report vaccine wastage as well as stock levels via SMS would significantly increase the number of messages required weekly. Moreover, due to the increased number of messages, the airtime costs of the project would more than double. Therefore, it may be more fruitful to send targeted messages to users who, for example, report erratic or disproportionately high vaccine usage rates.

 

 

This pilot study found that in an active subgroup of CHWs, a novel, incentivised mHealth solution was able to collect tracking data for 93.8% of vaccines. However, due to the significant number of CHWs who did not choose to engage with the study, the overall accuracy of tracking data was low. The study demonstrates that an automated mHealth system can record vaccine inventory in rural areas at low cost without relying on paper records and using CHWs' own mobile telephones; however, more research is required to identify techniques to encourage reluctant users to engage with such a system - in the present study there was no evidence that compensating users with airtime sufficiently incentivises users.

 

 

The authors declare no competing interests.

 

 

Both Camillo Lamanna and Lauren Byrne were involved in study design and data collection. Camillo Lamanna wrote the drafted the manuscript; both authors approve the final version submitted for publication.

 

 

Figure 1: example dialogue between community health worker (CHW) and automated short message service (SMS) replies

Figure 2: variability of DTP usage rate

 

 

1. Wakadha H, Chandir S, Were EV et al. The feasibility of using mobile-phone based SMS reminders and conditional cash transfers to improve timely immunization in rural Kenya. Vaccine. 2013; 31(6): 987-993. Epub 2012 Dec 13. PubMed | Google Scholar


2. Nichols M, Fred Stephen S, Singh A et al. Assessing Mobile Health Capacity and Task Shifting Strategies to Improve Hypertension Among Ghanaian Stroke Survivors. The American Journal of the Medical Sciences. 2017; 354(6): 573-580. Epub 2017 Aug 12. PubMed | Google Scholar


3. Sutcliffe CG, Thuma PE, van Dijk JH et al. Use of mobile phones and text messaging to decrease the turnaround time for early infant HIV diagnosis and notification in rural Zambia: an observational study. BMC Pediatrics. 2017; 17(1): 66. PubMed | Google Scholar


4. Mwingira U, Chikawe M, Mandara WL et al. Lymphatic filariasis patient identification in a large urban area of Tanzania: an application of a community-led mHealth system. Specht S, ed PLoS Neglected Tropical Diseases. 2017; 11(7): e0005748. eCollection 2017 Jul. PubMed | Google Scholar


5. WHO. Monitoring vaccine wastage at country level: guidelines for programme managers. Geneva: WHO; 200; 38-40. Accessed 14 Mar 2018.


6. Namisango E, Ntege C, Luyirika EBK, Kiyange F, Allsop MJ. Strengthening pharmaceutical systems for palliative care services in resource limited settings: piloting a mHealth application across a rural and urban setting in Uganda. BMC Palliative Care. 2016; 15: 20. PubMed | Google Scholar


7. Shieshia M, Noel M, Andersson S et al. Strengthening community health supply chain performance through an integrated approach: Using mHealth technology and multilevel teams in Malawi. Journal of Global Health. 2014; 4(2): 020406. PubMed | Google Scholar


8. Zaffran M, Vandelaer J, Kristensen D et al. The imperative for stronger vaccine supply and logistics systems. Vaccine. 2013; 31(S2): B73-80. PubMed | Google Scholar


9. WHO. Zambia improves real-time tracking of vaccines, reduces stock outs. Apr 2017. Accessed 14 Mar 2018.


10. BLN Discussion Meeting. BID Learning Network Discussion Meeting. Lusaka, Zambia. Sep 19-22, 2017. Accessed 14 Mar 2018.


11. infoDev Growing Innovation. Mobile at the Base of the Pyramid: Zambia. Washington, DC: World Bank. 2014.


12. Logistimo. Improving last-mile stock manager experiences and engagement in Zambia. 10^th Global Health Supply Chain Summit. Accra, Ghana Nov 15-17, 2017. Accessed 14 Mar 2018.


13. Zambian Ministry of Health. The 2012 List of Health Facilities in Zambia. Lusaka, Zambia. Accessed 14 Mar 2018.


14. Gibson DG, Ochieng B, Kagucia EW et al. Mobile phone-delivered reminders and incentives to improve childhood immunisation coverage and timeliness in Kenya (M-SIMU): a cluster randomised controlled trial. Lancet Global Health. 2017; 5(4): e428-e438. PubMed | Google Scholar


15. Kalk A, Paul FA, Grabosch E. Paying for performance in Rwanda: does it pay off? Tropical Medicine & International Health. 2010; 15(2): 182-190. Epub 2009 Nov 17. PubMed | Google Scholar


16. Paul E, Albert L, Bisala BN et al. Performance-based financing in low-income and middle-income countries: isn't it time for a rethink? BMJ Global Health. 2018; 3: e000664. Google Scholar


17. Shen GC, Nguyen HTH, Das A et al. Incentives to change: effects of performance-based financing on health workers in Zambia. Human Resources for Health. 2017; 15(1): 20. PubMed | Google Scholar


18. Greenspan JA, McMahon SA, Chebet JJ, Mpunga M, Urassa DP, Winch PJ. Sources of community health worker motivation: a qualitative study in Morogoro Region, Tanzania. Human Resources for Health. 2013; 11: 52. PubMed | Google Scholar


19. Lohmann J, Wilhelm D, Kambala C, Brenner S, Muula AS, De Allegri M. The money can be a motivator, to me a little, but mostly PBF just helps me to do better in my job. An exploration of the motivational mechanisms of performance-based financing for health workers in Malawi. Health Policy Planning. 2018; 33(2): 183-191. PubMed | Google Scholar


20. Wakadha H, Chandir S, Were EV et al. The feasibility of using mobile-phone based SMS reminders and conditional cash transfers to improve timely immunization in rural Kenya. Vaccine. 2013; 31(6): 987-993. PubMed | Google Scholar


21. Sanner TA, Roland LK, Braa K. From pilot to scale: towards an mHealth typology for low-resource contexts. Health Policy and Technology. 2012; 1(3): 155-164. Google Scholar


22. Hampshire K, Porter G, Mariwah S et al. Who bears the cost of "informal mhealth?" Health-workers' mobile phone practices and associated political-moral economies of care in Ghana and Malawi. Health Policy and Planning. 2017; 32(1): 34-42. Epub 2016 Jul 31. PubMed | Google Scholar


23. Fox S, Witter S, Wylde E, Mafuta E, Lievens T. Paying health workers for performance in a fragmented, fragile state: reflections from Katanga Province, Democratic Republic of Congo. Health Policy and Planning. 2014; 29(1): 96-105. Epub 2013 Jan 15. PubMed | Google Scholar


24. Ssengooba F, McPake B, Palmer N. Why performance-based contracting failed in Uganda: an 'open-box' evaluation of a complex health system intervention. Social Science & Medicine. 2012; 75(2): 377-83. Epub 2012 Apr 20. PubMed | Google Scholar


25. Lattal KA. Delayed reinforcement of operant behaviour. Journal of the Experimental Analysis of Behaviour. 2010; 93(1): 129-139. PubMed | Google Scholar


26. Pérez GM, Swart W, Munyenyembe JK, Saranchuk P. Barriers to pilot mobile teleophthalmology in a rural hospital in Southern Malawi. The Pan African Medical Journal. 2014; 19: 136. eCollection 2014. PubMed


27. Wallis L, Blessing P, Dalwai M, Shin SD. Integrating mHealth at point of care in low- and middle-income settings: the system perspective. Global Health Action. 2017; 10(sup3): 1327686. PubMed | Google Scholar


28. Phillip Sundin,Jonathan Kallan,Khanjan Mehta. Why do entrepreneurial mHealth ventures in the developing world fail to scale? Journal of Medical Engineering and Technology. 2016; 40(7-8): 444-457. Google Scholar


29. Anstey Watkins JOT, Goudge J, Gómez-Olivé FX, Griffiths F. Mobile phone use among patients and health workers to enhance primary healthcare: a qualitative study in rural South Africa. Social Science & Medicine. 2018; 198: 139-147. Epub 2018 Jan 10. PubMed | Google Scholar


30. Pop-Eleches C, Thirumurthy H, Habyarimana JP et al. Mobile phone technologies improve adherence to antiretroviral treatment in a resource-limited setting: a randomized controlled trial of text message reminders. AIDS (London, England). 2011; 25(6): 825-834. PubMed | Google Scholar


31. Baseman JG, Revere D, Painter I, Toyoji M, Thiede H, Duchin J. Public health communications and alert fatigue. BMC Health Services Research. 2013; 13: 295. PubMed | Google Scholar