The ‘Gupshup Potli‘ usability testing reminded me of my first day as part of the Shaping Demands and Practices project team at BBC Media Action.
It was my first meeting with our donors – the Bill & Melinda Gates Foundation (BMGF) – reviewing achievements and discussing how we could increase the reach of our project’s interventions among target audiences. It was in this meeting that the idea of leveraging Village Health Sanitation and Nutrition Days (VHSNDs) to reach pregnant women and mothers of children aged 0-2 years arose.
The next hour of the meeting was spent debating the advantages and challenges in leveraging VHSNDs. In the end, we decided to explore the possibility. Little did I know that we would spend the next few months breathing, talking and thinking about VHSNDs!
Various teams at BBC Media Action worked on producing content, identifying platforms and operational models. Once a prototype was created, we needed to test its efficacy. Here are some of the lessons we learned during the testing process:
The scale of opportunity
Our previous experience has taught us that our target audiences might not think the way we do. We wanted to create a product that would provide our target audiences with access to information effectively and conveniently. Considering the potential reach of the VHSND platform – roughly 20,000 VHSND sessions take place every month across eight focus districts and millions of women attend them – we could not afford to get it wrong.
The user-test entailed systemic observations under controlled conditions aimed at understanding how efficiently Community Health Workers (CHWs) could use our ‘product’ – an audio service played through mobile phones, connected to speakers (more about Gupshup Potli). It involved both qualitative techniques and usability testing in combination to better appreciate users’ motivations, perceptions and actions.
Operations and logistical efficiency
The study tested our operational capabilities as much as our research techniques. Six lab tests were to be managed simultaneously across three districts – wherein recruitments had to be done both before and after the VHSND. It gave us nightmares! Getting time from Auxiliary Nursing Midwives (ANMs) and Accredited Social Health Activists (ASHAs), our primary respondents, after their day’s duties – when they had to travel far back to their villages – wasn’t easy. Added to our worries were the erratic electricity supply (interrupting lab test recordings), availability of safe and conducive places for lab tests and transportation issues.
Don’t underestimate users’ capabilities
The first aspect of the usability testing focused on technology uptake among the ANMs – the custodians of the VHSND. Our hypothesis that our target audiences might have technology phobias and feel uncomfortable handling speakers with mobile phones proved incorrect. The CHWs displayed overt levels of confidence in using the technology.
ANMs, our target audience for the usability testing, were briefed to perform the following six tasks:
• Attaching the charging plug to cable
• Connecting the charger to the speaker
• Switching on the speaker
• Adjusting the volume of the speaker
• Connecting the speaker to the mobile
• Dialing the IVR number
To gauge spontaneous reactions, we decided not to help the ANMs unless they failed to complete a task three times in a row. Surprisingly there were very few cases where we had to help. ANMs were very willing to use their mobile and the speaker and explained to us how it would help them in their work by attracting women’s attention.
Technology phobia appeared to be of least concern – the ANMs intuitively recognised the ports for inserting the relevant speaker cable. Some minor issues were observed in handling the equipment such as confusion around the charging light of the speaker, adjusting volume, phone jacks being incompatible with the AUX wires of the speakers etc.
We were confident that these minor issues observed during the user tests could be easily ironed out by training the CHWs. However, it was particularly the comfort and confidence displayed by the ANMs while handling the technology that gave us the conviction to carry on.
Researchers at the usability testing lab
Less is more
The next step was to assess feedback on the content of the service. The content was also tested among the ultimate beneficiaries of the communication – that is mothers, aunts and grand mothers of children aged 0-33 months. The good news was that our key character Dr Anita was recalled by most of the respondents. But, recall and comprehension of messages was not very encouraging. We realised that there were too many messages in one audio capsule impacting the pace of delivery and therefore comprehension. We therefore decided to have only one clear communication objective in each audio capsule.
The ultimate test
The ultimate test of the users’ ability to handle the technology, and of the content, was on the VHSND day, when we did live testing. It helped us understand whether our operational model was feasible, effective and efficient. A full day spent at eight different VHSNDs provided us with insights that we would never have understood otherwise.
It was apparent at the VHSND location that ANMs had to spend most of their time on immunisation related work. On an average across eight locations, ANMs could play the audio content only once during the VHSND – with a few exceptions. ANMs suggested they could play the content at the end of VHSNDs or during other meetings. But organising a separate meeting was not considered a viable option.
Responsive, reflective & flexible methodology
Once we learned how overstretched ANMs were, the big question was who else we should involve to ensure our intervention reached a maximum number of women. We had only two alternatives – ASHAs or Anganwadi Workers (AWWs). ASHAs are very busy gathering women for the VHSND sessions, which meant AWWs were our most viable option. So we needed to test this operational model with AWWs – then and there. This meant revisiting research instruments, increasing sample size, and re-recruiting new respondents (AWWs) for user tests without extending timelines. The flexible research methodology enabled us to manage these challenges without affecting the quality.
Build upon existing infrastructure & systems
ANMs are the upholders of the VHSNDs, and providing job aids to AWWs at the VHSNDs could have led to discomfort between the two. To avoid this, we agreed to make ANMs custodian of the equipment and the event, while AWWs provided support for playing the content and following up with discussions with the beneficiaries. We briefed ANMS about our approach, and they agreed. Things worked almost perfectly – the AWWs played the content thrice, and were able to generate ‘Gupshup’ (conversations) with the target beneficiaries.
The ‘Gupshup’ created by the AWWs and ANMs with the mothers and grandmothers of children resulted in ‘Kilkari’ (laughter), and re-affirmed our conviction that placing our audience at the centre of design process works.