BBC Media Action has adopted an action research cycle approach to inform project interventions and gauge impact.
There are two fundamental reasons for trying to assess the impact of interventions:
Accountability: is the donor investment achieving the expected results?
Decision making: what is working and what is not? What can make a bigger difference?
A theory of change
Shaping Demand and Practices uses a combination of quantitative and qualitative research to answer these seemingly simple, but relatively difficult to answer questions. As a starting point, BBC Media Action designed a hypothesis of how change will occur through the project, a ‘theory of change’, that guides not only our interventions but also how we measure changes and ascribe them to our work. The theory of change begins by asking some fundamental questions:
What problems are we trying to solve?
How do we think change will happen and how long will it take?
How much money and resources will it take?
What will success look like and what will be the signs of progress on the way?
The answer to these questions and the resultant theory of change provided a framework for designing the monitoring and evaluation plan for the project.
The project uses formative assessments, quantitative cross-sectional surveys, longitudinal qualitative research, impact assessments, rigorous experimental designs, cluster evaluations, primary MIS data as well as secondary data through other Ananya partners to assess the impact of the project and its interventions.
Establishing baseline
The project kicked off with a baseline survey to establish benchmarks of knowledge, attitude, practices and self-efficacy of target groups vis-a-vis maternal and child health issues in the selected eight project districts of Bihar. In addition, social norms of the target communities were identified to comprehend existing realities, and ascertain triggers & barriers of desired behaviour. Click here to explore the results of the baseline survey.
The baseline survey highlighted problem areas that further guided project interventions, and helped set project’s goals and milestones. Various project interventions including Mobile Kunji (IPC tool kit), street theatre, television advertisements and radio programmes were designed to reach out to a large majority of people.
User testing of these interventions was conducted to explore comprehension, effectiveness of the medium chosen, usability (for mobile based interventions), and efficacy in terms of achieving behaviour change. Suggested corrective actions were taken to finalise project interventions.
Assessing impact of interventions
As well as this, various impact assessment studies are being conducted to assess the impact of specific outputs. These assessments measure indicators such as reach and recall of interventions, comprehension of content, intent to change behaviour, direct impact on decision making at household level, impact on generating inter-spousal or family level discussions etc.
To assess the actual contribution of our interventions we use a matched sampling design of pre versus post exposure or exposed versus non-exposed respondents, minimizing the potential variance caused by other factors such as demographics, impact of other programs etc. The impact assessments are designed to ensure that we continue to make the right investments.
Measuring change
The baseline survey established reference values for specific behaviours among each target groups. These values will be used mid-way during the project to identify intermediate indicators of change to ensure that the project is on track.
Plans are also underway to conduct rigorous evidence research for specific innovations, so that efficacy can be quantified and results can be shared more broadly. The aim is to help consolidate most potent investments and to help facilitate decisions related to scaling of these innovations beyond the eight priority districts in Bihar, as well as in other states and countries.
End-line survey will be conducted at the end of the project to measure absolute change in the knowledge, attitude, self-efficacy and practice of target groups. Lessons and learning from this project could be transferred to other projects across India and in other countries.