Lewis Carroll points to an important consideration when embarking on a learning exercise: if we want to prove or improve our work then we need to be able to describe clear intentions to direct our learning. We need to care where we are going. Without clear intentions, we are at liberty to define success in any way we like. This may sound appealing to some but is more likely to result in repetitive circles than learning.
The impact of academic research is traditionally evaluated via peer review to assess quality, relevance and accuracy; and citation analysis to assess uptake and reach. While both of these are important, neither helps us discover what influence the research may have had on policy (assuming it had an intention to do so), whether the research was worth undertaking and hence how to make it more effective. All we learn is how to make our research more attractive to other researchers!
Traditional M&E approaches – which rely on a simple feedback model with predefined indicators, collecting data and assessing progress towards pre-set objectives – are simply not adequate in the context of policy-influencing interventions. As explored in Chapter 1, many of the results we are looking for cannot be projected ahead of time in a linear fashion. The reality of the distributed capacities, divergent goals and uncertain change pathways that pervade many policy contexts means measuring progress along a predefined course is insufficient for monitoring.
Effective M&E requires a careful combination of sensing shifts in the wider context (policy, politics, economics, environmental, social), monitoring relationships and behaviours of diverse actors, weighing up different sources of evidence, being open to unexpected effects and making sense of data in collaborative enquiry. This kind of monitoring may seem challenging but it doesn’t have to be. ROMA aims to shift the emphasis from evaluation and more to ‘sense-making’ of monitoring information. This fits into current management practices to ensure decisions on responding to an unpredictable situation are evidence-based and widely owned.
The purpose of this chapter is to provide a practical monitoring approach that builds reflective and evaluative practice into the work of influencing policy, to support decision-making and demonstrate progress. The chapter is split into three: the first part describes why and what to monitor; the second introduces practical options for embedding and carrying out this kind of approach; and the third studies how to make sense of learning and decision-making. The principles that underlie this approach can be summarised as:
Appropriate to purpose, scale and context. In ROMA, the primary driver for monitoring is the users and how they will use the data and insights. But scale and context are also determinants. A small-scale intervention will require much lighter monitoring than a long-term, multi-strategy intervention. (If you are not sure of the scale of the intervention at this stage, Chapter 2 will guide you through the planning process.) As with context, simple problems will require only routine monitoring and performance management, whereas problems exhibiting one or more signs of complexity will need more sophisticated, responsive and multi-purpose monitoring systems. (If you are unsure about the level of complexity, Chapter 1 will introduce you to three clear signs to look for.)
Defines realistic results within the sphere of influence. The influence of an intervention has a definite limit based on resources, time, reach, politics etc. Beyond the sphere of influence is the sphere of concern, which is where the results that really matter lie (such as better education, quality health care, secure livelihoods). However, you have to rely on others to influence these results. ROMA considers only results within your sphere of influence. These are the ones that can be measured and can guide strategy and engagement. The planning stages in Chapter 2 as well as the monitoring areas and measures in this chapter are used to define the intervention and its sphere of influence. They point to the priority areas to monitor.
Focuses on actors and graduated change. Much policy-influencing work revolves around people. It follows that monitoring policy influence should also revolve around people. In ROMA, an intervention is monitored through its effect on key stakeholders – those people or organisations within the sphere of influence of the intervention and whom the intervention seeks to influence directly or indirectly. ROMA recognises that effects can come in many guises and it is important to be able to pick up a broad spectrum – the simple, immediate responses that show you are on the right track as well as substantive commitments that indicate you are close to your goal.10
Reasoned judgement about statistical significance. ROMA is an inductive approach that seeks to generate evidence that can be used to increase our understanding of our effect on policy. It does not seek to determine a statistical, numerical measure of policy influence.
3.1 What to monitor and why
Deciding what to monitor is an important first step in building a strong M&E plan. In order to choose which indicators to monitor, you need to know why you are doing it: the purposes behind M&E are usually viewed in terms of learning and accountability. But we need to be more specific. Read more
3.2 How to monitor – collecting and managing data
This section divides data collection into real-time collection methods and retrospective study methods. Find out about which methods are particularly suited to monitoring policy influence and engagement. Read more
3.3 Making sense of learning and decision-making
The information and insight gathered through monitoring and evaluation is useless if you don’t make space and time to make sense of your findings and decide what to do with them. Read more