ROMA is a suite of tools that an organisation or team can use at any stage of their policy engagement process to improve:
- how they diagnose the problem,
- understand the types of impact their work could have on policy-making,
- set realistic objectives for policy influence,
- develop a plan to achieve those objectives,
- monitor and learn from the progress they are making
- and reflect this learning back into their work.
This guide can be used in different ways by different people. Each problem requiring policy influence is unique, and the context in which you apply ROMA will be unique to that problem. So ROMA is best used with a degree of flexibility: choose where to begin based on your priorities or your role. Who are you?
ROMA’s main message is to consider policy engagement as an integral part of the research or implementation process. Clearly, it is important that any attempts to engage or influence are based on robust evidence: for research projects and programmes in particular there is a tension between wanting to wait until the results are conclusive before communicating them, and wanting to engage during the life of the programme to be assured of a receptive audience when the results do emerge. Using ROMA, you should be able to mix the two approaches, communicating throughout the life of your project so your stakeholders are aware of what may emerge and are able to consider possible responses.
Chapter 1’s framework for diagnosing the issues you are facing not only will not help structure your engaging and influencing strategies but also may help you refine what you do and whom you work with to bring about change.
Chapter 2’s workshop-based approach will help you decide whom to involve at what stage of the process to ensure your work is as cost-effective as possible.
Chapter 3’s templates have been designed so the information you collect on your progress fits seamlessly into the way you manage your projects and programmes, and enables you to report effectively to donors. They will also ensure you can streamline the learning that happens (from both successful and unsuccessful influencing attempts) into your work planning and team management.
Monitoring and evaluation specialists
ROMA links monitoring to ongoing learning, not just to evaluation. In the face of complex problems, the ubiquitous phrase ‘M&E’ can be unhelpful: unpredictability means it is better to rely on ongoing monitoring to guide your work, rather than initial planning to guide your work. Of course, evaluations are a key part of any project or programme, and the requisite information must be collected so that they are robust. ROMA suggests shifting the emphasis so more weight is given to ‘sense-making’ of monitoring information, fitting it into current management practices to ensure decisions about how to respond to an unpredictable situation are evidence-based and widely owned.
Chapter 3 clarifies why we monitor, as the three pressures from donors (demonstrable impact, value for money and the need to address very complex development problems) can overwhelm monitoring systems. It may require a little initial effort to design the systems, but this will pay off in the long term as information collection becomes more focused on supporting the decisions that will need to be taken and less on collecting it just in case it might be useful.
Chapter 1 shows why ROMA shifts the emphasis from ‘M&E’ to ‘M&L’ (monitoring and learning). The nature of complex problems means the ability to adapt is key, and this chapter helps you decide on what and with whom ongoing learning needs to occur.
Chapter 2 will help you develop a monitoring and learning system that is broadly based and widely owned. Complex development problems are likely to involve many stakeholders, and the workshop tools suggested in this chapter will bring them together around a shared vision for the project or clarify where there is disagreement and divergence. It will ensure the learning process does not ride roughshod over those with weaker voices.
Researchers and practitioners
People in the thick of implementing a project or conducting a piece of research may sometimes find it hard to step back from their work to consider what else could be done, or how to do things differently. The danger is of proceeding without testing assumptions about whom to collaborate with, what to do or how to communicate the evidence emerging from the work.
Chapter 1 will help you do just that, giving you a structured way of thinking about the issue you are working on, exploring the root causes and getting at a clear diagnosis that may affect the way you conceive of the issue and how you work on it. Clearly, work programmes cannot be thrown up in the air as soon as you notice a change in the wider context, but it is important to reflect on how those changes will affect the wider ramifications of what you are doing.
Chapter 2 gives you a set of practical tools that will help bring your partners and other stakeholders into this assumption-checking process. Although the workshops are not designed specifically to do this, they offer a space for you to reflect – with your collaborators – on what you are doing and whether there is anything any of you could do more, or differently, to promote sustainable change.
Chapter 3 shows why monitoring should not be left just to the specialists: it is not enough to continue with the work programme you initially defined and hope an external evaluator will mark you highly. Monitoring needs to be a whole-team process, which everyone understands and to which everyone contributes.
Good communications are a central pillar of the ROMA process. Achieving policy influence, particularly in complex policy environments, relies heavily on ensuring messages are well communicated in the right language to the appropriate audiences when they need them. The challenge for communicators is to ensure these audiences include people internal to the project or programme as well as those outside.
Chapter 2 shows why this is the case and helps you develop a strategy: ROMA’s workshop-based approach brings diverse stakeholders together. Maintaining good communications between them needs to happen for the life of your project or programme.
Chapter 1 reveals the importance of good communications and helps you focus on the key communication challenges. Communicating the nature of the problem you are facing is a key part of understanding what to do and how to do it; the more complex the problem, the more important effective, multi-directional communication is. Internal communications are as important as external communications, ensuring the project team shares this understanding will help to build links between the project and its wider stakeholders.
Chapter 3 will help you build a communications strategy that is constantly informed by the latest monitoring evidence. The larger the project or programme, the more important it will be for specialist communicators to be included in monitoring processes to ensure learning is as widely shared as possible.
Policy-makers and civil servants
Policy-makers can use the ROMA toolkit to improve their own strategies for changing policy or practice within their departments or ministries. Where issues cut across departmental boundaries or where it is important to engage a variety of external stakeholders, the ROMA principles and toolkit have even more to offer.
Chapter 1 shows that, for large cross-cutting issues, diagnosing the challenges faced may give you a set of different ideas about whom to engage with and how.
Chapter 2 sets out some useful tools you can use to map your own stakeholders, understanding where you might be able to use outsiders (such as researchers or non-government organisations) to help reinforce your position. It also helps you consider more closely what outcomes you might be seeking.
Chapter 3 provides insights into how your own policies can be monitored and how to draw learning from them effectively. Discussing the principles and practices in Chapter 3 with your delivery organisations will help you come to a good shared understanding of how to prioritise what you need to monitor, with the resources you have, to give you the picture of how effective your policies and programmes are.
Donors and research commissioners
Donors recognise the challenges of working in complex situations and welcome multiple approaches to solving them. However, this does not always translate well into the nature of the impacts they seek, which can over-emphasise the delivery of outputs according to predefined plans and under-emphasise the importance of adaptation and learning.
Chapter 1 shows the importance of diagnosing the complexity of a policy issue, and the three key aspects that need to be considered when designing projects and programmes to bring about policy change. Not all issues are complex, but it is important to ensure the tools and techniques used to design, implement, monitor and learn from strategies for policy influence are appropriately tailored to the nature of the issue at hand.
Chapter 2 shows why it is important to encourage those implementing projects and programmes to allow sufficient time for broad engagement throughout, not just at the beginning. This has resource implications: workshops are not cheap, but internal communications within messy partnerships are essential if everyone is to be able to contribute appropriately to broad-based sustainable development.
Chapter 3 shows why it is important to focus on learning when the goal is to influence policy change around complex problems. Approaches to monitoring based on Outcome Mapping complement more traditional programme and project management techniques, helping unpick the behavioural assumptions that often weaken theories of change (particularly between output and outcome levels in a logframe). To cope better with complexity, it would be helpful for donors to drop the term ‘M&E’ from the donor lexicon, changing the emphasis to M&L, and seeing evaluation as a separate issue.