How to Design Better Programmes in Complex Systems
A Palladium team recently published a paper in the American Journal of Evaluation. The paper outlines a new framework for improving the design and evaluation of complex programmes.
A programme theory (or a 'theory of change') is a key tool used to inform the design and delivery of development programmes. A programme theory outlines the steps in a pathway, from intervention to impact, and explicitly identifies the key risks and assumptions along that pathway. This helps ensure that a programme is designed from a strong evidence base and adapted for a particular context. Developing a programme theory is now accepted as good practice in the design of any development intervention, and is often required by programme funders and commissioners.
But developing programme theories has become more complex in recent years. First, programmes themselves have become more complex and multi-faceted, comprising multiple interventions in multiple locations, and targeting multiple stakeholder groups. In addition, programmes are trying to catalyse change in complex settings and environments, where a diverse range of factors, both within and outside of the programmes' control, is at play. A vacillating political landscape, unpredictable weather conditions and a changing security situation, for example, can all affect a programme’s ability to achieve results.
As complex programmes in complex settings become more common, it is increasingly important for programme designers to begin addressing some of this complexity in how they develop programme theory. So how can this be achieved in practice?
We propose three ways through which to better unpack some of this complexity in programme theory. We've codified our experience across a number of programmes and evaluations into a new approach called the Actor-Based Change (ABC) Framework, which is described in detail in a paper published in the American Journal of Evaluation.
1. Map out the actors in the system
Though development programmes are vastly different, many aim to address a development problem by changing the behaviour of specific groups of actors. Desired behaviours range from farmers adopting a new technology to improve yields to businesses investing in pro-poor business models to police officers being more responsive to women’s security concerns.
The literature around complexity and systems thinking has taught us that often times it is not enough to simply target actors with direct interventions. Systems thinking illustrates how actors are influenced by a network of relationships with other actors– their friends, colleagues, local authorities – and in response to their changing environments, whether social, economic or ecological. These various influences on behaviour are important to understand if we are to design effective interventions in a sustainable manner.
For instance, a development programme aimed at improving local police practices through a direct delivery approach might deliver trainings to support positive police practices. And indeed in the short term this might contribute to positive practice change. However when the programme ceases activity, the trained police might be transferred, and new police who join the force won't have been trained.
Looking at this same aim through a systems lens would help to identify the other groups who would also need to change their practice to sustain this change in local police practice in the long term. This could include revising the police training curriculum at the national level and ensuring that police superiors understand the importance of ensuring an adequately trained police force in transferring decisions.
2. Understand how behaviours change
To change the behaviours of a network of actors, a programme theory must identify what's driving current behaviours and how these drivers need to shift. The literature on behavioural science offers a number of useful tools, including the Behaviour Change Wheel from Michie et al. The Behaviour Change Wheel identifies three underlying determinants for behaviour: capabilities, opportunities and motivations. For instance, in the example above, some of the behavioural determinants of a farmer adopting a new technology would include the following: their knowledge of and skills in using the technology (capability), their access to and affordability of the technology (opportunity) and their willingness to adopt it, including seeing the benefit of the new technology and its cultural acceptability (motivation).These apply to a variety of actor groups, from individuals to organisations, firms and institutions. Applying a behavioural change framework, such as the Behaviour Change Wheel, to each actor group in programme theory then provides a better sense of what combinations of behavioural conditions need to shift in order to bring about the desired changes in the system.
3. Chart out the pathway from intervention to actor-based behavioural change
From this behavioural analysis, we can design interventions to shift the behavioural conditions for each actor group and bring about the desired change. It is then possible to chart out the detailed steps from programme intervention to changes in behavioural conditions to changes in behaviour for each actor. In this way, we can create a pathway from intervention to actor-based behaviour change.
We can use this pathway to identify the main risks and assumptions at each step of the pathway. Identifying actors' capabilities, opportunities and motivations is particularly useful here. It allows us to identify the behavioural conditions for each actor which are not addressed by the intervention but which must be met for behaviours to sustainably change.
The impact pathway provides the framework for establishing a robust measurement system to track whether interventions are working as intended: whether the target groups are being reached with interventions, and whether being reached with these interventions is leading to the desired behavioural change. This information, collected on a regular basis, then supports programme adaptation and improvement.
Better programme theory leads to better programme design
Applying these approaches on programmes and evaluations, whether as articulated in the ABC Framework or in combination with other approaches, can better capture complexity while strengthening causal logic. This means that your programme’s interventions will be better grounded in a strong evidence base, adapted to the local context and designed to ensure its long-term sustainability.
Taking a broad, system-oriented lens helps to situate a programme or intervention within the larger system of actors, helping programme teams be more honest about what changes they can expect to achieve, and in what timeframe. This means that programme designers can more realistically assess realistic results in the project timeframe to assist in developing and revising logical frameworks, and setting achievable programme outputs and milestones.
This article was written by Andrew Koleros, Director of Monitoring, Evaluation and Learning at Palladium. Additional contributors include Sean Mulkerne, Mark Oldenbeuving, and Danielle Stein.
Click DOWNLOAD to access the full research article or read in the American Journal of Evaluation.