Chapter 3, Module 3: Conflict-sensitive monitoring and evaluation

Purpose of module

To help the reader to design and implement a monitoring and evaluation system that captures the interaction between project and context, and to identify relevant indicators to monitor this interaction.

The monitoring system should seek to measure the impact of the intervention on the changing context and vice versa, and to enable programming to be adjusted if necessary to ensure optimum conflict sensitivity.

The evaluation system should seek to identify lessons for improving conflict-sensitive planning and implementation in the future.

Contents

1. Conflict-sensitive monitoring and evaluation
2. Key steps in conflict-sensitive monitoring and evaluation
3. Key issues in conflict-sensitive monitoring and evaluation
4. Endnotes
5. Annex 1: Sample indicators — links between context changes and project, and project changes and context.

1. What is conflict-sensitive monitoring and evaluation?

Monitoring is the process of regularly examining a project's actual outputs and impacts during implementation. It provides the project team with current information that enables them to assess progress in meeting project objectives, and to adjust implementation activities if necessary. It also generates data that can be used for evaluation purposes.

Conflict-sensitive monitoring will enable project staff to gain a detailed understanding of the context, the intervention, and the interaction between the two. It introduces an understanding of conflict actors, profile, causes and dynamics into traditional monitoring processes and activities to inform required adjustments and changes to project or programme activities. In this way, conflict sensitive monitoring helps ensure the intervention has as positive an impact as possible on conflict dynamics.

Evaluation is a one-off assessment that typically takes place at the end of a project, although it can also be undertaken as a mid-project review. On the basis of systematically applied objective criteria, an evaluation assesses the design, implementation and overall results of an ongoing or completed project in relation to its stated goals and objectives.

Conflict-sensitive evaluation introduces a detailed understanding of actors, profile, causes and dynamics into traditional evaluation activities and processes.

Conflict-sensitive evaluations are used to understand the overall impact a given intervention has had on its context, and the context on the intervention. These evaluations can then be used to adjust subsequent phases of an ongoing initiative, and / or provide lessons for future initiatives.

In other words, while traditional monitoring and evaluation focus primarily on assessing the intended and actual outputs of a given project, conflict-sensitive monitoring and evaluation also requires:

• an understanding of the context as it changes over time
• measuring of the interaction between the project and the context.

It should be noted that conflict sensitive monitoring and evaluation is still in the early stages of development; this module presents thinking current at the time of writing but should not be taken as the definitive statement on the subject. Outstanding challenges include, for example, the development of indicators (see steps 2 and 3 below). Because conflict-sensitive monitoring and evaluation is very much context- and activity- specific, it is difficult to offer specific guidance; indicators that are useful in one case are generally not transferable to other situations. Despite the challenges, this module does offer a perspective on current thinking in the area of conflict-sensitive monitoring and evaluation, including some new approaches to outstanding issues

2. Key steps in conflict sensitising monitoring and evaluation

Broadly speaking, traditional monitoring and evaluation processes are organised around the following steps:

The introduction of conflict sensitivity into the process does not change these basic steps; it does, however, change the way in which they are applied. The main differences are highlighted in this module.

2.1 Step 1: Decide when to monitor and evaluate

Traditional monitoring and evaluation processes are typically organised around pre-defined timeframes outlined in the project documents (eg quarterly and annual reports, end of project, new project phase). Integrating conflict sensitivity into the monitoring and evaluation processes and activities may require changes in timing to relate the timing of these processes to significant aspects of the conflict profile, causes and dynamics identified by the conflict analysis.

For example, a project's regularly scheduled monitoring work may inadvertently take place at the same time as an election or the period leading up to it, but the understanding of the local context gained from the conflict analysis may suggest that this is not appropriate (eg if there is a history of violence during election times, reliable information may be harder to obtain then than at other times). Alternatively, an evaluation trip may be scheduled to avoid monsoon rains or cold winter months in order to facilitate travel, logistics and comfort, but evaluators may then miss important aspects of human interactions and attitudes prevalent at those times and crucial to the assessment. Decisions about when to monitor and evaluate which are dictated by institutional and funding requirements should be systematically reviewed to assess the impact of the preferred timing on the context — that is, through linking the proposed timing to the conflict analysis. Such timing adjustments may prove challenging to both financial reporting requirements and funders.

2.2 Step 2: Design monitoring and evaluation process

In addition to typical outputs from traditional monitoring and evaluation, conflict sensitive monitoring and evaluation assesses the interaction between the context and the project. In order to understand this interaction the process should be designed around three primary issues: (a) understanding the context and changes in the context; (b) understanding the intervention, including its implementation; and (c) measuring the interaction between the two.

(a) Understanding the changing context

As outlined in Chapter 2, a conflict analysis can be used to provide an understanding of the context in which project interventions are situated, and to track changes that occur. In particular, the conflict indicators developed at the conflict analysis stage will help systematically monitor changes in the context in terms of conflict profile, causes, actors, and dynamics.

However, some organisations may not have a conflict analysis at the time they want to start sensitising their monitoring or evaluation; or they may have a conflict analysis that has become outdated.

For monitoring purposes, if a conflict analysis does not exist it will suffice to conduct a current analysis and to begin incorporating conflict indicators from this point forward. This conflict analysis will provide the baseline from which to monitor and later evaluate changes in the context. The depth and scope of the conflict analysis should be appropriate to the existing or anticipated intervention and your organisation's capacity. If on the other hand the conflict analysis is outdated, there is no need to redo it — simply develop (if none exist) or use the conflict indicators from the initial analysis to monitor changes in the operating context.

If you are conducting an evaluation, then given the importance of a conflict analysis to create a baseline, a retroactive conflict analysis should be undertaken using past reports and other information sources to estimate the situation prior to the start of the intervention.

(b) Understanding the project implementation

As conflict sensitive monitoring and evaluation focuses on the interaction between the context and the intervention, it is important to understand the project's intended and actual implementation.

Intended implementation, activities and approaches

• purpose and scope of the activity
• geographic location of the project
• project beneficiaries and partners
• timeframe
• funding level and sources.

The information outlined above can generally be found in the project proposal and approved implementation plans.

Actual implementation, activities and approaches

• who are the project partners and beneficiaries? And why?
• what have been successes and challenges?
• were any activities undertaken that had not been envisaged during the planning? Why?
• were any adjustments made from the initial strategy? Why?
• have any activities been changed or cancelled?
• were there problems with staff (eg security, motivation)?

This information is typically found through the monitoring of traditional project indicators that were designed in the planning stage. You may want to ensure that the questions above can be answered through your initial project monitoring indicators, and add or adjust indicators as necessary.

When gathering this information for an evaluation, reference can be made to previous monitoring reports. It is important, however, to gather other perspectives that may not be reflected in these reports: designed as they usually are for a specific audience they may not fully capture the project's implementation realities (see triangulation below, Chapter 2 Box 10, and Module 1 section 3.2 of this chapter).

(c) Understanding the interaction between the context and the project

As described in Module 1 of this chapter, there are three elements to conflict sensitive indicators:

• conflict indicators are used to monitor the progression of conflict factors against an appropriate baseline, and to provide targets against which to set contingency planning (see Chapter 2).
• project indicators monitor the efficiency, effectiveness, impact and sustainability of the project (see Module 1 and Annex 1 of this chapter).
• interaction indicators (see Module 1 of this chapter) are created at the planning phase of the project in order to measure the interaction between the context and the project.

Specifically, interaction indicators are used to monitor the impact of the project on the context, and of the context on the project. For example, if the context tells you that corruption amongst local government officials is a contributing conflict cause, and the project involves building the capacity of local government officials, then an interaction indicator will measure both:

• the project's effect on corruption amongst local government officials
• the effect of corruption amongst local government officials on the project.

A key challenge practitioners face when undertaking conflict sensitive monitoring and evaluation is the issue of agency or causality. For example, an NGO may be working in a remote village to provide access to water resources in a way that is equitable between two ethnic groups — an issue identified as key in a conflict analysis. Following the successful implementation of the project, evaluators using interaction indicators find that inter-marriage rates between the two ethnic groups have increased. The challenge of conflict-sensitive evaluation resides in the attribution of this change: is increased inter-marriage a result of the project intervention? Of interventions by other actors operating at the same and other levels? Or of changes in the context that are unrelated to external actors?

The highly simplified schematic that follows demonstrates the difficulty of determining the causal link between — in this example — the project intervention and a change in inter-ethnic marriage rates. Most contexts are substantially more complex than outlined in this diagram.

Conflict-sensitive monitoring and evaluation must recognise that there is not always a direct cause-and-effect relationship between the context and the project. In this sense, "good enough" thinking is required as conflict-sensitive monitoring and evaluation can never provide absolute certainty. It is nevertheless important to anticipate the challenge posed by causality when developing conflict-sensitive indicators. Good indicators often seek not to address directly the interaction between the project and the context, but to focus instead on more indirect causal manifestations of this interaction (eg not "did my project contribute to reduce discrimination?" but "are there parts of the district that are safe for some groups and not for others?").

Because every context is unique and can change dramatically over short periods of time, it is not possible to provide a definitive list of conflict-sensitive indicators that practitioners can use or adopt to their own situations. In addition to the guidelines outlined above, it is, however, possible to outline a general approach to developing indicators for conflict-sensitive monitoring and evaluation. Annex 1 uses a fictionalised context to provide a detailed breakdown of the type of analysis and indirect questioning that is useful for developing conflict sensitive indicators. Box 1 below provides some actual examples of using indirect indicators to help determine impact.

The three dimensions of conflict sensitive monitoring and evaluation outlined above — understanding the changing context, understanding the project implementation, understanding the interaction between the context and the project — provide a means of designing a conflict-sensitive monitoring and evaluation process. They may also inform the identification of required skills within the monitoring or evaluation team, which are likely to include:

• conflict analysis skills
• good knowledge of the context and related history
• sensitivity to the local context
• local language skills
• monitoring and evaluation expertise (including interviewing skills).

Finally, it is important to keep in mind that there is currently no clear way of assigning attribution for the consolidation of peace to any one particular actor. Given the complexity of most contexts, intervening actors will at best be able to demonstrate that their positive interventions coincided with positive changes in the context. Project and programme goals and objectives for building peace will need to be humble and realistic.

2.3 Step 3: Collect information

Collecting information is fundamental to the process of monitoring and evaluation. Conflict-sensitive information will need to include a combination of perception-based and objective data.

Perception-based information

As explained above, conflict-sensitive monitoring and evaluation cannot assume a direct causal relationship between the context and the project. In order to increase confidence in information collected, the perceptions of respondents can provide additional perspectives on causal relationships. Perception-based information can be derived from the following sources:

• executors of the activity: eg project staff, partners and implementing agencies
• beneficiaries of the activity: eg recipients of project outcomes (services, goods, training)
• observers of the activity: eg other organizations operating inside and outside the area, experts, academics, national and local leaders. Although this may not be an obvious category of respondents, their indirect involvement in the project and / or presence in the context may help ensure a more balanced understanding of the interaction.

The strength of perception-based information primarily depends on an honest and impartial composition of the list of respondents. If it is not possible to find unbiased respondents, it may help to get a balance of biases from among all interviewees. Evaluators also face a unique perception-related issue, as former project beneficiaries may use an end of project evaluation as an opportunity to deliver positive and uncritical feedback on the interaction between the project and the context, in the hope of securing future assistance or employment.

Objective information

Just as perception-based information helps address the issue of causality, objective information can be used to provide additional perspectives. Where perception-based information relies on views, beliefs and feelings of respondents, objective information seeks to provide less controversial or more "factual" data. Sources for objective data are entirely context specific — eg news media may sometimes be a good source of objective information, but in a different context or at a different time information reported may be entirely perception-based.

The principal reason for combining objective and perception-based information in the process of conflict sensitive monitoring and evaluation is triangulation. In other words, information received from one source is compared and contrasted to similar information received from another, in an effort to gain a more comprehensive understanding of the interaction between the intervention and the context. From a conflict-sensitive perspective, perceptions sometimes provide more information than "facts" or the "truth".

It is important to triangulate data within one information source, just as it is important to triangulate information sources. For example, within one community interviewers should talk to a representative cross-section of the population, from government officials to unemployed youth, as well as individuals (although perhaps not leaders) from major social and occupational groups. As mentioned above, project staff and observers not directly related to the project also provide a means of triangulating perception-based information from the field.

However, the perspectives of people involved in the community provide only one source of information (albeit diversified within the source), so it is also important to triangulate sources, for example, by reviewing secondary materials such as foreign government-sponsored country reports through a desk study, as well as soliciting the views of specialists. Focus groups, stakeholder and feedback workshops, and quantitative surveys provide other means of triangulating information sources.

Conflict-sensitive monitoring and evaluation requires that organisations acknowledge the potential impact of the monitoring or evaluation process itself on the conflict dynamics. Gathering information for monitoring and evaluation may have negative outcomes, such as putting community members at risk by raising suspicion or asking sensitive questions. Questions that are acceptable in one context may endanger interviewers and respondents in another. Interviewers may inadvertently upset respondents with probing, insensitive questions. Dialogue must always be based on mutual consent and respect, and the understanding that the consequences of the interview may last well beyond the discussion. Measures must be taken to ensure the safety at all times of both interviewers and respondents.

In situations of violent conflict, monitoring or evaluating projects often becomes quite difficult. While it is often possible to implement projects in such environments through local partners and community-based organisations or other means, it is sometimes not feasible to send external staff or consultants into the area to monitor or evaluate activities. Organisations tend to fear that "outsiders" may be at risk in violent environments, and that locals may be endangered by talking to these "outsiders". The struggle to monitor or evaluate their projects effectively under circumstances of violent conflict sometimes leads organisations to rely on telephone conversations and photographic evidence. Conflict-sensitive monitoring and evaluation must find ways of safely interacting with respondents in these types of environments; unfortunately this challenge remains unresolved.

2.4 Step 4: Analyse information

The analysis of the information gathered for conflict-sensitive monitoring and evaluation will require some attempt at discerning causal linkages, despite the difficulties.

Although there is no established framework for analyzing conflict sensitive information, two stages may help in the analytical process:

• find the most effective way to structure the information, in order to reduce the complexity of the data and, more importantly, to understand key linkages between the project and the context. For example, the grids or tools which most conflict analysis frameworks use to simplify the analytical process are sometimes also appropriate for conflict-sensitive monitoring and evaluation.
• further prioritise and deepen the linkages identified through triangulation. For example, in Central Azerbaijan, interviews with project beneficiaries in one village revealed that they were upset about the unequal distribution of benefits between them and the neighbouring village. Discussion with the implementation organisation showed that the benefits provided to both villages were identical. Rather than discount the disgruntled village perspective as incorrect, it is better to understand the community's perceptions about unequal benefit distribution, particularly in the light of contradicting objective information, as revealing an important issue for further investigation, monitoring, and possibly action.

2.5 Step 5: Recommend and redesign

2.5.1 Report

There is no need to write a specific conflict-sensitive report on monitoring or evaluation activities. It is however important to integrate the findings and recommendations of the analysis of the interaction between the context and the project into regular reporting (eg quarterly, annual, mid-term and final reports). It will be particularly helpful to outline explicitly the impact of the intervention on its context (ie the peace-building or conflict impact) and of the context on the intervention. This will provide a documented history of organisational learning on conflict-sensitive practice.

Consideration of the type and sensitivity of information to be included in reports should be determined by reference to the conflict analysis. In all cases, the sensitive handling of privacy and anonymity should be explicitly agreed upon not only for the monitoring and evaluation process but also at the reporting stage.

2.5.2 Feedback

Organisations need to take responsibility for the results of conflict-sensitive monitoring and evaluation, and for transforming those results into improved practice. Gaining organisational commitment to make these changes may however require a focussed feedback strategy to ensure that recommendations are implemented (see Chapter 5).

Recommendations from conflict-sensitive monitoring and evaluation may inform decisions regarding the (re-)design or further adjustment of project activities and their implementation, in light of the interaction between the context and the project. Module 2 on implementation provides guidance on how to take this process further.

3. Key issues in conflict-sensitive monitoring and evaluation

The following key issues should be kept in mind in the process of conflict sensitising a monitoring or evaluation process:

a) monitoring and evaluation are typically extractive processes, as interviewers take information from respondents and offer little in direct return. Conflict-sensitive monitoring and evaluation can also be an extractive process, or it can be more transformative. By involving respondents in the process of indicator development and analysis, monitors and evaluators can help people understand their own place in — and possibly even their contribution to — a given context. Transformative processes can potentially produce positive results; however, they involve risks similar to those outlined in step 2.3 (collect information) above, but with potentially more serious consequences

b) as with everything in this Resource Pack, the emphasis is on conflict-sensitising existing programmatic processes, rather than developing entirely new ones. For monitoring and evaluation, this means conflict-sensitising all existing steps in the process, from the design to reporting and beyond. The process of conflict-sensitising monitoring and evaluation will require additional resources. For instance, organisational and institutional support for increased staff capacity development will be needed (see Chapter 5). Sufficient time to review and adjust existing tools and processes, as well as additional time to monitor or evaluate conflict and interaction indicators will also be essential

c) there is sometimes a tendency in monitoring and evaluating to underestimate the importance of the profile, actors, causes and dynamics that function at other levels. A village-focused intervention may, for example, not consider the implication of national actors (eg political parties) or international dynamics (eg the foreign policies of other governments) on the local context. Alternatively, some monitors and evaluators will focus almost entirely on the macro context, and in particular on the macro political context, by emphasizing the activities and statements of warring factions, while ignoring the contribution made to conflict dynamics at the local level. Understanding the context as it is expressed at various different geographic scales is fundamental to understanding the context at the level the intervention is taking place

d) conflict-sensitive recommendations may prove challenging for staff within organisations, as well as within the institutional funding chain, as they require a different understanding of success. Organisations (and, if relevant, their funders) typically measure activities and outputs, such as number of houses built, number of wells dug, number of participants attending a meeting, rather than impact. A conflict-sensitive organisation will also want to place a high value on its projects' interactions with the context. Thus, a project that underperforms on the anticipated number of houses built may, from a conflict sensitive perspective, still be considered a success if it contributed positively to conflict dynamics. Given that the definition of a successful project can be controversial, organisations may have difficulty in valuing an under-performing conflict-sensitive project over a well-performing project that unintentionally exacerbates conflict (see Box 3). Enhancing the way an organization understands success requires an institutional willingness and ability to think differently about how it measures impact. (See Chapter 5).

4. Endnotes

1Adapted from Cathy McIlwaine and Caroline Moser, Urban Poor Perceptions of Violence and Exclusion in Colombia , Washington DC: World Bank, 2000: 65.

Annex 1 Sample indicators — links between context changes and project, and project changes and context

The following table is provided for the purposes of better understanding what situation-specific interaction indicators might actually look like. The table uses a fictional setting to outline the types of changes that might indicate an interaction between the context and the project, and associated indicators that could be used to better understand these changes and thus the interaction. Note that the sample interaction indicators provided use a combination of objective and perceptive questioning to help triangulate information collection; objective and perception based indicators are discussed in step 3 above.

Background

Kugan is a poor developing country. The national government is undertaking a road construction project through the northern region to create a trade link with the neighbouring country of Moyag. The road right-of-way has been cleared and levelled, but asphalting has not yet begun. Northern Kugan is a sparsely populated region dominated by pastoralists and cattle herders. There is violent conflict in the adjacent region of Moyag, and arms proliferation in Northern Kugan has become a problem. Another problem is the involvement of Kugan locals in rustling cattle for Moyag-based gangs.

Project's impact on context — are changes in the context linked to the project?	Context's impact on project — are changes in the project linked to the context?
Change in profile: Environmental degradation, and in particular, deforestation. Project-related question or suspicion: Is road construction contributing to deforestation? Indicators: 1: respondents who feel the natural habitat has improved/deteriorated 2: changes in the price of cut wood 3: percentage of road through forested areas.	Change in project: The road is now being constructed in a straight line and thus at a lower cost. Profile-related question or suspicion: Is there government pressure on pastoralists to surrender land so road can be constructed in a straight line? Indicators: 1: pastoralists' perception about the benefit of the road 2: changes in real construction expenditures compared to project budget 3: pastoralists' feelings about the road being a government project compared to other respondents (eg cattle herders).
Change in causes : Increase in small arms proliferation. Project-related question or suspicion: Is road construction facilitating the trafficking of small arms? Indicators 1: number of respondents who feel there has been an increase in small arms proliferation (since road construction began) 2: change in incidences of gun related violence along road 3: number of respondents who feel it is easier/ harder to purchase a gun compared to off-road respondents.	Change in project: Payroll offices being robbed. Causes-related question or suspicion: Are bandits using increasingly available small arms to rob construction payroll offices? Indicators: 1: incidences of robberies and amount stolen 2: number of robberies that involve small arms 3: respondents who perceive the road building project is not a proper prioritisation of community needs.
Change in actors: Access to education for rural youth. Project-related question or suspicion: Is road under construction already increasing transportation options for rural youth? Indicators: 1: number of youth attending schools accessed by road 2: number of days average student attends one of these schools 3: number of respondents who feel the road has increased access to schooling.	Change in project: Fuel being stolen from construction vehicles. Actors-related question or suspicion: Are poor cattle herders seeking to derive benefits from road project by stealing fuel from construction vehicles? Indicators: 1: litres of fuel stolen 2: cattle herders' feelings about expenditure on road 3: change in sales by cattle herders' traditional fuel sources.
Change in dynamics: Decreased incidences of cattle rustling. Project-related question or suspicion: Is increased access to employment and income undermining the need to rustle cattle? Indicators: 1: change in level of household income on road compared to off road incomes 2: percentage change in households that feel they have better livelihood options 3: change in incidences and number of cattle stolen.	Change in project: Labour for project has become hard to find. Dynamics-related question or suspicion: Are potential construction workers not seeking employment on the road project because of their concerns about increased insecurity? Indicators: 1: number of vacancies unfilled in road construction jobs 2: percentage of construction workers who "feel safe" working in the area 3: number of construction workers who leave the construction camps at night.