About the Context of Change
The change observed by the organisation takes place within a context that is itself likely to be changing, and playing a part in any change. In order to understand the real change that has come about as a consequence of the organisation’s work, it is important to understand the context of change around it.
The first and most fundamental issue is one of causation. This asks if the organisation’s activities and outputs are really driving the change, or if they are just taking place alongside it. If the organisation were to stop operating, for example, would the same change, and associated benefits, occur anyway? And while it is operating, what other changes are taking place in the context that need to be brought into the account?
Being able to answer these questions credibly allows the organisation not only to communicate its impact better, but also to develop its activities through being able to identify what is important and effective, and what is not.
What do the issues of deadweight, displacement, attribution, drop off and unintended consequences indicate about the observed change?
The impact sector has identified five major points regarding how a changing context may affect the understanding of the observed outcomes:
deadweight
Deadweight considers what would have happened anyway — i.e. what outcomes beneficiaries would be expected to experience in the absence of the organisation’s activities. (This relates to “the counterfactual” or “the baseline”, and may include positive as well as negative outcomes.)
A treatment of deadweight covers:
- beneficiary progress without intervention
Without intervention, beneficiaries may nevertheless make progress in relation to the desired outcomes. For example, with an employment training programme, the question would be, what proportion of the trainees would have found work on their own initiative; with an alcohol abuse programme, how many would have lowered their alcohol intake independently, etc. (If the success rate with the intervention is not significantly higher than that without, the real impact is clearly low.)
- negative consequences of no intervention
Conversely, without intervention, beneficiaries may suffer from deteriorating conditions, and fare significantly worse. For example, without particular forms of support, a proportion of beneficiaries may face more severe health problems, involvement with crime or substance abuse, etc. Here the impact is enhanced, as the real change is that over and above the averted negative scenario. (Under such circumstances, no change may be regarded as a significant positive impact.)
- other service providers
In the absence of the organisation, beneficiaries may typically access services from other providers — either ones currently existing, or those who would otherwise step in. These may include: the default government response; the default commercial or mainstream response; other social-purpose organisations. The outcomes beneficiaries achieve through use of these services are again compared with those being achieved with the organisation’s support to check for real positive change.
displacement
Displacement occurs when the positive outcomes experienced by the beneficiaries of a service are offset by negative outcomes experienced by another group elsewhere. For example, a new business that aims to create jobs in a deprived area may bring about the closure of another business nearby, with a consequent loss of jobs, thereby in effect displacing jobs from one part of the area to another; an initiative to reduce crime may in fact be displacing crime from one area to another, or from one form to another, and so on.
attribution
Attribution considers how much of the change that has been observed is the result of the organisation’s activities and outputs, and how much is the result of actions taken simultaneously by others (e.g. other organisations, government). This relates back to the other factors identified among the conditions for change, and the risk of the organisation’s intervention being non-significant. An assessment of attribution weighs these other factors for their contribution (how much of the change are they responsible for?) and for their criticality (would the change have been possible without them?).
It is important also to consider the measurement system for attribution, and whether there are omitted variables that are having an effect on the outcome indicators. The question is, to what extent can shifts in the measured results be attributed to the desired change, and are there other possible forces influencing the measurements? If there are, and they are measurable, they should be factored into the account.
drop off
Drop off occurs when, over time, the effect of the output and the observed outcome decreases. There may be drop off from the use of the service or product (this should be recorded within the primary set of results, and properly is an issue for the data collection processes of the measurement system), and similarly there may be drop off of the observed outcomes, as beneficiaries potentially relapse, lose the job or accommodation they attained, revert to previous behaviours, etc. The organisation’s mission, and its definition of intended outcomes, set the scope for how long the outcomes may be expected to last. Drop off occurring within this period should be acknowledged. The organisation should further be aware of which beneficiaries are dropping off, and whether there are common factors among them (if so, they may suggest improvements or additions to the services).
unintended consequences
In addition to the outcomes defined and measured by the organisation, there may be unintended consequences, which can be negative or positive. Some may be foreseen (for example, an intervention may impact on the environment or local community in ways that are not exactly “intended” but are a clear result of activities), and should be included in the prospective impact plan. Some may only become apparent once the plan is being carried out (for example, beneficiaries responding in an unexpected way, with further implications and outcomes), which should be picked up during the monitoring and evaluation stage, and incorporated into the impact plan. Of particular relevance may be those identified as the other stakeholders in the assessment of the context, and the question of how they are being affected by operations. Projects with a strong mission tend to encourage tunnel-thinking, and it is important to review the organisation’s activities periodically for unintended consequences, and what these may mean for the overall impact.
Has an appropriate address of these issues been made?
Analysis of the context of change indicates adjustments to the measured outcomes that need to be made in order to arrive at an assessment of the actual impact. Accurate analysis serves to reduce impact risk, and provides useful information as to how the organisation can enhance efficiency and maximise impact. Poor analysis, or an outright lack of it, can lead to organisations misattributing, and over- or under-valuing their impact. This presents investors with the risk of investing in impacts that are measured, but aren’t real.
However, a complete and accurate treatment of the context of change is difficult. Many factors, many of which are not easy to measure, and some of which the organisation may not be aware of, are likely to be involved. And, problematically, the counterfactual “what would have happened …” presents a case that specifically has not happened, and therefore cannot be measured directly.
The best and most scientific treatment of these issues is often to be arrived at through close observation of a control group, with the gold standard in most cases being a Randomised Control Trial (RCT). However conducting an RCT can be expensive and require manpower, expertise and time. It is also worth noting that RCTs are significantly easier to conduct for certain kinds of interventions than others (those where the intervention is relatively narrow and specific, the outcome easy to isolate, the timeframe reasonably short, and the scale large enough to support meaningful sample sizes, are much more amenable to RCTs1). An excessive demand on the part of investors for RCTs could restrict the flow of capital to “RCT-friendly” approaches and sectors, even though these are not necessarily the most impactful. Furthermore, even with experimental assessments, there is still potential for bias and inconsistency.
In the absence of a proper control, a less scientific analysis of the context of change involves working through the above five points, including making an estimate of the deadweight (with at least a research-based assessment of the typical beneficiary progress without intervention), and an address of the other four points, with additional estimates where circumstances suggest one or another could indicate significant adjustments to the impact calculation.
A more complete address involves working through the technical points of data analysis attendant upon the distinction between causation and correlation, and bringing evidence to bear wherever possible. These are:
reverse causality
Reverse causality occurs if there is a positive relationship between output and outcome indicators, but we cannot be sure which way causality runs. For example, an organisation providing support to microbusinesses conducts an impact assessment of its services and finds that access to finance is associated with better business outcomes. However, they cannot be sure whether access to finance causes better business outcomes, or whether being a more successful business causes an increase in access to finance.
omitted variable bias
Omitted variable bias occurs if an impact assessment results in a positive estimate of the relationship between outputs and outcomes, but in reality there are either observable or unobservable (or non-quantifiable) variables that affect the relationship. To continue with the previous example, suppose the organisation offers business support and mentoring services alongside loans to micro businesses. The organisation does not know whether the access to the loan caused the improvement in business success or whether it was some combination of these services. This would cause an overestimation of the effect of loans (alone) on business success.
selection bias
Selection bias occurs when the sample used to evaluate the impact is not random. This can happen in two ways. Continuing with the same example, sampling bias occurs if the organisation attracts micro-businesses that are run by individuals that are more motivated or have higher educational attainment. It is likely that even without accessing the organisation’s services, this sample of micro-businesses would perform better than the general population of micro-businesses. This causes an overestimation of the impact of the organisation’s services. Attrition includes changes in the sample due to dropout, non- or lower- response rates, withdrawal and protocol deviators. If those that are lost are less successful than those that remain, the impact of the organisation’s services will be overestimated as well.
measurement error
Measurement error occurs when the data used is not accurate itself. All data measurement is likely to have some form of measurement error. Random error will bias the estimation towards zero. Non-random, or systematic measurement error, can result in over- or under-estimated effects. For example, the organisation may be able to collect better and more accurate data only for more successful businesses, because they may have better systems to provide the data. This will cause an overestimate of the organisation’s impact.
1 For example, a drug trial, conducted on a statistically large group, where either the drug or a placebo is given, and after three months a blood test for antibody levels that clearly indicates the drug’s effect can be performed, provides an excellent case for an RCT. Conversely, with an after school football programme, in which twenty kids in North London are participating, and where the active factor may be the football (the activity), or a particular inspirational coach, and the key outcomes include a better future outlook and a reduced likelihood of being drawn into crime over the medium-term (e.g. the next 3-5 years), then trying to conduct an RCT is unlikely to be useful. The sample size is far too small in comparison with the magnitude of the change, and the active ingredient in the intervention is unclear. In this case, a much lighter research-based estimate of the deadweight, and an acknowledgement of other factors in the discussion of attribution, is more appropriate (alongside monitoring of the actual beneficiary group). ↑
