2.2 Impact Risk

The starting point for any structured and rational treatment of impact is being explicit. This involves ensuring that the impact plan displays:

• clarity

  The impact plan articulates clearly each of its components and the linkages between them. This includes setting out what will be done, what processes will be used, and how the activities — within the defined context, and in combination with other conditions — will bring about the desired change.

• concreteness

  The impact plan is specific and concrete about what is to be used (resources, budget), who will be effected (target beneficiaries and their context), what is to be achieved (how much, how many), and the timelines involved (when will the activities be carried out, and the change happen). The impact plan is concrete also regarding the measurement system that will be used to track what is taking place.

• completeness

  The impact plan gives a fair, true and complete picture of the processes and changes it presents, including implicit claims and assumptions, and appropriate consideration of how the change relates to other factors and the surrounding environment (including impacts upon other stakeholders). These are covered in the conditions for change and context of change sections of the impact plan. An impact plan that covers only the organisation’s own processes, with no address of the context, is deemed to be incomplete.

  A full address of the context, and all the ramifications of change (including deadweight, displacement, attribution, drop off, and unintended consequences), is likely to be beyond the scope of most impact plans, and the organisation must therefore make an assessment of materiality — i.e. a determination of the bounds of what is relevant and material to include in a true account of the impact. The impact plan is explicit as to where these bounds of materiality lie. The information that is deemed material is therefore provided, and gaps or holes in the information, or links that are unproven, are acknowledged and justified.

Once the impact plan and its various components have been laid out explicitly, attention turns to how well reasoned an overall narrative or theory of change it presents. Pertinent questions include:

• Do the mission and activities express a coherent response to the context (i.e. the problem and the target beneficiaries)?
• Is the link between the proposed outputs and the anticipated outcomes thought-through and convincing? Do the outputs really drive the outcomes? Have the conditions for change been addressed, and their role in the change soundly reasoned?
• Is the address of the context of change credible and fair, with the bounds of materiality set at a sensible level?

A full address of the context of change can most likely only be achieved through conducting a control experiment (typically a randomised control trial, or RCT). However this is often impractical given the resources and the scale of operations. Under such circumstances, investors and organisations are often reliant upon a reasoned treatment of the counterfactual (a hypothetical scenario of “what would have happened anyway, what is happening elsewhere, and the role of other factors” that can be used to deal with questions of deadweight, displacement, and attribution).

There may be uncertainties, and therefore impact risk, around how the outcomes are really brought about, and how reliably they are a result of the organisation’s work. Most important to the impact is that the organisation can make a compelling case for how it plays a critical role in the desired change (i.e. without it the change wouldn’t have happened). Backwards-mapping can be a powerful tool for testing the reasoning involved throughout the impact plan.

A form of impact risk may arise if there is a potential tension within the organisation between its impact-generating and revenue-generating activities. Where there is a clear financial motive for the organisation to pursue less impactful strategies, and the business and impact interests are in this sense not well-aligned, there is a risk that the operational needs of the business will threaten the impact.

This risk however is greatly reduced if the impact plan is integral to the organisation’s business strategy, operations, and revenue model. In this case, the business plan clearly supports the impact plan, with impact and operational sustainability going hand in hand.

Where there is tension and potential risk regarding the integration of impact into the business model, the investor may look to some form of mission lock or protection via the governance or legal structure of the organisation (e.g. governance obligations, incorporation as a registered charity or CIC).

The question of feasibility focuses mainly on the links in the impact plan between the organisation, its activities and its outputs. For the impact plan to be feasible, it must show:

• the organisation has the resources, capacity, skills and relevant experience to execute the plan
• the operational risks inherent in the plan are identified and addressed, with measures in place to mitigate them where appropriate

A significant aspect of the overall feasibility of the plan will relate to the financial and operational strength of the organisation. This however will generally fall within financial due diligence considerations, and typically go into a credit rating, and be given separate consideration. The question of feasibility, for impact risk therefore, focuses on those aspects not covered in the financial analysis — i.e. assuming credit-related issues are secure, is the impact plan feasible in other respects?

This may include attention to:

• key personnel

  Does the organisation have the right people to carry out the plan with respect to impact, with the necessary skills and relevant experience, as well as the vision, leadership and drive?

• operational processes

  Does the organisation have processes in place to manage activities, and ensure they are reaching the right beneficiaries, and having the desired effect? Are the activities an effective means to deliver the desired outputs?

• capacity

  Does the organisation have the staff, time, technology and facilities required to carry out activities?

• projections around other factors

  Where the impact is reliant upon factors beyond the organisation’s direct control (e.g. conditions in the local economy, support or services to be delivered by other organisations, among the conditions for change), and assumptions are therefore made about them, are these assumptions feasible?

Evidence may include:

• track record

  The organisation has carried out similar activities in the past, with robust impact measurement of past performance demonstrating the validity and effectiveness of the approach. For evaluating the track record, see quality of information and verification of results (in 4.2 Impact Reporting). To be considered as convincing evidence, a track record must demonstrate a change in the measured outcome (typically involving pre- and post-intervention measurements), and that, where used, samples are representative, and survey questions are neutral and non-leading. An independent evaluation of the activities and outputs of the organisation, where available, provides the best evidence on this front (and thereby lowest impact risk).

• precedents

  The track records of other organisations, working with similar methods and assumptions, and again appropriately evidenced by measurement, may be used to demonstrate the validity of the approach.

• research

  Studies or relevant expert knowledge may be used to back up the claims involved. Research can situate the organisation’s approach in the context of the problem and other relevant interventions, which it may align with or differ from according to the position taken. Research may in particular be used to support the assumptions implicit in the conditions for change, and the treatment of the counterfactual in the context of change. Where available, research on benchmarks can provide an anchor for the organisation’s past results and proposed future performance.

• control groups

  The most conclusive evidence of the effectiveness of an intervention is to demonstrate through the use of a control group the difference between the outcomes achieved when the organisation is active, and when it is not. This, properly speaking, is the demonstrable impact: the real change brought about as a clear result of the organisation’s work. However, while randomised control trials (RCTs) represent the gold standard in evidence, they are expensive to carry out, and require specialised skills. It is also important to note that RCTs are significantly more practicable, and therefore favour, interventions of a very specific nature, with easily isolated, testable, and relatively short-term outcomes. Furthermore, RCTs are meaningful only when the sample sizes are large enough for other factors to cancel each other out, and therefore are often applicable only when the intervention is taking place at a relatively large scale. While all this means that it is unlikely there will be a widespread adoption of RCTs throughout the social-purpose sector anytime soon (and especially not at the early-stage end of the spectrum), the lesson is nevertheless a powerful one: that for an intervention to be truly valid, it must be able to outperform a control group. If a specific control group is not set up and monitored, then some evidence as to what such a control group might look like, typically based on research with comparable situations elsewhere, can serve to lower impact risk significantly on this front.

The availability of a track record, precedents, extensive research, and control groups, will depend on a combination of the organisation’s stage of development, and the originality of its approach. Rarely will an organisation be able to provide an exhaustively evidenced treatment of the change, and its interplay with other factors, though it is important to look at what evidence there is, and to consider the impact risk it leaves. Evidence, in so far as it is available, should serve to promote confidence in the impact plan, and in particular in the relationship between the organisation’s proposed activities and outputs, and the outcomes and impact that it is hoped will follow.

For an organisation proposing a completely new idea, and therefore with little or no direct evidence of how well it works, there may still be relevant research it is responding to, and that has informed the development of the approach (i.e. less proving the approach than showing how different approaches have failed in the past, and how this one learns from them). However an organisation working with well-established methods will inevitably have more to draw upon regarding evidence.

As a result, excessive investor demand for high levels of evidence would lead to an inevitable bias toward mature organisations working with tried and tested methods, at the expense of investing in innovative, and in some cases possibly more effective, forms of intervention. The balance between conflicting desires for the impact plans to be, on the one hand evidenced, and on the other, to deliver something new, will depend upon an investor’s mission, strategy and appetite for impact risk. A less well-evidenced, and therefore riskier, approach may ultimately prove to be game-changing, and thereby high impact. These considerations will play into the investment decision when weighing impact risk against other criteria.

Where there is less evidence available, it becomes increasingly important, with regard to impact risk, for the impact plan to be convincingly reasoned (see 2.2.2 above), and evidenceable (see 2.2.6 below).

An evidenceable impact plan is one that incorporates processes to ensure that carrying out the plan will produce sufficient evidence to demonstrate the outcomes and impact, and prove the approach. This requires that:

• a robust impact measurement system is in place to track outputs and outcomes
• where a link, relationship, assumption or claim is unproven, it is identified, and checks are in place to validate it in the future
• measures will be taken to assess the other factors involved and the true role of the organisation’s outputs in the change (i.e. there is an anticipated address of the conditions for change and context of change — e.g. a reference is identified, or a control group set up, to establish a sense of what happens without the intervention, and to provide a degree of evidence in support of the hypothetical scenario of what would have happened anyway, what is happening elsewhere, and the role of other factors)
• the anticipated evidence is inclusive of the beneficiary perspective (evidence features feedback from beneficiaries, and is communicated to beneficiaries)

The impact plans of potential investee organisations are likely to present theories, links and impacts that are under-evidenced, and in some cases altogether untested. However these may still be testable, and the subject of planned tests. For the confidence of the investor to be gained, it is crucial that the organisation can show effective measures are in place to evidence its impact going into the future, especially when there is a lack of evidence currently.

The impact plan must be clear as to which parts are evidenced, which are unevidenced but will be evidenced by the activities and measurement system proposed, and which will remain essentially reasoned. The timeline for the evidence is also important: if an impact plan is full of unproven elements, the investor will want to know, if the investment is made, what evidence there will be to show whether or not the plan is working by year one, three, five etc..

As the organisation carries out its plan, over the course of operations, and the period of the investment, it is expected that more and more elements will become evidenced. Also, as the organisation matures and scales, its measurement system may be expected to grow in scope proportionally, thus expanding the range of evidenceable and subsequently evidenced aspects of the plan. This will correspond naturally with diminishing impact risk, as operations successfully manifest the impact.

Alternatively, if the approach is failing, the presence of evidence systems will be able to show this, giving the organisation and the investor the opportunity to change course.