10. Solution testing

Testing solutions against criteria and for robustness/sensitivity. The aim of the criteria evaluation is to ensure that the final solution corresponds to the future vision. The robustness analysis adds to this by aiding the identification of robust strategies, but also by shedding light on vulnerabilities and potential trade-offs.

At a glance



Goal

Selection of a solution for implementation.



Input

Solutions + criteria + drivers



Methods

Criteria testing; robustness/sensitivity testing; sustainability testing; modelling



Relative workload

High



Output

Internal scenarios/solutions evaluated against the criteria.
Internal scenarios/solutions evaluated for robustness using a futures plane.
A final, combined scenario.

Steps

1. Evaluate each solution (internal scenario) against all defined criteria, identifying which solution or combination of solutions meets the criteria in the best way.

2. Assess the robustness/sensitivity of each solution by evaluating it against each of the four external futures created in the Drivers module.

3. Drawing upon the outcome of the criteria and robustness/sensitivity testing, create a final combined scenario.

Criteria evaluation

Each solution (internal scenario) should be evaluated against all defined criteria. Identify which solution meets the criteria in the best way, or if it is possible to create a combination of several of the solutions.

There are a number of different methods for so called multi-criteria decision making (MCDM) and multi-criteria decision analysis (MCDA). A basic approach is to rate the performance of each solution against the set of criteria. The level of performance can be indicated using a scale (e.g. 1-5), where the lowest number corresponds to very poor performance and the highest to excellent performance. The arithmetic mean value for each solution suggests how well the solution in question meets the set of criteria. The rating could be based on discussion and consensus agreement. A more elaborate method is to define weighting coefficients for each criteria and calculate the weighted arithmetic mean.

Robustness

A strategy is robust if it is capable of coping well with variations (sometimes unpredictable variations) in its operating environment (in our case – the future conditions defined and illustrated by the previously developed external scenarios) with minimal damage, alteration or loss of functionality. The robustness testing helps to identify such robust solutions, characterise their vulnerabilities and evaluate trade-offs among them.

For the robustness testing, evaluate each scenario/internal solution against each of the four external futures created in the Drivers module (a starting point can be a discussion among the project team members about how well each solution would function in each future). Illustrate by drawing boundaries for a solution on the futures plane according to how robust the solution is in each future. A more robust solution (i.e. a solution well suited to cope with the variations of that future) should cover a larger part of the future.

Remember that you are not implementing the futures outlined in the futures plane, as the drivers (key uncertainties) that were used to create these futures are external to the system under study. At best, there could be the possibility to influence the futures. This is why it is important to design a solution/internal scenario that can function well in as many potential futures as possible.

As illustrated here, the characteristics of Solution A makes it a good fit for the “Poor disaster” and “Poor and safe” futures, but somewhat less appropriate for the “Rich disaster” and “Rich and safe” futures.

Solution B would be reasonably suitable for the “Rich and safe” future, but less so for the other futures. Also Solution C has characteristics that make it a good fit for the “Rich and safe” future, and it could do somewhat better than Solution B in the “Poor disaster” and “Poor and safe” futures. Additionally, the illustration suggests that there are aspects of Solution C that could be appropriate for the “Rich disaster” future. Now, could elements of Solution B and Solution C be incorporated into Solution A, to expand this solution’s overall suitability (making it more appropriate also for the “Rich disaster” and “Rich and safe” futures)?

Final scenario

Both the scenario testing against the criteria and the robustness testing reveal strengths and weaknesses of the proposed solutions. Based on the outcomes from the testings, decide on a final scenario (final solution) for implementation – often a combination of the internal scenarios.

Data collection

Criteria evaluation has been used in many different studies. By consulting other material such as scientific papers, governmental reports or project assessments, you may find suggestions for different types of criteria evaluation processes.

If you want to carry out a more comprehensive multi-criteria analysis, there are also many publications describing this method in more detail.

For more information about data collection, see participation and context.

Participatory elements

The criteria evaluation and robustness/sensitivity analysis should be as rigorous and participatory as possible.

Rating the level of performance of each solution against the set of criteria is not an objective process with a right and wrong answer. Rather, it is a delicate issue that requires careful and transparent discussions and input from various experts. That said, given the complexity of the issue, it might not be possible to directly involve (all of) the stakeholders in the explicit rating process. Instead, the project team can consult the stakeholders, aiming at aggregating the input and ensuring the various viewpoints are reflected in the rating process.

For more information about participatory elements, see participation and context.