Keeping Strategies on Track

Assessment combines monitoring, evaluating and reporting on the strategy. Assessment is primarily forward-looking; its purpose is to improve the strategy process, help it meet objectives and adapt it to changing needs.

Assessment should be an integral part of the strategy from the start and cover all aspects: objectives, participation, communication, role in the decision-making system, planning, implementation and results.

This chapter outlines an approach to assessing progress toward sustainable well-being. It is intended to be used by the people who advise, or in some way influence decision-makers involved in strategies.

As the national strategy process begins to take hold, it will need to be expressed in linked strategies at many levels: the household, farm, municipality, business, province or nation – anywhere that ‘stakeholder’ groups, or combinations of these, try to improve or maintain the well-being of people and ecosystems. The approach is meant to apply to assessment of all such strategies. Hence its essentials are simple and few. Details will vary from strategy to strategy, depending on the people and ecosystems involved. To make the main points clear, the chapter includes only a basic discussion.

The approach to assessing sustainability is described in five sections:

1. the purpose of assessment;

2. assessing the progress of society/ecosystem interactions;

3. assessing the progress of a particular strategy;

4. participation in assessment; and

5. making assessments useful.

The purpose of assessment

Assessments are essential for the success of any strategy, regardless of its scale or scope, or the education and income of its participants. Assessment is the process of judging progress toward the goal of sustainable development or well-being; asking and answering key questions about:

• human and ecosystem well-being, and their interactions and trends, so that the various strategy constituencies may progressively define, agree on and revise objectives and a strategy to achieve them; and

• the progress of the strategy itself, so that participants may improve its design and operation.

Assessment is best understood as a composite of various functions that are already well-known to strategy practitioners. In broad terms, these include the following processes and questions:

• Monitoring. What is happening?

• Evaluation. Is what was supposed to happen actually happening?

• Analysis. What should be happening now, and in the future?

The broader purpose of assessment is to evaluate and improve the progress society is making toward sustainable development or well-being. Its specific purpose is to enable people to:

• increase their understanding of human and ecosystem well-being and how to improve and maintain them;

• know what state they and their supporting ecosystems are in;

• determine where they and their supporting ecosystems are going;

• define where they want to be, and integrate trade-off objectives;

• chart a course for getting there; and

• change that course in response to changes in conditions, information, values and priorities.

Assessment is an effort to determine which potentials exist and which could be improved and how (not simply what is wrong). Since sustainable development is a dynamic process, and sustainable wellbeing a dynamic condition, any strategy for sustainability must also be dynamic. Regular assessment enables the strategy to both respond to, and influence, changing
conditions.

Who should do the assessment?

Two groups should undertake assessments: the stakeholders (people directly concerned) and independent outsiders. They do not have to do it together: ‘internal’ assessments by stakeholders are essential; ‘external’ assessments by others are desirable. The people directly concerned have most to gain from an assessment. They should be centrally involved; by participating in the assessment, they will know better what to do to achieve their objectives, and why. For a given set of decision-makers – at the level of the town, region or country – the emphasis placed on any particular topic, or the choice of specific measures, will vary depending on local conditions and priorities. Thus, it is essential that assessment of progress toward sustainability be driven by local participants.

At the same time, unbiased opinion and independent analysis can make a critical contribution to understanding. An external assessment can give stakeholders new insights, and avoid or overcome conflicts of interest involved in self-assessment.

When should assessments be done?

Assessment should be an integral part of decision-making. It should be a regular and integral activity rather than a sporadic and separate event and should, by and large, be done through normal operations, e.g. of management, to keep its potentially high costs within limits. Frequency of assessment will depend on how rapidly and significantly conditions are changing, and the magnitude of the risk to human or ecosystem wellbeing.

Assessment should be undertaken from the start, to create a baseline; and regularly thereafter as an integral part of any strategy. Assessment is implicit in the design and implementation of successful strategies. For example, an effective national strategy begins with the assessment of the strategy’s objectives and of the procedure for its design or formulation. Assessment continues throughout strategy formulation and implementation, covering both the relevance of the objectives and how they are being addressed: it also determines any revisions to the strategy.

The benefit of regular explicit assessment is that it encourages participants to rethink priorities, reset objectives, and rechart their course of action.

What should be assessed?

Assessment should provide and analyze two sets of information:

1. progress of society/ecosystem status and interactions toward sustainable well-being; and
2. progress of particular strategies toward their objectives and their contribution to the goal of sustainable well-being.

Assessing the progress of society/ecosystem status and interactions

The information that follows addresses both the broader social, economic and ecological context within which a strategy operates, and also some very specific criteria chosen to highlight the precise nature of people–ecosystem status and interactions. Four categories are suggested: ecosystems, people, interactions between people and ecosystems, and the synthesis of these. Each of the first three categories is portrayed as a hierarchy of information, ranging from specific measures at the bottom to complex systems at the top that build on and incorporate the lower levels (Figures 5, 6 and 7).

Ecosystems

The overall ecological goal is to maintain or improve ecosystem well-being. Assessment of progress toward this goal needs to consider the state of the ecosystem as a whole as well as selected resources, issues and criteria, such as air quality, water quality, soils, and plant diversity (Figure 5).

People

The goal is to improve or maintain human well-being. Assessment of progress toward this goal needs to consider the state of society as a whole as well as selected indicators, such as health, wealth, and happiness (Figure 6).

Interactions between people and ecosystems

The goal is for human activities to increase or maintain benefits or values from ecosystems while reducing stresses on them. Assessment of progress toward this goal needs to consider: how and to what extent human activities contribute to the provision of basic needs and the quality of life; how these activities are valued; how they stress or help to restore the ecosystem; and progress in meeting the goal through legislation, incentives, and other measures (Figure 7).

Synthesis

The goal is sustainable well-being. Analysis of the first three categories is likely to show that some aspects of the ecosystem, society and their interactions are getting better, others worse, and others are about the same. The most important aspects and the main links between them need to be identified to arrive at an overall picture of the state of human and ecosystem well-being. Two forms of synthesis may be required: a macro-level set of indicators akin to, for example, the UN Human Development
Index; and sample micro-level indicators at the sector, landscape, community or livelihood system level.

Assessing the progress of a particular strategy

A strategy is an evolutionary process, developing as it goes along and adapting to change. It is also cyclical, its main components – constituency-building, agenda-building, design, implementation and assessment – being repeated as it develops (Figure 8).

This means that a strategy need not and should not try to do everything at once. It can grow in scope, ambition and participation as objectives are achieved (or changed) and as capacities to undertake the strategy are built.

Assessment of a strategy needs to cover four main aspects:

1. Participants in the strategy; objectives of the strategy; and their relationship

Constituency-building and agenda-building should go together throughout the strategy. The participants decide the objectives, and the objectives determine the participants. Assessment should ask: Who are the stakeholders? What are their interests? Are interests being dealt with equitably? Who are the ‘winners’ and the ‘losers’? Are the interests of different groups compatible with the goal of sustainable development and well-being? If not, how can they be made compatible?

2. Communication among participants, and between participants and others

Communication is the lifeblood of a strategy; the means by which participants exchange information with each other, reach agreement with each other on actions, change or strengthen values and impart knowledge, and inform others about the strategy. It is necessary to assess the modes, frequency and effectiveness of communication, both among participants, and between the participants and others.

3. What actions are planned, decided on and taken, and by whom; and what are the obstacles?

Actions are likely to be taken if priorities are clear, the number of top priority actions is practicable, the actors are identified, the required resources are specified, and the resources are allocated or their probable sources identified. Assessment needs to ask:

• who participates/participated, and how do/did they participate, in (a) assessment, (b) designing the actions, (c) deciding the actions, and (d) taking the actions?

• what actions were (a) assessed as high priority, (b) designed, (c) decided, and (d) taken?

• what were the reasons for any discrepancy among actions assessed as high priority, planned actions, actions to be taken, and actions that were taken: ie what actions did not have majority agreement, or were considered difficult to implement?

• what were the obstacles to making priority actions effective and how could they be overcome?

4. Effectiveness in terms of the strategy’s objectives and the goal of sustainable wellbeing

This requires coordination between strategy monitoring and the society/ecosystem monitoring described above. Actions called for and taken as part of a strategy usually entail changing or strengthening one or more of:

• values (and habits);

• knowledge;

• technologies (and infrastructure);

• institutions (laws, incentive systems and organizations); and

• market conditions, eg price.

Any intended changes and improvements need to be identified clearly. Assessment will require an accurate description of the baseline situation (the people/ecosystem status assessment). To assess the impact of the actions on the strategy objectives, and to distinguish their impact from the effects of other factors, it is necessary to:

• clearly define the variables by which the strategy objectives are to be measured;

• monitor changes in these variables (through the people/ecosystem assessment);

• understand the relationship among the strategy objectives and values (and habits), knowledge, technologies (and infrastructure) and institutions – together with the relative importance of different factors (eg particular institutions); and

• determine the effect of the actions on values, knowledge, technologies and institutions.

Box 23 illustrates a range of questions used to monitor progress by a local strategies team in Pakistan.

Participation in assessment

The ‘how’ of assessment consists of two components:

1. how to use a participatory process to define the key questions; and
2. how to choose and use the right tools to help participants answer these questions.

People will often focus on the second component at the expense of the first, believing that the question of what is to be looked for it is already answered. Yet, repeatedly, attempts at assessment fail because those charged with the task do not ask themselves what questions need to be asked. They can establish this only by involving all the people who are affected by the issue. It is not possible to be prescriptive about the kinds of questions which strategy teams will need to ask.

Principles for participation

Assessments of progress toward sustainable well-being may be undertaken by corporations, communities, provinces, nations, or groups of nations. There will always be a role for scientific assessment – measurement of air, water, soil and biodiversity quality, etc. The real issue, however, is gaining an understanding of the evolving relationship between society and environment, and such assessments require broad-based participation. Regardless of who undertakes them, key rules or principles for guiding participatory assessments are:

Start with the story: If you want to learn what the problems are, don’t ask what the problems are. Ask what the story is and the problems will become evident. Start with developing a consensus about the story (or stories, if a single consensus proves impossible) of the community, corporation, nation or area being addressed. Use this mechanism to involve people from all arts of the community, especially anyone with a sense of history.
 
 

Build a broader community of interests: The different groups of decision-makers involved in the issues being assessed may not feel that they share interests. Ways of bringing them together into a common interest group include:

• identifying a broader community – by looking for other people who share the same or similar problems, the community can become broader and more powerful and understand its own problems better;

• act, don’t just talk – the sense of a community of interests and the understanding of its members can best be developed through joint activities, and communities that are brought together purely through talk are less likely to hold together; and

• look for ‘positives’ in common, ie those changes that the majority agree have improved their well-being and that of the environment; success stories will be important for keeping the strategy on track. Equally, a minimum base of community consensus can also be established by identifying those things that all participants agree they are against.

Recognize value differences: Although the community of shared interests is broader than people think, unavoidable conflicts often exist between the interests, needs and values of individuals, the local community, other communities or the larger society. It wwis better to bring these out into the open rather than present an illusory consensus.

Understand communication: At all levels, from conference papers to posters and television, it is essential to understand the media and the audience. Without such understanding, communication will not work (see Box 13).

Tools for participation

Although each of the tools for assessment – from thermometers to questionnaires – has its place, a few key considerations apply to the selection of tools:

• Learning by doing: We may break into the cycle of design-action-assessment at any point. Prolonged diagnostic exercises involving extensive questionnaires and paper studies usually yield fewer insights than a handful of thoughtful projects in which implementation is seen as a technique for learning. Action-oriented research and participatory inquiry are useful means (see Box 9).

• Maps: Maps of all kinds, from satellite images to sketches drawn on the ground, are powerful tools to understand problems, monitor change and communicate proposals. Although people unused to maps can experience problems, in most cultures simple map creation and reading is a skill that can be acquired quite easily.

• Meaningful indicators: Informative indicators can be developed only when we are clear about the question we are asking. A few well-chosen indicators are likely to be more useful than volumes of comprehensive statistics. Indicators should emerge from discussion and, where possible, should be those that people are already using. In many rural communities, indigenous technical knowledge can often supply more precise and revealing indicators of evolving society/ecosystem relationships than externally-defined ‘scientific’ indicators.

• Qualitative surveys: Assessment systems often focus on the accumulation of quantitative data. Although such data can be important, generally it needs to be accompanied by studies that reveal the story behind the numbers. A few anecdotal stories revealing how environmental change is affecting individual families or communities will illuminate the data and can often be more informative than extensive surveys.

• Open-ended questions: However thoroughly the problem has been discussed and however carefully the indicators are selected, the most useful information may be that which we are not looking for; the unexpected insightful observation that suddenly puts the problem in a new light. Questions should be phrased in ways that encourage comment rather than simply yes/no responses. Assessments should be structured in ways that throw people together in combinations from which new overlaps of knowledge and interest may emerge.

Making assessments useful

A useful assessment improves decision-making and facilitates action. It can do this, however, only if it provides information that helps decision-makers identify, agree on, and take such action. Decision-makers, whether individuals, communities, corporations, or governments, have three needs in common with respect to assessment of sustainability:

1. Relevance: The assessment must focus on issues that are relevant to the concerns, needs and priorities of the decision-maker.

2. Capacity to act: The decision-maker must be able to do something about the information provided by the assessment. Land-user families cannot do anything with information on ozone depletion. Business leaders cannot improve the sustainability of their operations with information on the crime rate.

3. Clarity: Decision-makers at all levels need clear signals that will help them decide what action they should take.

Comprehensive information buries signals in noise; information should be selective. Therefore, it is important to select aspects of ecosystem well-being, people-ecosystem interactions, and human well-being that:

• most reveal improvements or declines in these conditions and interactions; and

• are relevant, appropriate and clear to the decision-makers concerned.

If the people making the assessment and the decision-makers using the assessment are one and the same, this is not likely to be a problem. If they are different, special care will be needed to fulfill these requirements. It is therefore important to ask:

• Who is doing the assessment?

• Who can use the assessment?

• How do the two communicate?

• Are both committed to continuous assessment?

• Are the decision-makers concerned committed to act on information gained through the assessment?

Communicating assessments

Assessments should be communicated in whichever ways are most useful and meaningful to the decision-makers concerned. This includes:

• Starting by identifying the full range of decision-makers concerned. For example, a community that is weakened by the policies of central government would want the findings of its assessment to be communicated to decision-makers in central government and those who influence them.

• If necessary, communicating the assessment in different ways to different groups of decision-makers and other users: different products and events (not necessarily reports); and communication in different media.

• Using the right jargon for each group of users. Jargon tends to have a poor reputation. However, communication that tries to avoid it can end up being unintelligible (or simply boring) to its intended audience.

• Giving feedback (in a useful form) to people who provide information for the assessment.

Often the most useful task that an assessment can do is expose unsolved problems and identify untapped solutions. Such information is most likely to be obtained by processes that reward the constructive identification of failure. In turn, solutions are most likely to be implemented if the decision-makers concerned perceive them as reasonable, respectable and recognizable.

Conclusion

Monitoring and evaluating strategy performance has been one of the least developed elements of the strategy process. It is also one of the most important. Mechanisms need to be set in place so that nations or communities can steer their development according to the benefits of experience, and with the knowledge of changing circumstances, so that it stays on a sustainable path. This is not easy, for it requires the nation or community to have a practical vision of sustainability expressed in its own
terms. That vision will have a strong ethical and qualitative basis which has rarely been well-defined. How can a strategy assess its progress and course against goals that are intangible?

Usually, conventional methods of monitoring and evaluation rely on physical, economic or social indicators to measure what was achieved in the past, at either national or project level. These often focus on input supplies and their immediate results, especially at the project level. They might consider the number of hospital beds in a country or the number of trees planted in a village. The emphasis has been on measuring past performance through tangible products, and then considering what implications this has for future performance. These methods will continue to be valuable, although more emphasis is needed
on those underlying, less tangible qualities of development which lead to sustainability.

The experience so far has been sparse, but three directions for change are emerging:

1. Emphasis needs to be on indices governing the way things are done rather than what has been done.

2. In an environment of change and uncertainty, the concern should be primarily on modifying and influencing future performance and not on evaluating the past. The key to developing appropriate appraisal methods is understanding strategies in terms of constant change and adaptation to future needs.

3. These methods must themselves be ingredients of sustainable development and not something external to it. Concepts of action research or monitoring through action are appropriate here. They imply that those people involved in managing the strategy process, and in all elements of implementation, are each involved in a feedback loop of action –reflection–reaction. This works best at the local level where the reflection and feedback terms are more immediately beneficial to the participants.

As a consequence, assessment of strategies, including monitoring, evaluation and reporting, needs to stress process as well as products, and be anticipatory and action-based.