Assessing the Performance of Natural Resource Systems

Bruce Campbell¹, Jeffrey A. Sayer², Peter Frost³, Sonja Vermeulen⁴, Manuel Ruiz Pérez⁵, Anthony Cunningham⁶, and Ravi Prabhu⁷

• Abstract
• Introduction
• Conceptualizing a Framework for Assessing System Performance
• Bounding the System
• Developing Conceptual Models of Natural Resource Management
• Selecting Indicators: Using the Sustainable-Livelihoods Approach
  • The sustainable-livelihoods perspective
  • Coping with different spatial scales
  • Using qualitative indicators
• Incorporating Systems Modeling
• Developing a Participatory Approach to Assessment
• Using Typologies of Landscapes or Resource Management Domains
• Making an Integrated Interpretation of the Indicators
  • Combining indicators: simple additive indices
  • Combining indicators: derived variables using principal component analysis
  • Visualizing change: two-dimensional plots derived from PCA-type analyses
  • Visualizing change: radar diagrams
  • Combining indicators across scales: canonical correlations
• Conclusions
• Responses to this Article
• Acknowledgments
• Literature Cited

Assessing the performance of management is central to natural resource management, in terms of improving the efficiency of interventions in an adaptive-learning cycle. This is not simple, given that such systems generally have multiple scales of interaction and response; high frequency of nonlinearity, uncertainty, and time lags; multiple stakeholders with contrasting objectives; and a high degree of context specificity. The importance of bounding the problem and preparing a conceptual model of the system is highlighted. We suggest that the capital assets approach to livelihoods may be an appropriate organizing principle for the selection of indicators of system performance. In this approach, five capital assets are recognized: physical, financial, social, natural, and human. A number of principles can be derived for each capital asset; indicators for assessing system performance should cover all of the principles. To cater for multiple stakeholders, participatory selection of indicators is appropriate, although when cross-site comparability is required, some generic indicators are suitable. Because of the high degree of context specificity of natural resource management systems, a typology of landscapes or resource management domains may be useful to allow extrapolation to broader systems. The problems of nonlinearities, uncertainty, and time lags in natural resource management systems suggest that systems modeling is crucial for performance assessment, in terms of deriving “what would have happened anyway” scenarios for comparison to the measured trajectory of systems. Given that a number of indicators are necessary for assessing performance, the question becomes whether these can be combined to give an integrative assessment. We explore five possible approaches: (1) simple additive index, as used for the Human Development Index; (2) derived variables (e.g., principal components) as the indices of performance; (3) two-dimensional plots of indicators and cases emerging from multivariate techniques used to visualize change; (4) graphical representation of the five capital assets using radar diagrams; and (5) canonical correlation analysis to explore indicators at two different scales.

KEY WORDS: capital assets, conceptual models, decision support, livelihoods, modeling, multivariate statistics, natural resource systems, performance, Zimbabwe.

Published: December 31, 2001

There is wide agreement that the goals of eradicating poverty, attaining food security, and conserving the environment are highly interdependent. It has been suggested that integrated research on natural resource management is needed to address the emerging challenges, and that component research (e.g., on commodity crops) needs to be set within the context of natural resource management (Izac and Sanchez 2001). Integrated natural resource management (INRM) is a process of incorporating the multiple aspects of natural resource use (biophysical, sociopolitical, or economic) into a system of sustainable management to meet production goals of producers and other direct users (e.g., food security, profitability, risk aversion) as well as goals of the wider community (e.g., poverty alleviation, welfare of future generations, environmental conservation). The conceptual basis of INRM has evolved in recent years through the convergence of research in diverse areas such as sustainable land use, participatory planning, integrated watershed management, and adaptive management (Holling 1978, Pretty 1995, Holling and Meffe 1996, Walters 1997).

Research institutes and funding organizations have finite resources that they seek to allocate most efficiently. Therefore, they need to identify and assess priorities for research, monitor the progress of ongoing research, and evaluate the impacts of completed research. This is a difficult enough process in highly focused technological research projects, but is even more of a challenge for INRM research. Impact assessment of INRM research is in its infancy. For example, within the international research centers of the Consultative Group on International Agricultural Research (CGIAR) impact assessment has largely focused on germplasm adoption, with relatively little attention given to institutional impact, and almost none to INRM (e.g., Collinson and Tollens 1994; but see P. Frost, unpublished report, 1996). Impact assessment of INRM research would have to be based on an assessment of the performance of the natural resource management system, together with an assessment of the role that research plays in changing the development trajectory of the system.

The aim of this paper is to propose some methods for assessing system performance. In the first section, we conceptualize INRM and identify the role for performance assessment within a broader learning cycle. In the next two sections, we consider the importance of bounding natural resource management problems and using conceptual models. In the subsequent section, we turn to selection of indicators, suggesting that selection should be based on a sustainable-livelihoods approach. We make the case for systems modeling as a key component of INRM and the assessment of system performance. The problem of context specificity of INRM is then addressed. Finally, we look at some methods of integrating the indicator data.

We envisage INRM occurring within a specific geographical area, but at a number of scales, from farmers’ fields to entire catchments. Invariably, INRM would have to concern itself with sociopolitical, economic, and ecological variables (Fig. 1). The decision-making process and subsequent action take place within the context set by these variables. Almost all natural resource management systems involve multiple stakeholders, with multiple perceptions and objectives. There is likely to be a series of mechanisms by which stakeholder interests are integrated and traded off. To be effective and relevant, INRM has to be carried out at an appropriate scale and in a realistic context. At the level of smallholder farming systems, for example, research should be carried out mainly in farmers’ fields, where their problems reside, rather than on research stations. This would invariably involve a participatory component. Such a conceptual model for INRM indicates the numerous entry points for interventions and performance assessment.

Fig. 1. Components of INRM (modified from Swift et al. 1994).

Many interactions may need to be considered, e.g., upstream–downstream effects in a watershed; farm-level trade-offs among cash income, food security, risk aversion, and environmental conservation; and household choices about allocation of effort (e.g., as divergent as gold panning, out-migration, cropping particular species, building social capital). Complexities for INRM arise from:

• multiple scales of interaction and response;
• the high frequency of nonlinearities, uncertainty, and time lags in complex systems;
• multiple stakeholders with often contrasting objectives that complicate the task of identifying research and management aims and finding trade-offs among them;
• the context specificity of INRM sites; and
• the problem of maintaining integration in the face of numerous components and interactions.

Table 1. Key problems faced in assessing system performance in INRM. Problem/characteristic Way forward Comments 1. INRM systems are complex (multi-scales, multi-stakeholders, multi-sectoral, feedbacks, time delays, nonlinearities). Bound the system (clarify objectives, scale of research and particular intervention possibilities). Any reference to “clarification of objectives” is self-evident, but stresses the fact that performance assessment is an integral part of the whole research and learning cycle. Develop a conceptual model that simplifies the system and makes explicit the key components and interactions. This conceptual model would be at the level of the particular system being studied; e.g., it could be based on a site like Chivi (Fig. 2). Ensure careful indicator selection covering different scales, basing selection on the sustainable- livelihoods approach (Carney 1998). There is a need to strike a balance between simplicity and complexity.   2. Feedback, time delays, and non- linearities mean that performance assessment is complex. Develop simulation models as part of the performance assessment procedure. Simulation modeling may be essential to understand systems performance.   3. Participation is central to INRM, but external actors may have very different information needs from local stakeholders. Incorporate participatory assessment as well as more conventional systems. The participatory component is an ingredient in a feedback or learning process that is likely to increase the effectiveness of NRM.   4. INRM is context specific, but for general lessons, we need cross-site comparability. Situate INRM sites within a landscape or resource management domain typology.   5. Remaining integrated in the face of numerous indicators. Use techniques that can synthesize numerous indicators that may have been measured: multivariate statistics, radar diagrams.

One of the key lessons in dealing with complex systems is that management must be structured to promote active and conscious individual and social learning. Because of the inverse relationship between the complexity of systems and our ability to make significant statements about their behavior, an adaptive-management philosophy has been advocated (Holling and Meffe 1996). The steps within adaptive management are: design; act; monitor and observe; and reflect and revise. Maarleveld and Dangbégnon (1999) and Daniels and Walker (2001) characterize social learning as a continuous dialogue and deliberation among stakeholders that incorporates adaptive management as well as political processes related to conflict between stakeholders. Research thus becomes part of an ongoing cycle of planning, action, and evaluation. In performance monitoring and assessing research impacts, we envisage using an indicator-based approach within a social learning process. Many indicator approaches are based on a series of hierarchical concepts. The CIFOR Criteria and Indicator (C&I) team (1999) use a four-level hierarchy: principles, criteria, indicators, and verifiers. A similar hierarchy is envisaged for assessing the performance of natural resource management, although we would envisage a simpler hierarchical structure.

INRM can become a catch-all term for unfocused activities in which numerous system components are considered. Even assuming the same overall management objectives (e.g., “sustainability” or “equitable distribution” of benefits), the most appropriate indicators will vary with the scale at which management takes place and the scale at which prevailing social and economic processes operate. Interventions at one scale may have impacts at a different (higher) scale. Additionally, system performance might be assessed as being negative at one scale but positive at another; e.g., soil and water conservation interventions may improve crop yields at a specific site, but may show significant negative impacts at a larger scale by reducing water yields downstream. What, then, is the most appropriate level at which to judge the overall benefits? The answer depends on what types of impact are anticipated, the objectives of a specific assessment, the time scale used, the level of accuracy required, and the value system that is chosen by the evaluator.

Focusing INRM and assessing system performance therefore requires clearly stated objectives, a well-reasoned definition of spatial and temporal scales, and clear identification of particular intervention possibilities. The key to bounding the problem is the development of a conceptual model.

In implementing INRM, the starting point should be developing a conceptual model of the particular system under study, with a focus on identifying the key relationships among components of the system and the constraints operating on them. The model would be expected to address issues of spatial and temporal scale. A conceptual model could be viewed as a series of hypotheses about the processes operating. Thus, variables in the model should be theoretically and logically linked. The process of developing a conceptual model clarifies the nature of the problem itself, the bottlenecks to agricultural and natural resource production, the potential negative effects of resource development, and the possible entry points for interventions. The conceptualization should also identify the potential impacts resulting from interventions and management, and thus guide the selection of indicators. In this way, indicators can be selected that are causally and theoretically linked.

A conceptual model has been developed for Chivi, southern Zimbabwe (Fig. 2). This is a box-and-arrow conceptualization of livelihoods within the area. The model reflects the diverse livelihood options in the area, and some of the key “external” variables, such as AIDS and climate (in particular, drought). It was developed through a series of meetings involving various combinations of scientists, local people, and district officials. Although it is appropriate to initiate this activity at the start of the learning cycle, it should be revisited throughout the project, thus allowing for changing foci, interventions, etc., within the spirit of adaptive management. The model itself forms the basis for identifying key variables for assessing performance, but the process of developing the model is important in achieving a common understanding of the problems. Viewing indicators overlaid on a conceptual model illustrates their interconnectedness, an essential viewpoint if one is to achieve integration and understanding of the state of a natural resource management system.

Fig. 2. A conceptual model of a site in Chivi, Zimbabwe.

The literature indicates that there is no shortage of different indicators: in fact, the wealth of indicators is likely to mystify rather than enlighten. Thus the selection of indicators is a key step to be undertaken, preferably at the start of the INRM process. Simple indicator sets are desirable, but it would be foolish to expect simplicity when dealing with complex systems. Meaningful indicator sets will generally have to be extensive.

The sustainable-livelihoods perspective

In situations in which long-term gains in human welfare and maintenance or improvement of environmental quality are the goal, assessment of system performance could be based on the sustainable-livelihood concept. The concept integrates social, economic, and ecological dimensions (WCED 1987, Chambers and Conway 1992, Carney 1998, Bebbington 1999). The livelihoods framework identifies five core asset categories: physical, financial, social, natural, and human capital (Fig. 3). Principles for each of the five capital assets can be derived (Table 2), and indicators could be selected to cover each of the principles. The tendency to bias indicator selection to one particular discipline is thus avoided. The advantage of using the sustainable-livelihoods approach is that the concept has been vigorously debated in the literature and forms a relatively sound theoretical basis for indicator selection. In many indicator approaches, choice of indicators may be relatively ad hoc. Indicator selection would normally involve experts from different disciplines and the various stakeholders.

Fig. 3. The five capital assets (modified from Bebbington 1999 and Carney 1998).

Table 2. Some suggested principles for each of the capital assets, with examples of criteria for each of the principles. The example is for illustrative purposes only: the principles should not be seen as definitive. Capital asset Principle Examples of a criterion for each of the principles Natural capital Options for future use are maintained. Processes that maintain biodiversity are conserved.     Yield and quality of natural resource goods and services are maintained or improved. Ecosystem function is maintained.   Financial capital Financial capital is circulated within the system. Service and commodity outlets expand in the local and district centers.     Financial capital grows and is equitably distributed. Residents have reasonable share in economic benefits derived from resource use.   Physical capital Physical capital is maintained or improved over time. Housing physical status is maintained or improved.   Human capital Ability to provide added value is improved over time. Greater array of value-added products are produced locally.     Improved and equitable distribution of human capital. Level of skills with respect to running committees and organizations is improved.   Social capitala Maintenance of systems of social reciprocity. Economic and other shocks are buffered by systems of social reciprocity.     Maintenance of a set of dynamic rules and norms. Local rules are effective in controlling access to resources.   a We include organizational capital within this, although it could be argued that it forms a separate capital asset (e.g., see Bossel 1998). This covers, for example, by-laws at a district level and cultural norms and local rules at the community level.

The capital assets are closely linked to each other (Fig. 4). This figure focuses our attention on the dynamic nature of natural resource management, clarifying the interacting and integrated nature of indicators. Selecting indicators that do not represent the full spectrum of capital assets is inappropriate. For example, if financial capital is very low because it has been mobilized to improve human and physical capital, then the system may be judged to be more acceptable than systems in which financial capital is higher, but in which no financial resources have been transferred into other capital assets. It may be appropriate to develop the concept of lowest permissible limit, beyond which there would be a “capital bottleneck” limiting the achievement of a sustainable livelihood.

Fig. 4. The dynamic nature of capital assets.

Coping with different spatial scales

Hierarchy theory indicates that work at a particular scale of organization often requires insights from at least two other scales (Allen and Starr 1982, O’Neill et al. 1986). Thus work at the farm/household level may require component studies at lower levels, such as the plot level or the intrahousehold level, to understand the important processes that lead to the emerging characteristics at the household level. Work at the farm/household level will also require work at higher levels, e.g., into the institutional framework established by local government. Comprehensive assessment of natural resource management will invariably require that indicators be selected from a number of scales. More commonly, however, assessments focus on a single scale; although this might fulfill objectives defined by the evaluator, it results in an incomplete assessment. For example, assessments that focus on productivity gains from the application of insecticide, but ignore any deleterious effects of the herbicide on human health or the environment, are incomplete assessments.

Criteria and indicators attempting to capture similar phenomena will vary according to the scale of analysis (Noss 1990), as is demonstrated for Chivi (Table 3). Much of the work in Chivi is being conducted at the scale of a 4.5-km² micro-catchment. This catchment supports a well (Bromley et al. 1999), but the social catchment for the well extends beyond the focus catchment into others, one of which supports a small dam. In spite of the focus on the micro-catchment, scale issues are being considered, both for larger biophysical units (e.g., what are downstream impacts of the developments in the micro-catchments) and for larger institutional scales (e.g., how do the three traditional villages in the micro-catchment interact with the larger administrative units, up to the district-level government, and with water governance units established at national, catchment, and subcatchment levels). At lower scales, some key processes are being studied, e.g., tree–soil water relations (because trees are hypothesized to be a major cause of groundwater recession in the catchment). The need to use GIS tools within the context of multiple scales is self-evident.

Table 3. Different scales at the Chivi site and some potential criteria for those scales, with one criterion shown for each of five capital asset principles. Principles for each capital asset Potential criteria     Household/farm fields Village/micro-catchments District Natural capital: Yield and quality of natural resource goods and services is maintained or improved. Soil fertility in garden fields is maintained or improved. Groundwater resources for community well are maintained or improved. Siltation levels in main dams are reduced.   Financial capital grows and is equitably distributed. Household savings grow and are equitably distributed. Micro-credit scheme is maintained and expanded. Council budgets increase.   Physical capital is maintained or improved over time. Housing condition is maintained or improved. Water availability is improved. Road infrastructure is maintained or improved.   Improved and equitable distribution of human capital. Educational status of households improves. Level of skills with respect to running committees and organizations is improved. Budgetary control is maintained and improved.   Social capital: Maintenance of a set of dynamic rules and norms. Local rules are effective in controlling access to resources. Leadership at the district level is respected.

Using qualitative indicators

Performance assessment of natural resource management will invariably include a qualitative component. Conventional monitoring systems often only help to inform us of outcomes that are expected or predictable. Many outcomes may not be covered by monitoring systems because they are not expected. In Chivi in 1981, it would have been difficult to predict that gold panning, which had been all but absent, would become one of the most important livelihood options by the end of the decade. It would have been difficult to predict that there would be over 25 woodcraft markets on a 100-km stretch of road by 1995 (a nearly fourfold increase from 1991), and that AIDS would wreak havoc in the community in the last five years of the millennium. Performance assessment may have to rely on qualitative indicators for unexpected phenomena that occurred and for which quantitative data were not initially recorded.

During the course of INRM, local people’s feelings about the direction of change can be recorded (given that some outcomes may only be measurable many years after a management intervention). By capturing local people’s perspectives, albeit often qualitative, we would be integrating numerous variables. In addition, considering that the political arena in any local venture is highly charged (and that researchers are stakeholders with particular agendas that are challenged and modified by local people), it becomes particularly important that performance assessment is informed by anthropological perspectives, which usually provide qualitative data.

The outcome of natural resource management can be defined as the difference between what happened (as a consequence of the management) and what would have happened anyway. In many cases, baseline data are collected at the start of a management cycle in order to assess change in system characteristics. This is an inadequate approach, as the baseline data do not reflect the dynamics of “what would have happened anyway.” Alternatively, assessment of management interventions could be based on large-scale experimentation (i.e., implementing components of a program in some localities but not in others), in conjunction with a statistical sampling program. Such an approach is also unrealistic because of the high expense (Walters 1997). Given the dynamism of natural resource management systems, and the fact that large-scale experimentation is usually not feasible, one of the few solutions for performance assessment is the use of systems modeling. It will often be more appropriate to compare measured indicator values with values derived from systems models for the “what would have happened anyway” scenarios than to compare them with baseline data.

The need for systems modeling is clear in savanna regions, where biological productivity depends, to a large degree, on rainfall, and where each year brings markedly different rainfall conditions. Any attempt to monitor change, and to attribute such change to management, is fraught with difficulty, because many changes will be driven by rainfall patterns. Under these circumstances, systems models are ideal for exploring systems performance. Similar arguments can be applied to many of the external drivers of natural resource management systems.

Systems modeling has diverse functions within INRM. There are two major applications, first to compare observed changes with those expected in the absence of particular management interventions, and second to gain insights regarding likely future impacts of different kinds of management. In both cases, the emphasis is on improving understanding to increase the efficiency and effectiveness of natural resource management. In terms of the learning cycle, systems modeling is implemented soon after the initiation of INRM, with data inputs being best bets and the modeling results being used to set priorities and guide the action phase of the work. Later in the learning cycle, or in subsequent cycles, the models may become more sophisticated, allowing greater confidence in the exploration of likely impacts of management. Systems modeling is thus a tool for understanding the consequences of both short- and long-term changes in the components of a system, at a range of scales. In the evaluation phase of the learning cycle, the systems model, combined with indicator measurements, becomes a tool for assessing systems performance.

Systems analysis can be conducted as a multistakeholder participatory process, as in the case of the systems models of van der Belt et al. (1998) and Lynam et al. (2002). Although systems modeling was, until recently, relatively inaccessible to the non-expert, the software that is now available makes it highly accessible to stakeholders in natural resource management systems, as indicated by the building of a land-use and forestry model for Mzola State Forest and adjacent communal areas in a two-week period during a modeling training course (Campbell et al. 2000).

The need for a participatory approach within INRM is implicit, almost by definition, but here we focus on the assessment component of that participation. There is an extensive literature on participatory assessment, the process by which indicators are identified and used, and the negotiation of a shared understanding of what constitutes “favorable outcomes” (e.g., Abbot and Guijt 1998, Guijt 1998). Participatory assessment becomes a vital ingredient in a feedback or learning process that, in turn, increases the effectiveness of the overall process of participatory management. The Landcare program in Australia (Campbell 1998) is an example in which conservation extension groups involving a broad cross-section of rural people with a stake in catchment planning are using techniques such as GIS and aerial surveys for assessment. For researchers, there is also a pragmatic component to using a participatory approach: it provides a cost-effective alternative to expensive statistical sampling programs.

In our view of participatory assessment, local stakeholders are involved both in the design of the assessment system, including the selection of indicators, and in the collection of information from it. Thus a fundamental aspect of the design and use of indicators requires negotiating a common framework that allows for maximum overlap between the information interests of the concerned stakeholders.

Local systems of assessment can be rich in detail and incorporate indicators that satisfy several of the information demands of complex systems. There is, however, one fundamental problem with local information systems: they are developed in the context of a community of local users, with shared interests and paradigms, managing resources that they consider their own, and isolated from the needs and demands of other stakeholders. Thus feedback from utilization other than their own is inadequately captured, downstream impacts may be considered unimportant, planning takes little account of external demands and needs, assumptions about rights become controversial, and the language and idiom of communication tend to shut out external stakeholders.

For particular components of the system, detailed data may be required to assess system performance. The data may be more or less meaningless without further analysis (e.g., they may act as points to calibrate a systems model; they may require detailed statistical analysis to detect trends). To expect a community to participate in data collection that requires a considerable time outlay, without clear benefits to them, is unrealistic. In the Chivi site, local people were hired to collect hydrological data that were considered critical to assessing the impacts of land use (Bromley et al. 1999). The local monitors benefit financially from this work, and use some of the information to change their own activities or to convince others to change, but they would not collect such information without financial reward. Thus, although we see a component of the assessment of natural resource management being undertaken within a participatory framework, another component would involve more extractive data collection systems.

Although selecting indicators to address general features of natural resource management systems will be necessary for effective cross-site comparisons, this may not be sufficient for effective natural resource management, as particular problems and sites have specific contexts that also need to be addressed. What we have suggested as principles (Table 2) may apply to a wide variety of natural resource management systems, but a generic set of indicators must take into account the context of the particular site. Indicators vary widely across different ecosystem types in southern Africa (see Table 4). The problem of defining indicators for systems performance must be addressed at two (or more) levels: a broad level of indicators that help to evaluate the effectiveness of management generally; and a narrower, more context-specific set of indicators that relate to the particular sociopolitical, economic, and ecological conditions of a defined system.

Table 4. Examples of criteria for each of five principles drawn from different capital assets for three landscape types in southern Africa. Principles for each capital asset Criteria   Arid woodlands on Kalahari sands Miombo woodlands on nutrient poor soils Dry woodlands on rich soils Natural capital: Yield and quality of natural resource goods and services are maintained or improved. Frequency of hot fires reduced. Soil fertility levels in garden fields are maintained or improved. Key resources for grazing are maintained.   Financial capital grows and is equitably distributed. Revenues from logging and hunting are increased and equitably distributed. Revenues from communal water points are increased and cover maintenance costs. Livestock fund for recovery programs after droughts is maintained.   Physical capital is maintained or improved over time. Firebreaks are maintained. Numbers of bore holes for irrigation are increased. Dip tanks are maintained.   Improved and equitable distribution of human capital. Community business skills in dealing with tourism operators are improved. Community skills for running micro-credit and water point committees are improved. Community skills for dealing with livestock diseases are improved.   Social capital: Maintenance of a set of dynamic rules and norms. Rules of access to the forest and fire control rules are maintained and improved. Rules of access to communal water points are adhered to. Rules of access for grazing in different key resources are maintained.

In this regard, and within a global research agenda, it would be useful to develop a landscape typology or a typology of resource management domains. Land use is an expression of both the opportunities and constraints presented by the interactions among biophysical, economic, social, and technological components operating in an environment at a particular time, with a particular history. It should be possible to produce a typology of land-use systems that focuses on the key relationships among these components and the constraints that they impose on the predominant land uses. Then one can identify the more context-specific indicators that are sensitive to, and reflective of, the particular features of a given land-use system. Many international research centers have already gone some way toward producing appropriate landscape typologies.

Given that a number of indicators are necessary for assessing systems performance, often at a variety of spatial and temporal scales, the question then becomes whether these can be used to give an integrative summary of performance. By using conceptual and systems models in INRM, in which indicators are explicitly linked, some degree of integration across spatial and temporal scales will be achieved. We examine five further methods, not mutually exclusive, that can assist in ensuring integration. The data for these illustrations have been derived from systems models. In actual performance assessment, observed values would also be used and compared to simulated values for the “what would have happened anyway” scenario.

Combining indicators: simple additive indices

Approach

A simple additive index can be calculated in much the same way as is done for the Human Development Index (UNDP 1994). For each indicator considered, a maximum and a minimum are defined. These can be the actual minima and maxima expected in the data or, where the data under consideration do not cover the full spectrum of possible variation, expected values can be based on theory. For example, a measure of minimum woody basal area could be the minimum permissible limit that is required to satisfy basic household livelihood needs. A standardized value for each indicator is then calculated, using the formula: (Indicator value at time x - minimum)/(maximum - minimum). For each indicator, the potential values run from 0 (least desirable) to 1 (most desirable). A composite index is calculated as the average of the indicator values.

Example application

The method is illustrated using variable values derived from a systems model of Chivi. This model was produced using the Stella modeling package. The model included crop and livestock keeping, forest product collection, and various ecological sectors: rainfall, vegetation dynamics, and fire. To keep it simple, we selected only four variables, two representing natural capital (basal area of woody plants, area of cropland per household), one representing physical capital (numbers of livestock per household), and one representing financial capital (disposable income, i.e., cash income minus cost of inputs for crop and livestock production). The values were generated for every fifth year from the time of project implementation for a 20-yr period. Three simulation scenarios were run: (1) no interventions (Scenario A); (2) crop yield and livestock pen feed raised by 20% per year, and 10% of all trees removed in year 2 (to stimulate grass production for rough grazing) (Scenario B); (3) crop yield and livestock pen feed raised by 50% per year, and 20% of all trees removed in year 2 (Scenario C).

Example results

The additive index of capital assets fluctuates widely, but generally declines over time (Fig. 5). A less marked pattern is due to the intervention, with higher index values for scenarios with interventions. The fluctuations are largely related to rainfall and its impacts on agricultural production. The decline reflects the long-term trend toward smaller land holdings and lower numbers of livestock per household, given the rise in household numbers in an already heavily populated landscape. It is predicted that the interventions will make a difference, but their impact can be masked by other phenomena.

Fig. 5. Change over time in Chivi for three scenarios (rainfall patterns in the different scenarios are the same) using a simple additive index. Scenario A, no interventions; Scenario B, crop yield and livestock pen feed raised by 20% per year, and 10% of all trees removed in year 2; Scenario C, crop yield and livestock pen feed raised by 50% per year, and 20% of all trees removed in year 2. The index is derived from average values for four variables, with the values being derived from a Stella simulation model.

The problem with the additive index is that the variation in individual indicator values is reduced to a single number for a particular time period. To understand this single figure, one has to go to the original data and look at the values for each of the indicators that make up the index. This may not be a problem when there are only four indicators, as in the example, but is problematic when there are numerous indicators. In the example, differences between intervention scenarios are largely due to changes in livestock holdings and woody plant basal area. Because these variables show opposite trends, the simple index may be hiding important differences among variables (Fig. 6).

Fig. 6. A comparison of the variable values for year 15 for Scenarios A and C (see Fig. 5) for woody plant basal area and livestock numbers. The variable values were derived from a Stella simulation model.

Combining indicators: derived variables using principal component analysis

Approach

A more sophisticated method of combining indicators into a single variable is to use principal components analysis (PCA), or a related multivariate technique. PCA-type methods are often used in data reduction to identify a small number of factors that explain most of the variance observed in a much larger number of variables. The first new factor (first principal component [PCI]) or derived variable, Y₁, is a linear combination of the original variables, i.e., Y₁ = aX₁ + bX₂ + cX₃ . . ., where X₁, X₂, X₃ . . . are the standardized original variables, and a, b, c are the fitted coefficients (it is not necessary for the analyst to standardize the variables prior to analysis; the standardization procedure is routine within statistical packages). Y₁ is constructed so that it accounts for the maximum possible information in the original set of variables. Further factors can be derived, each explaining some residual variation in the matrix (Y₂, Y₃ . . .).

Example application and results

The same data used to illustrate the additive index were submitted to a PCA, using a data matrix of the four variables and 15 cases (data from five different years for each of three scenarios; Fig. 7). Measured data on the variables could be included as additional cases. The first PC is dominated by the woody plant basal area variable, while the second PC is dominated by the influence of livestock numbers per household (the equations are illustrated in Fig. 7; the higher the coefficients for a variable, the higher the influence of that variable in the component). Both PCs show a decline over time (largely a result of declining natural and physical capital). It is predicted that the intervention will cause a greater decline than the non-intervention for the first PC (largely related to natural capital, loss of woody plant biomass), but will result in higher values than the non-intervention for the second PC, illustrating the positive effect of the intervention on physical capital (livestock numbers). Such multivariate techniques become particularly powerful when more variables are being used.

Fig. 7. The changes in the derived variable (Principal component I and Principal component II) over time. The make up of these derived variables from the original variables is indicated; the larger the absolute values of the displayed coefficients in these equations, the more effect that variable has in the derived variable. The original variable values were derived from a Stella simulation model.

Combining indicators for each capital asset

Each capital asset comprises a number of different variables (e.g., social capital is a function of the size of the extended family, connectedness to other members of the community, membership of groups, extent of reciprocal relations, social indebtedness). We have illustrated the use of PCA-type tools to combine all possible indicators into an overall index (e.g., Fig. 7). The procedure would only be recommended when dealing with relatively few indicators (e.g., less than 20), or when there is not a pressing reason to maintain a capital-assets perspective. A conceptually neater technique would be to use a PCA-type method with the set of indicators that fall under social capital to get a single, derived variable for social capital, and so on for the other capital assets, and then to display the capital-asset situation as a radar diagram.

Visualizing change: two-dimensional plots derived from PCA-type analyses

Approach and example application

With a multivariate technique such as PCA, we also have a visual means of displaying the results, which does not require that we deal with indices or derived variables. The method is displayed for the simple four-variable Chivi case (Fig. 8). The points on the graph are coded A for Scenario A, B for Scenario B, and C for Scenario C, and the time when the data were collected is coded 0 for time 0, 5 for 5 years after the start, etc. Actual measured indicators could be incorporated into the data matrix and would thus also be displayed on the diagram. The distance between the points on the two-dimensional graph (A0, A5...., C20) represents the degree of difference between these cases, in terms of their values. Thus, A0, B0, and C0, all being closely placed at the right of the x-axis, have very similar values, whereas C15 is very different. The technique also displays the variables used in the analysis (in this case, basal area, income, cattle numbers, and crop area). Thus A0, B0, and C0 at the right-hand extreme of the graph have high values for woody basal area and low values for income, whereas points at the left-hand extreme (e.g., C15) would show the opposite pattern. The y-axis is largely related to livestock numbers, with points at the top of the graph (e.g., C5, B5) having high numbers, whereas points at the bottom of the graph (e.g., A20) have low numbers. In this example, for any specific time period, the cases with the interventions are toward the top of the y-axis, primarily indicating higher livestock numbers.

Fig. 8. Scatter diagram showing the distribution of cases in two dimensions, with the distance between the case positions representing the degree of difference of the cases. The cases are coded as: A, Scenario A (no intervention); B, Scenario B (intermediate intervention); and C, Scenario C (large intervention; see Fig. 5); with numbers from 0 to 20 indicating the start (Year 0) to 20 years. Also shown are the four variables used to produce the diagram, with the variable positions indicating the cases (those near that position) that have generally high values of the variable. The variable values were derived from a Stella simulation model.

Example results

The results indicate that the main trend (first PC) is the decline with time of woody plant basal area and cropping area, a decline that is greater in the intervention scenarios because of the reduced woody plant basal area. The interventions maintain higher livestock numbers than the non-interventions. The lack of simple patterns in the diagram (e.g., C0, C5, C10, and C15 are not neatly in order) is due to fluctuations in the variables, caused by rainfall.

Visualizing change: radar diagrams

Approach and example application

Radar diagrams, available in Microsoft Excel, for example, can be used to display the state of all capital assets (Fig. 9). We have used another model for Chivi, on the impact of micro-credit schemes, to illustrate the use of a radar diagram. The numbers were generated by a decision support system based on a Bayesian Belief Network (derived from that of J. Cain, unpublished data, 2000). For each of the capital assets, a proxy variable was selected: (1) physical capital, percentage of households with “improved roofing” (income generated from activities sponsored by the micro-credit scheme are often used to improve household holdings); (2) financial capital, percentage of households achieving a “high” level of savings; (3) natural capital,percentage of households having soil-improved fields; (4) social capital, percentage of households adhering to community-based rules; and (5) human capital, percentage of committees exposed to, and practicing, improved methods of organization.

Fig. 9. Radar diagram showing the impact of a micro-credit scheme on capital assets. The values for the assets are standardized values (running from 0 to 1) derived from variables in a decision support system based on a Bayesian Belief Network.

Results

For these simulations, some degree of soil moisture security (e.g., irrigation or high rainfall years) was envisaged; without such security, the impacts of micro-credit are very limited. The results indicate that the impact of the micro-credit scheme is likely to be improved social capital, and to some extent, improved natural capital, rather than improved financial capital (Fig. 9). The broader research program has focused on developing social capital as a precursor to common property resource management; hence, the impact on social capital. Actual measured indicator values could be compared to the simulated values by including a third pentagon on the radar diagram.

One challenge to using a radar diagram is that a single indicator must represent a capital asset (as used for Fig. 9), or that we must collapse all of the individual indicators under a particular capital asset into one index of that asset. The latter can be done using principal component analysis, or a related technique, as described earlier.

Combining indicators across scales: canonical correlations

Approach

Although the methods that we have mentioned are suitable for one of the spatial scales within a system, they are not easily extended to multiple spatial scales, as is necessary in INRM. The indicators from different scales could be entered into the same data matrix. In this way, where there are two scales, there are two sets of indicators, and techniques very similar to principal components analysis can be used. Additionally, however, the relationships between the two sets of indicators can be explored. The limitation is the numbers of indicators that are generated, requiring ever increasing observations for each of the indicators. With five capital asset indicators and two scales, there are 10 indicators, requiring more than 10 observations (e.g., with and without simulated scenarios for five time periods plus one measured set of values for one time period would give sufficient cases). If one has such data, then canonical correlation can be used.

Example application and results

In the example, we look at the impact of the micro-credit scheme on the village where it is implemented, as well as the larger district in which the village falls. Data for the first two years for the village have been derived from the decision support model used in the previous section, but data for the other years and for the district have been made up. The nonlinear canonical correlation analysis shows the relationships among the variables (Fig. 10) and the cases (different scenarios and year) (Fig. 11). Many of the village-level variables are correlated with each other and with social capital at the district level (Fig. 10). This is because there was a conscious effort by the researchers to involve the district government in building governance systems. The second dimension indicates that natural capital at the district level is negatively correlated with human, physical, and financial capital at the district level. Cases with the micro-credit scheme (B1, B2...) are on the right of the x-axis, with high levels of most of the capital assets, whereas cases without the credit scheme have low levels and are on the left side of the x-axis (Fig. 11). Time is captured by the y-axis, with early observations at the bottom (high natural capital) and late observations at the top (higher levels of other types of capital).

Fig. 10. Incorporating variables from two scales. Scatter diagram showing the distribution of variables in two dimensions, with the distance between the variable positions representing the degree of correlation between variables. The figure is illustrative only because, although the data for the first two years are from a decision support system based on a Bayesian Belief Network, the subsequent year’s data have been made up.

Fig. 11. Incorporating variables from two scales. Scatter diagram showing the distribution of cases in two dimensions, with the distance between the cases representing the degree of difference of the cases. The cases are coded as A, a scenario with no project, and B, a scenario with a project; while the numbers indicate yearly time-steps from the first year to the sixth year (0-5). Also shown are the capital assets that vary across the diagram. The figure is illustrative only, for although the data for the first two years (coded 0 and 1) are from a decision support system based on a Bayesian Belief Network, the subsequent year’s data have been made up.

Results suggest that the micro-credit scheme will make a difference, but mostly at the village level, except that district-level social capital will be built up through the stakeholder negotiations and district-level governance efforts that are part of the research. However, the positive impact at the village level must be set in the context of other changes at the district level, notably the decline in natural capital, but improvement in other capital assets.

We advocate an approach to the assessment of systems performance that is part of a learning process fully integrated within participatory research. This has a number of implications, most notably the need for constant iteration between management and assessment. The approach requires the use of many qualitative indicators. Many components of performance assessment need to be initiated at the start of the INRM learning cycle, e.g., bounding the system, developing a conceptual framework, selecting indicators, initiating the development of a systems model, and situating the site within a typology of landscapes or resource management domains. Although many of these activities are part of the learning cycle for reasons other than performance assessment, they are also crucial for assessment. During the evaluation phase of the learning cycle, data from numerous indicators will generally be available: a challenge is to remain integrated. We suggest various tools for making integrated statements about trends in indicators, across scales, including the use of radar diagrams and multivariate techniques. Given the numerous external influences on natural resource management systems, simply viewing indicator data collected from the field may prove meaningless because they may be reflecting trends unrelated to management. Thus indicator values measured in the field may have to be compared with values derived from systems models. This should not be interpreted to mean that assessing the impact of INRM research will, itself, constitute a major research undertaking. In reality, most of the data required would have been collected anyway in the course of INRM. What is advocated in this paper is the organization of data on indicators into an adaptive-management framework that will allow for constant enhancement of the performance of the system. Well-conceptualized performance assessment frameworks should render research and management more efficient and may reduce data requirements by suggesting redundancies in the overall process. What we suggest is a radical departure from conventional impact assessment studies, as they have been applied to agricultural research. It is, however, consistent with moves toward greater use of action research, greater participation, and a general move down the research–management continuum. We believe that this sort of INRM will be needed to address the complex natural resource management problems that will determine the development options for the world’s poor in the 21st century.

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.

Much of the empirical results on which this paper is based are from Chivi in Zimbabwe, the outcomes of a research project funded by the Department for International Development of the United Kingdom (Zimbabwe: Micro-catchment and Common Property Resources - R7304) and the Swiss Agency for Development and Cooperation (Stakeholders and Biodiversity in the Forests of the Future), jointly undertaken by the Center for International Forestry Research, the Centre for Ecology and Hydrology and the Institute of Environmental Studies. However the views are those of the authors and do not necessarily reflect those of the funders. We thank Hans Gregersen, Mike Spilsbury and an anonymous reviewer for comments. We acknowledge the inputs of Edward Chuma and Steve Waddington to earlier discussions.

Abbot, J., and I. Guijt. 1998. Changing views on change: participatory approaches to monitoring the environment. S.A.R.L. (Sustainable Agriculture and Rural Livelihoods) Discussion Paper Number 2, IIED (International Institute for Environment and Development), London, UK.

Allen, T. F. H., and T. B. Starr. 1982. Hierarchy perspectives for ecological complexity. University of Chicago Press, Chicago, Illinois, USA.

Bebbington, A. 1999. Capitals and capabilities: a framework for analyzing peasant viability, rural livelihoods and poverty. World Development 27:2021-2044

Bossel, H. 1998. Earth at a crossroads: paths to a sustainable future. Cambridge University Press, Cambridge, UK.

Bromley, J. A., J. A. Butterworth, D. M. J. MacDonald, C. J. Lovell, I. Mharapara, and C. H. Batchelor. 1999. Hydrological processes and water resources management in a dryland environment I: An introduction to the Romwe Catchment Study in Southern Zimbabwe. Hydrology and Earth Sciences 3:322-332.

Campbell, A. 1998. Fomenting synergy: experiences with facilitating Landcare in Australia. Pages 232-249 in N. G. Röling and M. A. E. Wagemakers, editors. Sustainable agriculture and participatory learning. Cambridge University Press, Cambridge, UK.

Campbell, B. M., R. Costanza, and M. van den Belt. 2000. Land use options in dry tropical woodland ecosystems in Zimbabwe: introduction, overview and synthesis. Ecological Economics 33:341-352.

Carney, D. 1998. Sustainable rural livelihoods. What contribution can we make? Department for International Development, London, UK.

Chambers, R., and G. Conway. 1992. Sustainable rural livelihoods. Practical concepts for the 21st century. IDS Discussion Paper 296, Institute for Development Studies, Brighton, UK.

CIFOR C&I team. 1999. The CIFOR criteria and indicators generic template. Center for International Forestry Research, Bogor, Indonesia.

Collinson, M. P., and E. Tollens. 1994. The impact of the International Agricultural Research Centers: measurement, quantification, and interpretation. Issues in Agriculture 6:1-41.

Daniels, S., and G. Walker. 2001. Working through environmental conflict: the collaborative learning approach. Praeger, London, UK.

Guijt, I. 1998. Participatory monitoring and impact assessment of sustainable agriculture initiatives. S.A.R.L. (Sustainable Agriculture and Rural Livelihoods) Discussion Paper Number 1. IIED (International Institute for Environment and Development), London, UK.

Holling, C. S., editor. 1978. Adaptive environmental assessment and management. John Wiley, New York, New York, USA.

Holling, C. S., and G. K. Meffe. 1996. Command and control and the pathology of natural resource management. Conservation Biology 10(2):328-337

Izac, A.-M. N., and P. A. Sanchez. 2001. Towards a natural resource management paradigm for international agriculture: example of agroforestry research. Agricultural Systems 69:5-25.

Lynam, T., F. Bousquet, P. d’Aquino, O. Barreteau, F. Chinembiri, and B. Mombeshora. 2002. Adapting science to adaptive managers: spidergrams, belief models and multi-agent systems modeling. Conservation Ecology 5(2): 24. [online] URL: http://www.consecol.org/vol5/iss2/art24

Maarleveld, M., and C. Dangbégnon. 1999. Managing natural resources: a social learning perspective. Agriculture and Human Values 16:267-280.

Noss, R. F. 1990. Indicators for monitoring biodiversity: a hierarchical approach. Conservation Biology 4:355-364.

O’Neill, R. V., D. K. DeAngelis, J. B. Waide, and T. F. H. Allen. 1986. A hierarchical concept of ecosystems. Princeton University Press, Princeton, New Jersey, USA.

Pretty, J. N. 1995. Regenerating agriculture: policies and practice for sustainability and self-reliance. Earthscan, London, UK.

Swift, M. J., L. Bohren, S. E. Carter, A. M. Izac, and P. L. Woomer. 1994. Biological management of tropical soils: integrating process research and farm practice. Pages 209-227. in P. L. Woomer and M. J. Swift, editors. The biological management of tropical soil fertility. John Wiley, Chichester, UK.

UNDP.. 1994. Human Development Report 1994. United Nations Development Programme, New York, New York, USA.

van den Belt, M., L. Deutsch, and A. Jansson. 1998. A consensus-based simulation model for management in the Patagonia Coastal Zone. Ecological Modelling 110:79-103.

Walters, C. 1997. Challenges in adaptive management of riparian and coastal ecosystems. Conservation Ecology 1(2):1. [online]1(2):1 URL: http://www.consecol.org/vol1/iss2/art1

WCED. 1987. Our common future. World Commission on Environment and Development, Oxford University Press, Oxford, UK.