INTRODUCTION

A scenario is a coherent, internally consistent, and plausible description of a potential future trajectory of a system (e.g., Heugens and van Oosterhout 2001). Scenario planning exercises aim at articulating multiple alternative futures in a way that spans a key set of critical uncertainties (Peterson et al. 2003a, Kok and Van Delden 2009), using qualitative and quantitative methods and data (Swart et al. 2004, Carpenter et al. 2015). Scenario planning has its roots in operations research developed in the Second World War and was substantially elaborated upon in corporate strategic planning in the 1970s. It has been increasingly applied in diverse environmental research contexts during the past 25 years, including biodiversity assessments, the management of protected areas, ecosystem services (ES), and their relationship to human well-being, climate change, and land-use change in general, and more specifically, desertification and land degradation (e.g., Sala et al. 2000, Brown et al. 2001, Kok et al. 2004, Bradfield et al. 2005, Jessel and Jacobs 2005, Pereira et al. 2005, Carpenter et al. 2006, Gude et al. 2007).

The steady increase of scenario planning in environmental research can be attributed to a number of perceived benefits. These include fostering long-term and complex thinking that allows for an exploration of the dynamics and sustainability of social-ecological systems. The adaptability and accessibility of scenario planning compared to other modeling approaches might also explain this increasing trend. Although lack of rigor is a potential weakness in scenario planning exercises, this is often compensated by its utility to clarify, distinguish, and explore social-ecological feedbacks and potential surprises that cannot be easily represented in more formalized modeling approaches (Bennett et al. 2003). Addressing feedbacks and surprises, however, is fundamental when managing sustainability in complex social-ecological systems (Kok et al. 2007, Walz et al. 2007).

Scenario planning processes are often oriented toward influencing decisions (Wollenberg et al. 2000), which means they can potentially have a wide range of implications for a diverse set of stakeholders. Accordingly, scenario planning in environmental research and management of natural resources has become more participatory. Involving diverse stakeholders with influence and interest in the social-ecological system, including those potentially most affected (Bohnet and Smith 2007, Kok et al. 2007), might foster social learning and collective action to achieve desired goals (Butler et al. 2014, 2015). Hence, participatory scenario planning (PSP) is a process in which stakeholders, frequently guided by researchers, are engaged in a highly collaborative process and develop a leadership role within some or all stages of a scenario development process to investigate alternative futures.

The rationale for stakeholder engagement in scenario planning follows normative and pragmatic arguments, many of which relate to process-oriented results that are emerging from broader participation discourses (Stringer et al. 2006, Butler et al. 2013, 2015); to empower stakeholders (Reed et al. 2013a); to stimulate innovation (Butler et al. 2015); to mitigate conflicts (e.g., Kahane 2012); to encourage social learning (e.g., Volkery and Ribeiro 2009); and to integrate different types of knowledge (e.g., scientific, local), perceptions, expectations, and aspirations (e.g., Bohnet 2010, Von Wirth et al. 2014). In particular, PSP processes can facilitate discussions regarding the future effects of drivers of change on human well-being, ecosystem services and their trade-offs, biodiversity, or other social-ecological components across multiple spatial, temporal, or institutional scales. Further, PSP can be viewed as a solutions-oriented technique because it can increase adaptive capacity (Kahane 2012, Carlsen et al. 2013), and identify policy recommendations for sustainable development (e.g., Cork et al. 2005, Bohensky et al. 2011a, b, Palomo et al. 2011) and adaptation pathways (Butler et al. 2014). PSP can elicit how stakeholders might respond to future challenges, hence contributing to the management and understanding of complexity in social-ecological systems.

Despite the increased application of PSP, as far as we know, there has been no comparison or review that assesses the multiple claims of PSP studies, e.g., social learning, innovation, or empowerment, and synthesizes the knowledge gathered. This limits the understanding of the applicability of different methods, and the strengths and weaknesses of different processes relative to different goals and contexts within PSP (van Vliet et al. 2012). Such an understanding is needed to improve the rigor, inclusiveness, and effectiveness of PSP, and to inform future practice as PSP becomes more common through its adoption by global initiatives such as the Intergovernmental Panel on Biodiversity and Ecosystem Services (IPBES).

METHODS

Case selection

This paper is based on a structured ex-post multiple-case enquiry (Yin 2009) and reflection by researchers who have been involved in place-based PSP in social-ecological systems as part of an action research project or as a participatory component of a larger research project. As a starting point, a core group of researchers from multiple disciplines and with relevant experience in qualitative scenario exercises initiated this synthesis activity at the “Resilience 2014: Resilience and Development: Mobilizing for Transformation” conference. A subsequent snowball search procedure among colleagues in the resilience and social-ecological systems research communities resulted in identifying 23 scenario cases that were included in our analysis (Table 1). Criteria for inclusion were discussed by the first coauthors to be involved in the process and were the following: (1) first-hand experiences would be contributed by the principal investigators of the scenario cases; (2) cases would feature place-based PSP addressing and linking social and ecological dimensions; and (3) major gradients in terms of geography, ecosystems, socioeconomics, and natural resource management would be covered. Our cases were thus selected through information-oriented sampling focused on maximizing variation, and are not necessarily representative of all PSP exercises recently conducted (Flyvbjerg 2006). Our systematic comparative analysis aimed at understanding the commonalities and differences in PSP exercises that have been conducted within diverse social-ecological systems.

Data collection and analysis

We developed an analytical framework for the analysis of the 23 PSP exercises. The cases included in this study were conducted between 2003 and 2014 (Table 1). This framework was tested on sample cases and reviewed by 18 authors who refined and translated it into a survey of 75 open and closed questions (Appendix 1) that were grouped into nine categories: (1) case details, e.g., basic information such as study title, name and role of contributor, references; (2) context and case identity, e.g., location, scale, ecological, socioeconomic, and governance context, type of stakeholders in the case study, and thematic focus; (3) the original motivation of the study and its objectives, e.g., main general aim and specific objectives; (4) methodological approach, e.g., background information and guidelines used, the process to identify drivers of change, the scenario design; (5) methodological process, e.g., stakeholders engagement, process stages, tools used, storyline types, etc.; (6) content of scenarios, e.g., storyline characteristics, consideration of ES, biodiversity, human well-being, trade-offs; (7) outputs, e.g., type of outputs, such as reports, drawings, collages, videos, etc.; (8) monitoring and evaluation, e.g., impacts of the exercise and if monitoring and evaluation phases were developed; and (9) lessons learned, e.g., main strengths and weaknesses of the process, key insights, and reflections. Two rounds of data collection took place to clarify responses and to incorporate additional questions arising from the first round.

The information from each of the aforementioned categories was analyzed by a subgroup of coauthors following a four-step process: (1) where applicable, responses were coded into pre-existing or emergent typologies; (2) the diversity of the responses to each question was summarized, including notable outliers; (3) particularly strong trends, dominant approaches, common findings or lessons were noted; and (4) descriptive and multivariate analyses were performed. Multiple correspondence analysis (MCA, the counterpart of Principal Component Analysis for large sets of categorical data) and Hierarchical Cluster Analysis (HCA) were applied to explore the linkages and associations between different variables and similarities between cases, respectively. To define the number of axes retained for the HCA, we employed two criteria: scree test (Cattell 1996) and eigenvalue, which determines the inclusion of MCA axes with inertia above 0.15 (Hair et al. 1998). We used Euclidean distance as the dissimilarity matrix coefficient and Ward’s method as clustering technique to minimize the error in sum of squares (Ward 1963). Clusters of case studies were then associated with original motivation for performing the study and lessons learned. Data were analysed with Excel (Microsoft Office) and Xlstat 2012 (Addinsoft) software.

RESULTS

Case context and identity

Geographical and temporal distribution

The case studies were located in 17 different countries and six continents (Fig. 1). Most cases were from Latin America (seven cases), closely followed by Europe (six cases), then North America (three cases), and Australia (three cases). Africa and Asia were represented by two case studies each. Case studies were most frequently located in the tropical and subtropical forest biomes (Fig. 1; Olson et al. 2001). Some of them were conducted in agroecosystems and others, such as the German, Kenyan, and South African cases, included urban and peri-urban areas. Although terrestrial settings prevail, three case studies also dealt with estuaries or coastal wetlands (#6, #13, and #22), tropical islands (#18, #19, and #20) and a tropical coral reef (#17). The end year of the PSP research projects ranged from 2003 to 2014 with most processes finishing between 2012 and 2014.

Scale and system boundaries

Almost half of the cases (11) were defined by political boundaries, e.g., municipality, district, province, or officially recognized community boundaries, while another 10 defined their boundaries according to natural features such as watersheds, coastal regions, or landscapes (Appendix 2). In sixteen of the cases, protected areas were included within the research area. Twenty one cases were developed at a local scale, e.g., communities, municipalities or subdistricts, and only six explicitly used a multilevel approach, i.e., included analysis at local, regional, national, and/or global scales.

Governance and institutional contexts

The vast majority of cases involved complex governance and institutional arrangements. The most prominent institutions participating were municipalities (22 cases), regional and national governmental institutions (20 cases); community councils, indigenous organizations, and tribal forms of organization (16 cases); and conservation groups, NGOs, comanagement groups, and natural resources regulatory agencies including park authorities (22 cases; Appendix 2). In eight cases, supranational governmental organizations, such as the European Union (through the Water Framework Directive and Common Agricultural Policy) and international trade agreements, like the North-America Free Trade Agreement, were also mentioned as influential. In 14 cases, large natural resource industries like fishing, mining, and palm oil industries were noted as key actors, even if not formally considered part of the environmental governance system. In the two Colombian cases (#5 and #13), criminal and guerrilla groups were also considered part of the governance system.

Economic contexts and livelihoods

In most cases (20) agriculture was the primary sector supporting local livelihoods (Appendix 2). The services sectors, including trade and tourism, were also important (18 cases), while extractive industries, such as fishing, mining, palm oil and timber plantations, were important in 11 cases. Nine cases dealt with subsistence economies or economies with a strong dependence on subsidies or remittances. In four cases, illegal economic activities, such as coca plantations or illegal timber extraction and mining, were an important part of the local economy.

Subject and objectives of the PSP exercise

The main subject of PSP included conservation, e.g., biodiversity, wildlife, and natural habitat protection, sustainable development pathways, and natural resource management (Appendix 3). Following van Notten et al. (2003), cases were classified according to the following (Fig. 2A):

1. Their goals: classified as (a) exploratory, i.e., creating scenarios to examine plausible drivers of change, (b) prepolicy or decision support, i.e., building scenarios to examine futures according to their desirability, or (c) both exploratory and prepolicy;
2. Their treatment of norms: classified as (d) descriptive, i.e., developing scenarios without considering researchers’ preferences, (e) normative, i.e., including researchers’ preferences or interests in scenario development, (f) both descriptive and normative;
3. The function of the scenario exercise: classified as (g) process-oriented to stimulate reflexivity, creative thinking, and raising awareness about the future of the social-ecological system, (h) product-oriented, i.e., leading scenarios to create an outcome, e.g., a set of narratives of plausible scenarios, or (i) both process- and product-oriented.

Four main reasons were given for the use of place-based PSP (Appendix 3): (1) to guide and support local stakeholders’ decision making by promoting reflection on likely impacts of future drivers of change in social-ecological systems (9 cases); (2) to generate social learning and knowledge integration among multiple stakeholders to find ways to respond to potential changes (6 cases); (3) to complement research projects by incorporating stakeholders’ views in the research process (5 cases); and (4) to raise local stakeholders’ awareness of future changes and to confront skepticism, e.g., about climate change (3 cases).

Methodological approach

Background information

All case studies collected background information (Appendix 4), often through desk research (13 cases) or in a participatory process, e.g., with workshops or focus groups (12 cases), for a range of purposes (Fig. 2B).

Type of scenario design

In 14 cases the scenarios were designed through stakeholder-driven approaches. In the remaining cases stakeholders participated in other stages of the PSP as explained below (Appendix 4). Twenty-one cases used a projected year, ranging from 2025 to 2090, although 2030 was the most commonly projected year (9 cases). The time span between the creation of the scenario and the projected year was, in most cases, between 10 and 20 years (14 cases).

About half of the cases (15) created four scenarios. A common motivation to the number of scenarios created was that it should be a manageable and feasible number for further discussion and deliberative purposes (9 cases).

Drivers of change

All but 2 cases identified drivers of change through participatory methods, particularly through workshops (17 cases), but also by way of interviews and surveys (9 cases; Appendix 4). Formal scientific knowledge from outside of the participatory process, e.g., previous research or predefined drivers by researchers, was also used to identify drivers in all cases. The majority of cases used alternative states of key drivers as the basis for the storylines. Among all the available reasons for using drivers of change in PSP, inspiring the creation of qualitative storylines was the most common (15 cases). A 2x2 matrix approach (e.g., Carpenter et al. 2006) was also quite common (10 cases), while only 4 cases used drivers to derive formal models.

The number of drivers of change varied widely across the cases (from 2 to 392), but most commonly 10 or fewer drivers of change were identified (10 cases). The process to prioritize drivers, once they had been identified, was usually by ranking (10 cases), based on their impact, probability of influence, importance, and relevance for a given social-ecological system (SES). The majority of drivers identified were related to social issues, e.g., demographics, governance, economics, market conditions.

Process

Duration of the scenario planning process

The duration of the complete process varied from 2 to 60 months (median 12, average 16 months), with between 1 and 18 workshops (median 3, average 5 workshops) lasting between half a day and 4 days (median of 1 day, average of 1 and a half days).

Engagement of participants and facilitators

In 19 cases a research team identified stakeholders jointly with (or with significant input from) local stakeholders (Appendix 5). In some cases specific stakeholder identification methods were used, including stakeholder analysis and mapping techniques (12 cases), such as the 2-axis importance/relevance and interest/concern tool (e.g., Reed et al. 2009), social network analysis (2 cases), and/or snowball sampling (4 cases).

The total number of participants involved in the cases ranged from 14 to 167, with an average of 52 (median 50), although the average number of participants per workshop was 26 (median 22; Fig.2C). The diversity of stakeholder groups considered in the PSP exercises ranged from only 1 group to 7 different groups (Fig. 3). Almost all cases involved the local community, local policy makers, natural resources management agencies, and nongovernmental organizations (NGOs). Other commonly involved groups included supra-local policy makers, academics, and representatives from the business and recreation sector (Fig. 2D).

Workshops were typically facilitated by 4 or 5 facilitators, entailing an average 1:2 facilitator–participant ratio. In 21 of the studies, facilitators came from their own research team, sometimes after facilitation training (14 cases) and often with previous experience in future scenarios workshops (10 cases). Only 4 cases used independently contracted facilitators.

In most cases, the researchers had prior knowledge of the participants, either through research team members who were local to the study region or because of previous engagement with stakeholders. Conflicts sometimes emerged during the participatory process (7 cases), mostly between participants with different views but also between participants and researchers (1 case) and between funders and researchers (1 case).

In almost all cases (19) participants collaborated in the envisioning process, i.e., imagining drivers interacting to form future events, and the identification or selection of guidelines or drivers (18 cases, Appendix 5). Participatory methods/process design, i.e., the design of the methods/process itself, took place in 11 cases. Eleven cases also received feedback and comments from participants. In roughly one-third of cases (7) participants were involved in the back-casting, i.e., analyzing how desirable future outcomes can be reached for long-term complex issues (Dreborg 1996, Carlsson-Kanyama et al. 2008), and a similar number did participatory modeling (6 cases).

Methodological tools applied

A wide range of tools and techniques were used to support PSP. Group discussions were implemented in all cases, often in small groups (17 cases; Appendix 5). Other common tools included individual reflections (11 cases), drawings (11), capturing ideas on post-its and index cards (10), mental models (9), quantitative models or data about climate change or land-use change (9), rankings of different issues (8), interviews (8), and maps (6). Less common tools (5 cases) included collages, stock-flow diagrams, wall-mounted time-lines, fictional newspaper headlines, and storytelling.

Storylines were elaborated in almost all case studies through a combination of methods. Storylines were developed by participants (10 cases) or the research team (8 cases) and the storylines were spatially explicit, at least partly, in 10 of the cases.

Type of data analysis

In all cases, the research team analyzed data using qualitative analysis, through descriptive analysis and narrative development, while just under half the cases also carried out quantitative analysis. Quantitative analysis focused on assessing, and sometimes modeling, ES trends (e.g., #22), human well-being trends (e.g., #12), tendency of drivers of change (e.g., #8), as well as the analysis of policy responses (e.g., #10).

Presentation of results

Some case studies (11) presented the results of the PSP in a separate workshop with this specific aim while others presented results within the same workshop (4 cases; Appendix 5). Most cases performed some kind of validation or plausibility check of the scenarios, either in workshops (9 cases), by commenting on the scenario (7 cases), or within larger meetings that other stakeholders attended (4 cases). In 7 cases a draft of scenarios was sent to particular stakeholders to receive comments for validation.

Uncertainty and vulnerability

Uncertainty is inherent to scenario planning but only 16 cases mentioned it explicitly during the PSP, usually in the analysis of drivers of change. Vulnerability was explicitly analyzed in 14 cases, through the analysis of ES trends (10 cases), stakeholders’ vulnerability (5 cases), and in some cases specifically through vulnerability with regards to food security (7 cases).

Content of scenarios

Guidelines and scenario names

To aid in developing the scenarios, most cases (18) provided participants with guidelines and 12 cases used focal issues and drivers. The Millennium Ecosystem Assessment (2005) and MedAction (Kok et al. 2007) were sources of inspiration in 4 cases (Appendix 6).

Titles of the scenarios were chosen by researchers or by participants. Fifteen cases had four scenario names, ranging from the commonly used best future to business as usual (BAU), and one or two somewhere in between. Examples of names given by participants were: “Doom and Gloom,” “A Confused State,” “Slow Boil,” and “New Mombasa.” Some examples of scenario names given by researchers were: “Privatization and Urban Solutions: Don’t stop me now,” “Rising Fences: Another one bites the dust,” “Market forces: reallocation of resources,” “Less is more,” “Chaos,” “Grand transitions: a new paradigm of sustainability,” “Rural-urban migration,” “A double-edge,” “Back-to-the-future: Transhumance moves,” “Our land, their wealth,” “Balance brings beauty,” “Enjoyment Brings Misery,” “Smiling West,” “Shaky future,” “What’s Ours is Yours,” “Adaptive Doñana - Wet and Creative.”

Variation in the scenarios’ content

The content of scenarios in 8 of the cases varied according to mixes of 2 main factors (related to the drivers) in each scenario. For example, case study #7 presented 10 scenarios that mixed various extents of intensive land use vs. management for a range of other ES, with landscape planning and management being the key issues addressed. Case study #8 presented 4 scenarios that mixed various extents of traditional land use vs. population growth and development, with forest conservation being the key issue addressed. Although the case studies that presented scenarios based on mixtures of two main factors were highly diverse, they were all essentially variants of conservation or sustainable management vs. unfettered growth or industrialization/mining, with a range of issues being addressed throughout the variations (Appendix 6 [8 i]). Seven cases included scenarios that varied their content according to mixes of 3 main factors in each scenario. For example, case study #11 presented 2 scenarios with a mixture of varying extents of real estate development vs. agricultural intensification vs. habitat conservation, with biodiversity being the key issue addressed (Appendix 6 [8 ii]). Half of the cases using mixes of 3 main factors introduced a contrast between locally-driven vs. globally/externally-driven (e.g., # 15, #21, and #23). Seven cases varied according to mixes of 4 or more main factors in each scenario (Appendix 6 [8 iii]) of which 4 introduced an explicit governance dimension (cases # 4, #10, #17, and #19).

Topics discussed in the PSP

Thirteen case studies discussed ES provision in the different scenarios (Fig. 2E, Appendix 6). When ES were explicitly addressed, the Millennium Ecosystem Assessment list of ES was sometimes provided to participants and certain ES were used in discussions and subsequent models. In the cases in which ES were not made explicit, certain ES appeared in the storylines. In other cases the scenarios were created around values, e.g., intrinsic value of nature, rather than ES.

Almost all cases (21) discussed biodiversity topics (Fig. 2E, Appendix 6), either explicitly, e.g., through a model output for biodiversity (or involvement of stakeholders that explicitly represent biodiversity), or only implicitly, e.g., through biodiversity related drivers that were discussed in the context of the SES.

All except two cases discussed human well-being via one or more variables (Fig. 2E, Appendix 6). When human well-being was made explicit, it commonly focused on livelihoods. Otherwise, poverty alleviation, social development goals, or employment were mentioned.

All cases dealt with trade-offs among different social-ecological components, though in different ways (Fig. 2E, Appendix 6). The explicit trade-offs tended to be between winners and losers in relation to the use of ES, between development and well-being, between scenarios, or between ES and human well-being. When trade-offs were not addressed explicitly, they featured strongly in the narratives and emerged during discussions.

Outputs

Types of outputs

The majority of cases (20) produced creative or artistic outputs (Fig. 2F, Appendix 7) such as collages, drawings, or illustrations to visualize the scenarios and facilitate the PSP process (Fig. 4). Illustrations, for example, included timeline illustrations, colorful drawings depicting scenarios, cartoons, and oil on canvas paintings. In one case, the process of creating collages (#22) activated the groups and allowed other people that were less willing to speak, to participate “in another language.”

Besides artistic outputs, a wide variety of outreach material was produced in the case studies including posters (15 cases; Fig.4C,D), reports (23), scientific journal articles and books (21), leaflets (5; Fig.4A), postcards (5; Fig.4B), and videos (10; Fig. 2F). Other outputs mentioned were cartoons, animations, game boards, newspaper articles, radio interviews, a TV show, and a children’s book.

Process and target audience for outputs

All of the PSP study cases produced outputs to communicate the results of the scenario project to different audiences, especially local communities (19 cases), academic audiences (16 cases), participants (15 cases), and policy and decision makers (15 cases; Appendix 7). In addition to developing outputs for communication purposes, the creative process itself offered alternative ways to engage with stakeholders. One case (#17) used the scenario outputs to inform later interviews with a different set of stakeholders and at another scale.

Monitoring and evaluation

Monitoring of PSP impacts

Monitoring of PSP impacts, i.e., systematic collection of data to track the extent of progress and achievement of outcomes and impacts using indicators (Appendix 8), was performed in 11 cases (Appendix 9), either solely within the project timeframe (8 cases) or also extending beyond the project timeframe in three cases, which were led by the same research team. An equal number of cases identified their reasons for monitoring as a contractual obligation, to assess learning, or to assess outcomes.

In about half of cases monitoring was impossible because of constraints of time, personnel, or finances (11 cases; Appendix 9). In two cases (#3 and #21) monitoring was not necessary or important to the goals of the PSP. Two cases (#14 and #23) found it impractical to monitor because it was too early or because detecting impacts seemed intractable.

Evaluation

Evaluation, i.e., assessment of the scenario design, implementation, and results through a formal methodological approach, was conducted in 15 cases by a range of different methods including interviews (9 cases), surveys (8 cases), and observation (4 cases; Appendix 9). As with monitoring, resource constraints were the main reason for the lack of evaluation of the scenario planning exercise in 9 cases. In 5 case studies it was too soon to evaluate the effects of scenario planning.

Assessing participants’ learning was the top reason for conducting evaluations (6 cases) followed by assessing the usefulness of the process, and providing feedback to the research team. Note that these two objectives were inter-related, i.e., assessment of process can also include an assessment of social learning. In at least three cases, evaluation was intentionally addressed to both (I. Brown, J. Martin-Ortega, K. Waylen, and K. Blackstock, unpublished manuscript).

Outcomes and impacts

The majority of cases (20) did not formally evaluate, and correspondingly did not detect evidence of outcomes or impacts (Appendix 9). However, in all these cases informal evaluations were undertaken. Strong and moderate evidence of short-term impacts was found in some cases that did not undertake a formal evaluation (6 cases), but strong evidence of long-term impacts was found exclusively in 2 cases (#18 and #19) undertaking a formal evaluation. There is strong evidence of either short- or long-term impacts in 7 cases, of which 5 are the formally evaluated case studies, suggesting that with structured evaluation processes the other case studies may have discovered that they had in fact generated more profound effects.

Strengths and weaknesses

The most commonly identified (21 cases) strengths of PSP processes were related to the added value of engaging stakeholders actively in the research process and to the technical and methodological advantages of developing participatory workshops to explore feasible futures (19 cases; Appendix 10). Among the weaknesses, the most frequently reported dealt with the technical development of the PSP processes (20 cases) and the quality of results (15 cases).

Stakeholders’ engagement

PSP’s strengths were generally attributed to their potential as a research tool to engage a wide diversity of stakeholders, i.e. women, men, young, old, local people, researchers, etc., in a knowledge sharing process that ultimately led to a shared understanding of the social-ecological system, its dynamics, and future management challenges. Such a process was referred to as social learning or mutual learning in 13 cases (Appendix 10). The creation of partnerships among different stakeholders, including researchers, was also identified as a strength in 11 cases. In addition, involving participants in the process raised awareness of local management challenges—overcoming initial skepticisms—and of the relevance of taking action in local planning (5 cases). In 4 cases participants’ engagement also led to an increase in social cohesion at the community level and involved community members who usually had been excluded from decision making, e.g., women, young people.

Nevertheless, some weaknesses in terms of stakeholder engagement were highlighted. The lack of diversity of stakeholders and the continuity of their involvement were recognized as constraints for the success of the participatory process (8 cases). Power relations between stakeholders are inherent in every SES so when the process fails to incorporate someone’s voice, these relations and inequities might be hidden. Eight cases in fact reported limitations due to the low representativeness or absence of powerful stakeholders, e.g., industry or big landowners, and decision makers that undermined the credibility of the process. Five cases reported biases due to the researchers’ authority and voice undermining ownership of the process by stakeholders as a weakness. In three cases (#2, #3, and #4) the absence of powerless actors and especially gender discrimination in participation were reported as weaknesses because of the potential underrepresentation of power asymmetries. In two cases (#8 and #16), cultural barriers relating to indigenous people were pointed out as explanatory factors of limited engagement.

Technical development

The methodological and technical design of the PSP was key in engaging stakeholders in the process. Facilitating discussions among stakeholders on the drivers of change in each scenario and how to respond to them was the strength most frequently perceived (9 cases). Other strengths mentioned in this sense were: (1) the adaptability and dynamism of the design and the use of multiple approaches during the workshops; (2) the adoption of a systematic and/or interdisciplinary approach; (3) the exploration of comprehensive drivers, trade-offs (winners and losers) and values; and (4) the previous training of facilitators in scenario exercises.

However, some of the constraints for the success of PSP were related to the methods and tools used. In 11 cases PSP was recognized as expensive, not only in economic terms, but also in terms of time and energy consumption. Nine cases cited the lack of quantitative information, statistical and data-based testing, or modeling to support trends analysis as weaknesses. Five cases reported as a relevant weakness the unavoidable trade-off between the accuracy requested by the science base, which includes high complexity of scientific information, versus the social relevance of the process. In fact, some authors recognized that the methodological choices sometimes reflected the research purposes rather than the social learning objectives. In a few cases, linguistic and cultural barriers (3 cases) as well as logistic and facilitation problems (6 cases) hampered the process.

Outcomes

Nine cases highlighted the strong policy relevance of the findings and outcomes, because scenarios were used to discuss and guide implementation of potential adaptation strategies. In 7 cases the inclusion of a diversity of worldviews in the results was mentioned as a strong direct added value of PSP. By contrast, in 5 cases, authors reported that the preferences, cultural attitudes, or background of some participants or researchers might have biased the understanding of drivers, e.g., farmers’ belief in fate’s role shaping their daily life, and the way of thinking about the future, e.g., indigenous understanding of time and the future. The substantive results of the scenario analysis was perceived as too polarized in 3 cases and/or repetitive and limiting creativity and novelty in 4 cases because of the excess of guidance by researchers. A poor incorporation of drivers of change or indicators, e.g., for well-being analysis, were recognized as key weaknesses in 5 cases.

Similarities among case studies and associations between objectives, methods, and lessons learned

Variables associated with methodological procedures were selected for the MCA (for the definition of the variables see Appendix 1). The first 3 axes presented an inertia above 0.15 and together explained 69.0% of the total variance (F1: 50.2%; F2: 12.2%, F3: 6.6%; Appendix 11). The HCA of these 3 axes identified 4 groups of PSP studies, characterized by the particular techniques and methods used (Fig. 5). Cluster 1 corresponded to those case studies that performed desirability and vulnerability analysis, variables that are basically associated with negative scores of F1 and positive scores of F3, respectively. Cluster 2 is characterized by those PSP exercises that identified stakeholders and drivers of change before workshops, and developed back-casting during the participatory process. Cluster 2 is associated with positive scores of F1. Cluster 3 comprises those case studies that identified direct drivers of change prior to PSP and explicitly included uncertainty, being associated with positive scores of F2. Finally cluster 4 is characterized by case studies that used modeling as a quantitative technique after the workshop and monitoring processes, being associated with negative scores of F3. The abovementioned characteristics of each cluster are, however, not exclusive of the cases that are grouped under that cluster.

These results seem to indicate a connection between motivations for performing PSP, specific methods used, and lessons learned in terms of learning process, stakeholder relationships fostered, and management outcomes (Figs. 5 and 6). For example, cases of cluster 1 explicitly analyzed vulnerability to broaden the thinking of social actors about social-ecological systems and they also identified the stimulation of creative and complex thinking as a strength. Cases of cluster 2, through performing back-casting, aimed to understand the social and institutional mechanisms behind management decisions and they recognized insights for landscape management as a positive outcome. Cases of cluster 3 that explicitly incorporated uncertainty aimed to promote community-based solutions and recognized as a positive outcome to have engaged social actors that are unrepresented in decision making. Finally, cases of cluster 4 aimed to facilitate sharing experiences among stakeholders in a creative and collaborative way. In this cluster, a complex understanding of the current situation and the colearning process between scientists and nonacademic stakeholders were highlighted by researchers as positive outcomes.

DISCUSSION: LESSONS LEARNED

Across the diversity of PSP cases reviewed in this paper and the experiences of the involved researchers, three main questions were addressed: How was PSP useful to participants and researchers? How did PSP contribute to decision-making? And what are common methodological challenges for PSP?

How was PSP useful to participants and researchers?

This review demonstrates that PSP almost always has a process function that promotes stakeholders’ active engagement in place-based social-ecological research that is or can be linked to environmental decisions. Stakeholders’ engagement in this type of research is beneficial because it contributes toward improving the equity, legitimacy, and quality of environmental decision making. Involving stakeholders in the research process through place-based PSP provides voice to multiple perspectives on social-ecological futures (Ravera et al. 2011a, Reed et al. 2013a, Mistry et al. 2014), which can potentially reduce power asymmetries and provide more equitable decision making. By including stakeholder responses in scenarios and across scenarios, PSP can also potentially increase the legitimacy and acceptance of policy options across stakeholders involved in a process (e.g., Peterson et al. 2003b, Bohensky et al. 2011a,b, Ravera et al. 2011a). Further, by including knowledge and information from a diversity of sources the quality of scenarios and identified policy options can be increased (e.g., Hill et al. 2010, Palomo et al. 2011, Ravera et al. 2011a, Vilardy Quiroga et al. 2011, Martin-Ortega et al. 2014), and innovative strategies and opportunities for collaboration among multiple stakeholders can be identified (Butler et al. 2015).

Many of the examined cases demonstrate how PSP processes succeeded in increasing dialogue, resolving conflicts, producing outputs that otherwise were not possible, and enhancing multiple learning outcomes between stakeholders, researchers, and policy makers in natural resource management planning (e.g., Ravera et al. 2011a, Hamann et al. 2012, Oteros-Rozas et al. 2013, Plieninger et al. 2013, Martin-Ortega et al. 2014; Fig. 6).

The scenario processes increased stakeholders’ awareness of the existence of local and global drivers of change and threats, and the need for long-term planning to deal with such changes (I. Brown, J. Martin-Ortega, K. Waylen, and K. Blackstock, unpublished manuscript). The scenario processes enabled collective reflections and discussions of potential policy options to deal with current and future environmental and socioeconomic changes in SES. By enabling discussions and creating shared understanding, PSP can further facilitate mobilization of stakeholders to respond to newly identified threats or opportunities. New partnerships among actors might also be created or reinforced and new leaders emerge to address new issues of interest (Plieninger et al. 2013).

Finally, PSP can encourage complexity thinking, i.e., clusters 1 and 4 (e.g., Ravera et al. 2011a, Waylen et al. 2015), which is a key aspect of resilience (Biggs et al. 2015). By requiring participants to reflect upon and characterize their SES’s internal dynamics, as well as how the SES interacts with external processes, the PSP enhances participants’ social-ecological understanding, and integrates their qualitative, context-specific local knowledge of the system. Scenarios also engage participants in embracing uncertainty, surprises, and contradictions (e.g., Oteros-Rozas et al. 2013, Butler et al. 2014, Martin-Ortega et al. 2014). However, nearly half of the cases did not explicitly address uncertainty during the PSP. Greater attention to this aspect could enhance participants’ learning.

PSP content and outcomes contributing to decision making

By bridging multiple knowledge systems PSP can bring together and produce new knowledge for environmental decision making. PSP can enhance the ability of environmental decision making to engage with complexity. In our review the two dominant ways this occurred was first by exploring complex social-ecological trade-offs, and second by creating novel solutions.

PSP has proved to be an arena where multiple knowledge systems interact (e.g., Palomo et al. 2011, Ravera et al. 2011b, Oteros-Rozas et al. 2013, Reed et al. 2013a) to cocreate a new understanding of the present situation and shared visions of possible future developments. PSP can provide a platform that supports stakeholders from different knowledge-systems by enabling communication and interaction to coproduce synthetic social-ecological knowledge as well as codesign new environmental management strategies (Martín-López and Montes 2015). The new Intergovernmental Platform of Biodiversity and Ecosystem Services (IPBES) plans to bring together different knowledge systems in its global and regional assessments to coproduce knowledge and design management strategies to face the challenge of biodiversity and ecosystem services conservation (Tengö et al. 2014, Díaz et al. 2015). A participatory and interdisciplinary research process such as PSP can be seen as a parallel research process, helpful to complement and strengthen existing research based on nonparticipatory methods (Peterson et al. 2003a). Although, it is noticeable that PSP is a useful tool to explicitly combine local or traditional knowledge with technical knowledge (i.e., cluster 2; Fig. 6), greater attention to nonformal and indigenous governance may assist in effectively utilizing opportunities to engage multiple knowledge systems (Hill et al. 2012). This might be particularly important in regions of the world that are underrepresented within the group of cases assessed here, such as Africa and Asia.

PSP studies typically go beyond simplistic win-win assumptions (Daw et al. 2015). Rather, they acknowledge the multiplicity of ES, for instance, by explicitly considering the trade-offs around them. Trade-offs occur when the provision of one ES is reduced as a consequence of increased use of another service (Rodríguez et al. 2006), or because of certain practices or management techniques that enhance one ES while another one is decreased. They occur along various dimensions (Mouchet et al. 2014): (1) supply-supply, i.e., conflicts between simultaneously provided ES; (2) supply-demand, i.e., spatial or temporal lags between ES supply and social benefits; and (3) demand-demand, i.e., arbitration between different and divergent stakeholders’ interests. In most of the cases reviewed here, a particular focus was set on different stakeholder groups that would benefit or lose from trends in ES supply in the respective scenarios, i.e., on demand-demand trade-offs. By this, PSP may foster the awareness for visible and invisible social conflicts and power relations around ES, which is an underdeveloped field in ES research (Sikor 2013). The inclusion of an explicit governance dimension in about half of our cases supports the usefulness of PSP to address key aspects of governance such as the influence of local vs. global-drivers of change; centralized government vs. collaborative governance; fragmented weak governance with and without innovators; and community vs. neoliberal orientations.

Another strength of PSP is that the participatory processes bring the research closer to a complex reality to support adaptive governance (Waylen et al. 2015.), as well as creativity, which is fundamental to promote resilience (Berkes et al. 2003; see clusters 1 and 4, Fig. 6). On one hand, PSP leads to a focus on plausible futures to discuss concrete actions, strategies, and policy options according to both scientific information, local knowledge, and stakeholders’ perceptions of SES and its dynamics (Daw et al. 2015). On the other hand, PSP outputs, for example in the form of images, video, and storylines, are also attractive and useful tools to engage wider sections of society, as well as to invite reflections about the future from the public (Sheppard et al. 2011). Both pragmatism and creativity are fundamental to support adaptive governance and to promote resilience (Garmestani and Benson 2013). PSP’s capability to bring governance discussion and learning to the fore is useful given the recognition that governance is both a key determinant of humanity’s ability to respond to environmental change, and very challenging for a wide range of stakeholders to understand and incorporate in their analyses (Simon and Schiemer 2015). Furthermore, PSP provides data on locally perceived changes and impacts of possible futures that are useful in achieving a better and holistic understanding of the current, and future system’s conditions and dynamics at local and regional spatial and political scales (Butler et al. 2014).

Challenges and opportunities

Our review identified four widely shared challenges in conducting PSP. The first is the tension between explorative and normative analysis. The second is navigating conflict among diverse unequal stakeholders. Third is the challenge of communicating with a diverse group, and fourth the challenge of assessing impact.

PSP processes usually contain an inherent tension between explorative and normative analysis of SES dynamics. In our review, although we found that the most reported approach to PSP was strictly explorative (Fig. 2A), many of the scenario names suggest that normative judgments were important. Carpenter et al. (2006) follow much scenario practice (Wack 1985, van der Heijden 2000) in arguing that scenario planning is most powerful when a small set of scenarios explore clear and striking differences. Normative scenarios are distinctive in their portrayal of futures that “should be” (e.g., Opdam et al. 2001) and they can inspire policy by providing images of landscapes that could meet societal goals (Nassauer and Corry 2004). Value judgments clearly have a role in generating the vivid and distinct choices that Carpenter et al. (2006) advocate, and our analysis suggests that it would be helpful to more explicitly discuss and present these value-choices in the scenario generation. This is particularly important because most scenarios conducted here were funded and conducted as sustainability science projects that are explicitly not value neutral but prosustainability, and consequently have specific normative frameworks that are assumed rather than articulated (Abson et al. 2014). Articulating values is important because it enables them to be discussed and used in deliberation or comparison of alternatives. However, value-laden discussions are often emotionally charged and require substantial efforts to manage in an effective participatory process.

The diversity of stakeholders and their inherent power dynamics within a PSP process can also present challenges and requires substantial investment in facilitation (Butler et al. 2015). In these case studies PSP has usually been built upon previous research within the study region that has identified multiple actors shaping and impacted by the region’s dynamics, which may explain the high diversity of stakeholders considered (Fig. 2D; Kok et al. 2007). However, even if stakeholders identified as relevant in the SES usually match the actors involved in the PSP, some frequently remain absent, particularly industry representatives and indigenous people, hence possibly misrepresenting power relations that can be important within the SES dynamics. Therefore, if the aim is to coconstruct future scenarios and share the pros and cons of each of them among the stakeholders involved, to conduct a systematic identification of stakeholders relevant to the SES and matching those with actors invited to the PSP is highly recommended. In addition, the high diversity of stakeholders necessary for inclusive participatory processes can trigger the appearance of social conflicts.

Communicating PSP results is another challenge shared across scenarios. Because of the requirement of engaging with a diverse set of stakeholders, communication requires careful thought and substantial effort. We recommend different types of outputs, from the common scientific outputs, i.e., papers and technical reports that pursue the academic audience and environmental and development technicians, respectively, to those outputs that combine the arts and science, such as posters, drawings, illustrations, or videos (Fig. 2F). For example, in “The role of visual arts as a communication tool in scenario planning” session performed at the Resilience 2014 conference in Montpellier (for more details, see http://ideas4sustainability.wordpress.com/2014/05/08/the-role-of-visual-arts-as-a-communication-tool-in-scenario-planning/), it was highlighted that artwork not only served as a tool for communicating PSP results, but also as a tool for facilitating communication among different stakeholder groups during the PSP process and afterward. However, the role of art in PSP could be further explored and the results assessed.

Although a goal of PSP is to promote action, it is challenging to produce evidence that PSPs have actually led to management actions, new partnerships and collaborations between stakeholders, or social learning processes. This gap exists both because identifying the impact of interventions is difficult and our sampling strategy within the 23 case studies might not have been sufficient to record all outcomes, but also because monitoring and evaluation stages were largely missing in the cases we assessed. The extent to which scenarios achieve outcomes is highly variable and often unknown because of a lack of formal mechanisms to evaluate outcomes (Fazey et al. 2014) and to the potential time lag between the end of the exercise and the delivery of certain outcomes. Thus, broad claims of attribution between PSP and impacts cannot be clearly substantiated. Adopting an explicit adaptive management approach (Peterson et al. 2003a) or articulating a theory of change (Butler et al., in press) might assist with embedding PSP within larger and longer term projects that may help researchers to plan their projects and then formally evaluate their outcomes and impacts. This would also facilitate the comparison and contrast between experiences, and would therefore enhance the opportunity to learn from and refine PSP methods. Particularly, systematic long-term monitoring and evaluation of PSP in other studies has shown that this approach can generate social innovation, collective action, and encourage transitions to sustainability (Butler et al. 2015). Comparative studies that allow for an assessment of impacts as well as the pros and cons of different methods within PSP to develop scenario quality criteria are therefore needed (van Vliet et al. 2012). Project timescales and budgets need to allow for evaluation and monitoring.

Future of participatory scenario planning

Participatory social-ecological scenario planning is increasingly used to explore ecosystem services in alternative futures. Furthermore, given the expectation that IPBES will produce a variety of global, regional, and local biodiversity and ecosystem service assessments, its practice can be expected to increase further. Although such scenarios enable diverse and qualitative knowledge about ecosystem services to be combined with quantitative models, it is currently difficult to compare and build upon specific scenario processes because they are wedded to particular people, times, and places. Based on this review we believe that there are a number of practical guidelines which could promote good practice for PSP and its practitioners.

As discussed, conducting participatory social-ecological scenarios is challenging, time consuming, and requires integrating diverse types of knowledge. The success of PSP processes can be increased by recognizing the challenges associated with them and planning accordingly. Consequently, PSP processes should be designed for multiple iterations that maintain focus, but use multiple methods and approaches to elect and reflect people’s definitions of system and theories of change. One of the ways of increasing the efficiency and policy relevance of this process is to build upon existing work, both in terms of future visions contained in official documents, other scenario processes, existing social-ecological networks, as well as existing ways that diverse stakeholders are connected to one another, through policy networks, NGOs, governments, education, or other social institutions. Although all PSP processes should learn from previous work when starting a new project, processes need to be planned for the particular social-ecological context in which it is occurring and be based on reflections about the potential consequences of every phase of the process for the participants and SES (Martín-López and Montes 2015).

We believe that the practice of PSP would be improved by building a community of practice that uses a portfolio of common methods, addresses shared issues, and shares results, methods, and challenges in a comparative way to improve the ability of PSP to bridge across scales and cases. The field of PSP is emergent, and connects many diverse actors across, within, and outside of academia. Building such a community of practice should enable access to tools, ideas, and people. As such, PSP researchers should work on making their methods and results accessible, open access, and nontechnical, but also be aware of other efforts that take a PSP approach. This paper is a step toward building such a community of practice, and we hope that both scientists and the larger IPBES community can act to promote the knowledge sharing, training, and translation that are needed to develop such a community.

CONCLUSION

PSP is an increasingly used approach in place-based social-ecological research, and has been applied with a wide diversity of methodological approaches, processes, outcomes, and outputs. Across the 23 case studies assessed here, PSP enhanced stakeholder engagement and supported the diversity, equity, and legitimacy of environmental decision making. PSP also improved the quality of dialogue among stakeholders with complementary types of knowledge and has the potential to support creativity and social innovation. PSP also created new local understanding of the impacts of global and local environmental change that has the potential to lead to new partnerships among stakeholders. Finally, PSP also enhanced complexity thinking among participants, especially the ability to embrace uncertainty, surprise, and contradictions. In addition, the scenarios produced by PSP can be disseminated to trigger engagement and reflection among the wider public.

However, despite these benefits PSP is time consuming and subject to particular challenges. First, balancing the normative and explorative aspects of PSP requires careful reflection of what values are being promoted or suppressed. Second, systematic short-term process combined with evaluation and long-term monitoring of impacts is often difficult because people and resources are rarely available for long term commitment. Third, the design of a PSP process needs to fit scientific goals as well as the local social-ecological context, the different types of knowledge, and the way they are integrated.

Participatory social-ecological scenarios are increasingly used to explore ecosystem services in alternative futures. Based on this review of cases, we believe that this method has enriched environmental management and improved scientific understanding. To improve the future success of PSP, including those evolving within IPBES, we suggest that scientists and practitioners engaged in PSP should be more self-aware and build a community of practice to improve the quality of individual PSP processes, as well as provide a platform for diverse, new groups of people to conduct PSP processes that build on and improve current methods, tools, and processes. We hope that this comparative assessment is a first step toward building such a community.

LITERATURE CITED

Abson, D. J., H. von Wehrden, S. Baumgärtner, J. Fischer, J. Hanspach, W. Härdtle, H. Heinrichs, A. M. Klein, D. J. Lang, P. Martens, and D. Walmsley. 2014. Ecosystem services as a boundary object for sustainability. Ecological Economics 103:29-37. http://dx.doi.org/10.1016/j.ecolecon.2014.04.012

Beach, D. M., and D. A. Clark. 2015. Scenario planning during rapid ecological change: lessons and perspectives from workshops with southwest Yukon wildlife managers. Ecology and Society 20(1):61. http://dx.doi.org/10.5751/es-07379-200161

Bennett, E. M., S. R. Carpenter, G. D. Peterson, G. S. Cumming, M. Zurek, and P. Pingali. 2003. Why global scenarios need ecology. Frontiers in Ecology and the Environment 1:322-329. http://dx.doi.org/10.1890/1540-9295(2003)001[0322:WGSNE]2.0.CO;2

Berkes, F., J. Colding, and C. Folke, editors. 2003. Navigating social-ecological systems: building resilience for complexity and change. Cambridge University Press, Cambridge, UK. http://dx.doi.org/10.1017/cbo9780511541957

Biggs, R., C. Rhode, S. Archibald, L. M. Kunene, S. S. Mutanga, N. Nkuna, P. O. Ocholla, and L. J. Phadima. 2015. Strategies for managing complex social-ecological systems in the face of uncertainty: examples from South Africa and beyond. Uncertainty 20(1):52. http://dx.doi.org/10.5751/es-07380-200152

Bohensky, E., J. R. A. Butler, R. Costanza, I. Bohnet, A. Delisle, K. Fabricius, M. Gooch, I. Kubiszewski, G. Lukacs, P. Pert, and E. Wolanski. 2011b. Future makers or future takers? A scenario analysis of climate change and the Great Barrier Reef. Global Environmental Change 21(3):876-893. http://dx.doi.org/10.1016/j.gloenvcha.2011.03.009

Bohensky, E. L., J. R. A. Butler, and D. Mitchell. 2011a. Scenarios for knowledge integration: exploring ecotourism futures in Milne Bay, Papua New Guinea. Journal of Marine Biology Article ID 504651. http://dx.doi.org/10.1155/2011/504651

Bohensky, E, J. R. A. Butler, J. Rainbird, T. Skewes, V. McGrath, F. Nai, Y. Maru, F. Morseu, and A. Lankester. 2014b. Erub yesterday, today and tomorrow: community future scenarios and adaptation strategies. National Environmental Research Program. Reef and Rainforest Research Centre Limited, Cairns, Australia.

Bohensky, E, J. R. A. Butler, J. Rainbird, T. Skewes, V. McGrath, F. Nai, Y. Maru, F. Morseu, and A. Lankester. 2014a. Mabuiag yesterday, today and tomorrow: community future scenarios and adaptation strategies. National Environmental Research Program. Reef and Rainforest Research Centre Limited, Cairns, Australia.

Bohnet, I., and D. M. Smith. 2007. Planning future landscapes in the Wet Tropics of Australia: a social-ecological framework. Landscape and Urban Planning 80(1-2):137-152. http://dx.doi.org/10.1016/j.landurbplan.2006.07.001

Bohnet, I. C. 2010. Integrating social and ecological knowledge for planning sustainable land- and sea-scapes: experiences from the Great Barrier Reef region, Australia. Landscape Ecology 25(8):1201-1218. http://dx.doi.org/10.1007/s10980-010-9504-z

Bradfield, R., G. Wright, G. Burt, G. Cairns, and K. Van Der Heijden. 2005. The origins and evolution of scenario techniques in long range business planning. Futures 37(8):795-812. http://dx.doi.org/10.1016/j.futures.2005.01.003

Brown, K., W. N. Adger, E. Tomkins, P. Bacon, D. Shim, and K. Young. 2001. Trade-off analysis for marine-protected area management. Ecological Economics 37:417-434. http://dx.doi.org/10.1016/S0921-8009(00)00293-7

Butler, J. R. A., E. Bohensky, T. Skewes, Y. Maru, C. Hunter, S. Busilacchi, W. Rochester, J. Johnson, and J. Doupe. 2012e. Torres Strait futures: regional stakeholders’ future scenarios and livelihood adaptation strategies. National Environmental Research Program. Reef and Rainforest Research Centre Limited, Cairns, Australia.

Butler, J. R. A., T. Handayani, P. Habibi, T. Skewes, P. Kisman, and E. Bohensky. 2012a. Kecamatan Janapria case study scenario planning workshop report. AusAID-CSIRO Research for Development Alliance. CSIRO Climate Adaptation Flagship, Brisbane.

Butler, J. R. A., T. Handayani, P. Habibi, T. Skewes, P. Kisman, and M. Putranta. 2011. Nusa Tenggara Barat Province scenario planning workshop report. AusAID-CSIRO Research for Development Alliance. CSIRO Climate Adaptation Flagship, Brisbane, Australia.

Butler, J. R. A., J. Rainbird, T. Skewes, V. McGrath, F. Nai, E. Bohensky, Y. Maru, and F. Morseu. 2013. Masig yesterday, today and tomorrow: community future scenarios and adaptation strategies. National Environmental Research Program. Reef and Rainforest Research Centre Limited, Cairns, Australia.

Butler, J. R. A., T. Skewes, R. Wise, E. Bohensky, N. Peterson, N. Bou, and B. Masike-Liri. 2012c. Bali-Witu Local Level Government Futures Workshop Report. The Coral Triangle Initiative on Coral Reefs, Fisheries and Food Security. CSIRO Climate Adaptation Flagship, Canberra, Australia.

Butler, J. R. A., T. Skewes, R. Wise, E. Bohensky, N. Peterson, N. Bou, and B. Masike-Liri. 2012d. Hoskins local level government futures workshop report. The Coral Triangle Initiative on Coral Reefs, Fisheries and Food Security. CSIRO Climate Adaptation Flagship, Canberra, Australia.

Butler, J. R. A., T. Skewes, R. Wise, E. Bohensky, N. Peterson, N. Bou, and B. Masike-Liri. 2012b. West New Britain futures workshop report. The Coral Triangle Initiative on Coral Reefs, Fisheries and Food Security. CSIRO Climate Adaptation Flagship, Canberra, Australia.

Butler, J. R. A., W. Suadnya, K. Puspadi, Y. Sutaryono, R. M. Wise, T. D. Skewes, D. Kirono, E. L. Bohensky, T. Handayani, P. Habibi, M. Kisman, I. Suharto, Hanartani, S. Supartarningsih, A. Ripaldi, A. Fachry, Y. Yanuartati, G. Abbas, K. Duggan, and A. Ash. 2014. Framing the application of adaptation pathways for rural livelihoods and global change in eastern Indonesian islands. Global Environmental Change 28:368-382. http://dx.doi.org/10.1016/j.gloenvcha.2013.12.004

Butler, J. R. A., I. W. Suadnya, Y. Yanuartati, S. Meharg, R. M. Wise, Y. Sutaryono, and K. Duggan. In press. Priming adaptation pathways through adaptive co-management: design and evaluation for developing countries. Climate Risk Management.

Butler, J. R. A., R. M. Wise, T. D. Skewes, E. L. Bohensky, N. Peterson, W. Suadnya, Y. Yanuartati, Y. Handayani, P. Habibi, K. Puspadi, N. Bou, D. Vaghelo, and W. Rochester. 2015. Integrating top-down and bottom-up adaptation planning to build adaptive capacity: a structured learning approach. Coastal Management 43:346-364. http://dx.doi.org/10.1080/08920753.2015.1046802

Carlsen, H., K. H. Dreborg, and P. Wikman-Svahn. 2013. Tailor-made scenario planning for local adaptation to climate change. Mitigation and Adaptation Strategies for Global Change 18(8):1239-1255. http://dx.doi.org/10.1007/s11027-012-9419-x

Carlsson-Kanyama, A., K. H. Dreborg, H. C. Moll, and D. Padovan. 2008. Participative backcasting: a tool for involving stakeholders in local sustainability planning. Futures 40:34-46. http://dx.doi.org/10.1016/j.futures.2007.06.001

Carpenter, S. R., E. M. Bennett, and G. D. Peterson. 2006. Scenarios for ecosystem services: an overview. Ecology and Society 11(1):29. [online] URL: http://www.ecologyandsociety.org/vol11/iss1/art29/

Carpenter, S. R., E. G. Booth, S. Gillon, C. J. Kucharik, S. Loheide, A. S. Mase, M. Motew, J. Qiu, A. R. Rissman, J. Seifert, E. Soylu, M. Turner, and C. B. Wardropper. 2015. Plausible futures of a social-ecological system: Yahara watershed, Wisconsin, USA. Ecology and Society 20(2):10. http://dx.doi.org/10.5751/es-07433-200210

Cattell, R. B. 1996. The meaning and strategic use of factor analysis. Pages 131-203 in R. B. Cattell, editor. Handbook of multivariate experimental psychology. Rand McNally, Chicago, IIIinois, USA. http://dx.doi.org/10.1007/978-1-4613-0893-5_4

Cork, S., G. Peterson, G. Petschel-Held, J. Alcamo, J. Alder, E. Bennett, E. Carr, D. Deane, G. Nelson, T. Ribeiro, C. Butler, E. Mendiondo, W. Oluoch-Kosura, and M. Zurek. 2005. Four scenarios. Pages 223-294 in Millennium Ecosystem Assessment, editor. Ecosystems and human well- being: scenarios. Island Press, Washington, D.C., USA.

Daw, T. M., S. Coulthard, W. W. L. Cheung, K. Brown, C. Abunge, D. Galafassi, G. D. Peterson, T. R. McClanahan, J. O. Omukoto, and L. Munyi. 2015. Evaluating taboo trade-offs in ecosystems services and human well-being. Proceedings of the National Academy of Sciences 112:6949-6954. http://dx.doi.org/10.1073/pnas.1414900112

Díaz, S., S. Demissew, J. Carabias, C. Joly, M. Lonsdale, N. Ash, A. Larigauderie, J. R. Adhikari, S. Arico, A. Báldi, A. Bartuska, I. A. Baste, A. Bilgin, E. Brondizio, K. M. Chan, V. E. Figueroa, A. Duraiappah, M. Fischer, R. Hill, T. Koetz, P. Leadley, P. Lyver, G. M. Mace, B. Martin-Lopez, M. Okumura, D. Pacheco, U. Pascual, E. S. Pérez, B. Reyers, E. Roth, O. Saito, R. J. Scholes, N. Sharma, H. Tallis, R. Thaman, R. Watson, T. Yahara, Z. A. Hamid, C. Akosim, Y. Al-Hafedh, R. Allahverdiyev, E. Amankwah, T. S. Asah, Z. Asfaw, G. Bartus, A. L. Brooks, J. Caillaux, G. Dalle, D. Darnaedi, A. Driver, G. Erpul, P. Escobar-Eyzaguirre, P. Failler, A. M. M. Fouda, B. Fu, H. Gundimeda, S. Hashimoto, F. Homer, S. Lavorel, G. Lichtenstein, W. A. Mala, W. Mandivenyi, P. Matczak, C. Mbizvo, M. Mehrdadi, J. P. Metzger, J. B. Mikissa, H. Moller, H. a Mooney, P. Mumby, H. Nagendra, C. Nesshover, A. A. Oteng-Yeboah, G. Pataki, M. Roué, J. Rubis, M. Schultz, P. Smith, R. Sumaila, K. Takeuchi, S. Thomas, M. Verma, Y. Yeo-Chang, and D. Zlatanova. 2015. The IPBES conceptual framework — connecting nature and people. Current Opinion in Environmental Sustainability 14:1-16. http://dx.doi.org/10.1016/j.cosust.2014.11.002

Dreborg, K. H. 1996. Essence of backcasting. Futures 28:813-828. http://dx.doi.org/10.1016/S0016-3287(96)00044-4

Fazey, I., L. Bunse, J. Msika, M. Pinke, K. Preedy, A. C. Evely, E. Lambert, E. Hastings, S. Morris, and M. S. Reed. 2014. Evaluating knowledge exchange in interdisciplinary and multi-stakeholder research. Global Environmental Change 25:204-220. http://dx.doi.org/10.1016/j.gloenvcha.2013.12.012

Flyvbjerg, B. 2006. Five misunderstandings about case-study research. Qualitative Inquiry 12(2):219-245. http://dx.doi.org/10.1177/1077800405284363

Garmestani, A. S., and M. H. Benson. 2013. A framework for resilience-based governance of social-ecological systems. Ecology and Society 18(1):9. http://dx.doi.org/10.5751/es-05180-180109

Gude, P. H., A. J. Hansen, and D. A. Jones. 2007. Biodiversity consequences of alternative future land use scenarios in Greater Yellowstone. Ecological Applications 17:1004-1018. http://dx.doi.org/10.1890/05-1108

Hair, J. F. R. L. Tatham, R. E. Anderson, and W. Black. 1998. Multivariate data analysis. Fifth edition. Prentice-Hall, Upper Saddle River, New Jersey, USA.

Hamann, M., V. Masterson, R. Biggs, M. Tengö, B. Reyers, L. Dziba, and M. Spierenburg. 2012. Social-ecological scenarios for the Eastern Cape 2012 - 2050. Stockholm Resilience Centre, Stockholm, Sweden. [online] URL: http://www.sapecs.org/wp-content/uploads/2013/08/Eastern-Cape-Scenarios-Report-Aug-2012_-final.pdf

Hanspach, J., T. Hartel, A. I. Milcu, F. Mikulcak, I. Dorresteijn, J. Loos, H. von Wehrden, T. Kuemmerle, D. Abson, A. Kovács-Hostyánszki, A. Báldi, and J. Fischer. 2014. A holistic approach to studying social-ecological systems and its application to Southern Transylvania. Ecology and Society 19(4):32. http://dx.doi.org/10.5751/es-06915-190432

Heugens, P. M. A. R., and J. Van Oosterhout. 2001. To boldly go where no man has gone before: integrating cognitive and physical features in scenario studies. Futures 33(10):861-872. http://dx.doi.org/10.1016/s0016-3287(01)00023-4

Hill, R., C. Grant, M. George, C. J. Robinson, S. Jackson, and N. Abel. 2012. A typology of indigenous engagement in Australian environmental management: implications for knowledge integration and social-ecological system sustainability. Ecology and Society 17(1):23. http://dx.doi.org/10.5751/es-04587-170123

Hill, R., K. J. Williams, P. L. Pert, C. Robinson, A. P. Dale, D. A. Westcott, R. A. Grace, and T. O’Malley. 2010. Adaptive community-based biodiversity conservation in Australia’s tropical rainforests. Environmental Conservation 37(01):73-82. http://dx.doi.org/10.1017/s0376892910000330

Jessel, B., and J. Jacobs. 2005. Land use scenario development and stakeholder involvement as tools for watershed management within the Havel River Basin. Limnologica 35:220-233. http://dx.doi.org/10.1016/j.limno.2005.06.006

Kahane, A. 2012. Transformative scenario planning. Working together to change the future. Berrett-Koehler, San Francisco, California, USA.

Kok, K., R. Biggs, and M. Zurek. 2007. Methods for developing multiscale participatory scenarios: insights from southern Africa and Europe. Ecology and Society 13(1):8. [online] URL: http://www.ecologyandsociety.org/vol12/iss1/art8/

Kok, K. B., M. Patel, and D. S. Rothman. 2004. Final report of European and Mediterranean scenarios: upscaling the results from the target area scenarios. MedAction Deliverable 4. International Centre for Integrated Assessment and Sustainable Development (ICIS) Working Paper I04-E002. Maastricht University, Maastricht, The Netherlands.

Kok, K., and H. van Delden. 2009. Combining two approaches of integrated scenario development to combat desertification in the Guadalentín watershed, Spain. Environment and Planning B: Planning and Design 36(1):49-66. http://dx.doi.org/10.1068/b32137

Lamarque, P., A. Artaux, C. Barnaud, L. Dobremez, B. Nettier, and S. Lavorel. 2013. Taking into account farmers’ decision making to map fine-scale land management adaptation to climate and socio-economic scenarios. Landscape and Urban Planning 119:147-157. http://dx.doi.org/10.1016/j.landurbplan.2013.07.012

Lamarque, P., S. Lavorel, M. Mouchet, F. Quétier. 2014. Plant trait-based models identify direct and indirect effects of climate change on bundles of grassland ecosystem services. Proceedings of the National Academy of Sciences 111(38):13751-13756. http://dx.doi.org/10.1073/pnas.1216051111

Martín-López, B., and C. Montes. 2015. Restoring the human capacity for conserving biodiversity: a social-ecological approach. Sustainability Science 10:699-706. http://dx.doi.org/10.1007/s11625-014-0283-3

Martin-Ortega, J., K. Waylen, J. P. Martin-del-Molino, K. Blackstock, and I. Brown. 2014. Deliverable 5.1. Participatory report on synthesised scenarios: summary and comparison of the scenario building processes and outcomes in the three case studies. The James Hutton Institute, Aberdeen, Scotland.

Millennium Ecosystem Assessment. 2005. Ecosystems and human well-being: synthesis. Island Press, Washington, D.C., USA.

Mistry, J., C. Tschirhart, C. Verwer, R. Glastra, O. Davis, D. Jafferally, L. Haynes, R. Benjamin, G. Albert, R. Xavier, I. Bovolo, and A. Berardi. 2014. Our common future? Cross-scalar scenario analysis for social-ecological sustainability of the Guiana Shield, South America. Environmental Science and Policy 44:126-148. http://dx.doi.org/10.1016/j.envsci.2014.05.007

Mouchet, M. A., P. Lamarque, B. Martín-López, E. Crouzat, P. Gos, C. Byczek, and S. Lavorel. 2014. An interdisciplinary methodological guide for quantifying associations between ecosystem services. Global Environmental Change 28:298-308. http://dx.doi.org/10.1016/j.gloenvcha.2014.07.012

Nassauer, J. I., and R. C. Corry. 2004. Using normative scenarios in landscape ecology. Landscape Ecology 19:343-356. http://dx.doi.org/10.1023/B:LAND.0000030666.55372.ae

Olson, D. M., E. Dinerstein, E. D. Wikramanayake, N. D. Burgess, G. V. N. Powell, E. C. Underwood, J. A. D’amico, I. Itoua, H. E. Strand, J. C. Morrison, C. J. Loucks, T. F. Allnutt, T. H. Ricketts, Y. Kura, J. F. Lamoreux, W. W. Wettengel, P. Hedao, and K. R. Kassem. 2001. Terrestrial ecoregions of the world: a new map of life on Earth: a new global map of terrestrial ecoregions provides an innovative tool for conserving biodiversity. BioScience 51(11):933-938. http://dx.doi.org/10.1641/0006-3568(2001)051[0933:TEOTWA]2.0.CO;2

Opdam, P., R. Foppen, and C. Vos. 2001. Bridging the gap between ecology and spatial planning in landscape ecology. Landscape Ecology 16:767-779. http://dx.doi.org/10.1023/A:1014475908949

Oteros-Rozas, E., B. Martín-López, C. A. López, I. Palomo, and J. A. González. 2013. Envisioning the future of transhumant pastoralism through participatory scenario planning: a case study in Spain. The Rangeland Journal 35(3):251-272. http://dx.doi.org/10.1071/rj12092

Palomo, I., B. Martín-López, C. López-Santiago, and C. Montes. 2011. Participatory scenario planning for protected areas management under the ecosystem services framework: the Doñana social-ecological system in southwestern Spain. Ecology and Society 16(1):23. [online] URL: http://www.ecologyandsociety.org/vol16/iss1/art23/

Pereira, E., C. Queiroz, H. M. Pereira, and L. Vicente. 2005. Ecosystem services and human well-being: a participatory study in a mountain community in Portugal. Ecology and Society 10(2):14. [online] URL: http://www.ecologyandsociety.org/vol10/iss2/art14/

Pert, P. L., R. Hill, K. J. Williams, E. K. Harding, T. O’Malley, R. A. Grace, A. P. Dale, I. Bohnet, and J. R. L. A. Butler. 2010. Scenarios for community-based approaches to biodiversity conservation: a case study from the Wet Tropics, Queensland, Australia. Australian Geographer 41(3):285-306 http://dx.doi.org/10.1080/00049182.2010.498037

Peterson, G. D., T. D. Beard Jr., B. E. Beisner, E. M. Bennett, S. R. Carpenter, G. S. Cumming, C. L. Dent, and T. D. Havlicek. 2003b. Assessing future ecosystem services: a case study of the Northern Highlands Lake District, Wisconsin. Conservation Ecology 7(3):1. [online] URL: http://www.consecol.org/vol7/iss3/art1/

Peterson, G. D., G. S. Cumming, and S. R. Carpenter. 2003a. Scenario planning: a tool for conservation in an uncertain world. Conservation Biology 17(2):358-366. http://dx.doi.org/10.1046/j.1523-1739.2003.01491.x

Plieninger, T., C. Bieling, B. Ohnesorge, H. Schaich, C. Schleyer, and F. Wolff. 2013. Exploring futures of ecosystem services in cultural landscapes through participatory scenario development in the Swabian Alb, Germany. Ecology and Society 18(3):39. http://dx.doi.org/10.5751/es-05802-180339

Quinlan, A. 2012. Using future scenarios to explore alternate governance trajectories. Chapter 8 in Assessing ecosystem service governance: interactions among actors in a rural watershed in eastern Ontario. Dissertation, Carleton University, Ottawa, Ontario, Canada.

Ravera, F., K. Hubacek, M. Reed, and D. Tarrasón. 2011a. Learning from experiences in adaptive action research: a critical comparison of two case studies applying participatory scenario development and modelling approaches. Environmental Policy and Governance 21:433-453. http://dx.doi.org/10.1002/eet.585

Ravera, F., D. Tarrasón, and E. Simelton. 2011b. Envisioning adaptive strategies to change: participatory scenarios for agropastoral semiarid systems in Nicaragua. Ecology and Society 16(1):20. [online] URL: http://www.ecologyandsociety.org/vol16/iss1/art20/

Reed, M. S., J. Kenter, A. Bonn, K. Broad, T. P. Burt, I. R. Fazey, E. D. G. Fraser, K. Hubacek, D. Nainggolan, C. H. Quinn, L. C. Stringer, and F. Ravera. 2013a. Participatory scenario development for environmental management: a methodological framework illustrated with experience from the UK uplands. Journal of Environmental Management 128:345-362. http://dx.doi.org/10.1016/j.jenvman.2013.05.016

Reed, M. S., A. Graves, N. Dandy, H. Posthumus, K. Hubacek, J. Morris, C. Prell, C. H. Quinn, L. C. Stringer. 2009. Who’s in and why? A typology of stakeholder analysis methods for natural resource management. Journal of Environmental Management 90:1933-1949. http://dx.doi.org/10.1016/j.jenvman.2009.01.001

Reed, M. S., K. Hubacek, A. Bonn, T. P. Burt, J. Holden, L. C. Stringer, N. Beharry-borg, S. Buckmaster, D. Chapman, P. J. Chapman, G. D. Clay, S. J. Cornell, A. J. Dougill, A. C. Evely, E. D. G. Fraser, N. Jin, B. J. Irvine, M. J. Kirkby, W. E. Kunin, C. Prell, C. H. Quinn, B. Slee, S. Stagl, M. Termansen, S. Thorp, and F. Worrall. 2013b. Anticipating and managing future trade-offs and complementarities between ecosystem services. Ecology and Society 18(1):5. http://dx.doi.org/10.5751/ES-04924-180105

Rodríguez, J. P., T. D. Beard Jr, E. M. Bennett, G. S. Cumming, S. J. Cork, J. Agard, A. P. Dobson, and G. D. Peterson. 2006. Trade-offs across space, time, and ecosystem services. Ecology and Society 11(1):28. [online] URL: http://www.ecologyandsociety.org/vol11/iss1/art28/

Sala, O. E., F. S. Chapin III, J. J. Armesto, E. Berlow, J. Bloomfield, R. Dirzo, E. Huber-Sanwald, L. F. Huenneke, R. B. Jackson, A. Kinzig, R. Leemans, D. M. Lodge, H. A. Mooney, M. Oesterheld, N. L. Poff, M. T. Sykes, B. H. Walker, M. Walker, and D. H. Wall. 2000. Global biodiversity scenarios for the year 2100. Science 287:1770-1774. http://dx.doi.org/10.1126/science.287.5459.1770

Sheppard, S. R. J., A. Shaw, D. Flanders, S. Burch, A. Wiek, J. Carmichael, J. Robinson, and S. Cohen. 2011. Future visioning of local climate change: a framework for community engagement and planning with scenarios and visualization. Futures 43(4):400-412. http://dx.doi.org/10.1016/j.futures.2011.01.009

Sikor, T. 2013. The justices and injustices of ecosystem services. Routledge, London, UK.

Simon, D., and F. Schiemer. 2015. Crossing boundaries: complex systems, transdisciplinarity and applied impact agendas. Current Opinion in Environmental Sustainability 12:6-11. http://dx.doi.org/10.1016/j.cosust.2014.08.007

Stringer, L. C., A. J. Dougill, E. Fraser, K. Hubacek, C. Prell, and M. S. Reed. 2006. Unpacking “participation” in the adaptive management of social-ecological systems: a critical review. Ecology and Society 11(2): 39. [online] URL: http://www.ecologyandsociety.org/vol11/iss2/art39/

Swart, R. J., P. Raskin, and J. Robinson. 2004. The problem of the future: sustainability science and scenario analysis. Global Environmental Change 14(2):137-146. http://dx.doi.org/10.1016/j.gloenvcha.2003.10.002

Tengö, M., E. S. Brondizio, T. Elmqvist, P. Malmer, and M. Spierenburg. 2014. Connecting diverse knowledge systems for enhanced ecosystem governance: the multiple evidence base approach. Ambio 43:579-591. http://dx.doi.org/10.1007/s13280-014-0501-3

van der Heijden, K. 2000. Scenarios and forescasting: two perspectives. Technological Forecasting and Social Change 65:31-36. http://dx.doi.org/10.1016/S0040-1625(99)00121-3

van Notten, P. W. F., J. Rotmans, M. B. A van Asselt, and D. S. Rothman. 2003. An updated scenario typology. Futures 35(5):423-443. http://dx.doi.org/10.1016/s0016-3287(02)00090-3

van Vliet, M., K. Kok, A. Veldkamp, and S. Sarkki. 2012. Structure in creativity: an exploratory study to analyse the effects of structuring tools on scenario workshop results. Futures 44(8):746-760. http://dx.doi.org/10.1016/j.futures.2012.05.002

Vilardy Quiroga, S. P., B. Martín-L+pez, E. Oteros-Rozas, and W. Renán-Rodríguez. 2011. Escenarios de futuro en la Ciénaga Grande de Santa Marta. Pages 172-193 in S. P. Vilardy, and J. A. González, editors. Repensando la Ciénaga: nuevas miradas y estrategias para la sostenibilidad en la Ciénaga Grande de Santa Marta. Universidad del Magdalena y Universidad Autónoma de Madrid, Santa Marta, Colombia.

Volkery, A., and T. Ribeiro. 2009. Scenario planning in public policy: understanding use, impacts and the role of institutional context factors. Technological Forecasting and Social Change 76(9):1198-1207. http://dx.doi.org/10.1016/j.techfore.2009.07.009

Von Wirth, T., U. Wissen Hayek, A. Kunze, N. Neuenschwander, M. Stauffacher, and R. W. Scholz. 2014. Identifying urban transformation dynamics: functional use of scenario techniques to integrate knowledge from science and practice. Technological Forecasting and Social Change 89:115-130. http://dx.doi.org/10.1016/j.techfore.2013.08.030

Wack, O. 1985. Scenarios: shooting the rapids. Harvard Business Review 63(6):139-150.

Walz, A., C. Lardelli, H. Behrendt, A. Grêt-Regamey, C. Lundstöm, S. Kytzia, and P. Bebi. 2007. Participatory scenario analysis for integrated regional modelling. Landscape and Urban Planning 81(1-2):114-131. http://dx.doi.org/10.1016/j.landurbplan.2006.11.001

Ward, J. H., Jr. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association 58:236-244. http://dx.doi.org/10.1080/01621459.1963.10500845

Waylen, K. A., J. Martin-Ortega, K. L. Blackstock, I. Brown, B. E. Avendaño Uribe, S. Basurto Hernández, M. B. Bertoni, M. L. Bustos, A. X. Cruz Bayer, R. I. Escalante Semerena, M. A. Farah Quijano, F. Ferrelli, G. L. Fidalgo, I. Hernández López, M. A. Huamantinco Cisneros, S. London, D. L. Maya Vélez, P. N. Ocampo-Díaz, C. E. Ortiz Guerrero, J. C. Pascale, G. M. E. Perillo, M. C. Piccolo, L. N. Pinzón Martínez, M. L. Rojas, F. Scordo, V. Vitale, and M. Zilio. 2015. Can scenario-planning support community-based natural resource management? Experiences from three countries in Latin America. Ecology and Society 20(4):28. http://dx.doi.org/10.5751/ES-07926-200428

Wollenberg, E., D. Edmunds, and L. Buck. 2000. Using scenarios to make decisions about the future: anticipatory learning for the adaptive co-management of community forests. Landscape and Urban Planning 47(1-2):65-77. http://dx.doi.org/10.1016/s0169-2046(99)00071-7

Yin, R. 2009. Case study research design and methods. Fourth edition. Sage, Thousand Oaks, California, USA.