INTRODUCTION

The social-ecological systems framework (SESF) remains one of the most highly cited and empirically applied conceptual frameworks for diagnosing social-ecological systems (Ostrom, 2007, 2009, McGinnis and Ostrom 2014). Notably, the SESF does not have a methodological guide or a standardized set of procedures to empirically apply it. This is to some extent by design, to allow flexibility in how methods are adapted to diverse contexts (McGinnis and Ostrom 2014). However, this has led to highly heterogeneous applications and challenges in designing a coherent set of data collection and analysis methods across cases.

A main challenge is that methodology is a general term, which actually refers to a set of stepwise specific procedures which can include study design, conceptualization of variables and indictors for data collection, empirical or secondary data collection, data processing and cleaning, data analysis, as well as data visualization, communication, and sharing. Although the SESF provides a uniform set of variables, it does not indicate any of the other necessary steps for a robust scientific study. Applying the SESF is not a method itself, but it is arguably a theory-derived conceptual guide for focusing the methods a researcher does choose on a set of variables that have previous empirical support in shaping commons, institutional development and change, and/or collective action outcome. Thus, scholars are forced to either mirror previous studies or develop their own procedures, leaving heterogeneous applications that enable contextually tailored approaches but hinder comparability across studies.

The focus of this study is to explicitly synthesize the methods applied in SESF studies by systematically reviewing published quantitative applications of the SESF and to develop a methodological guide for the framework’s continued application while highlighting the challenges in current literature. A guide is useful so that scholars can map their methodological choices more transparently, sparking reflections for their own study designs and better enabling the systematic communication of study methodological decisions to others. To apply the SES framework, a series of methodological steps are needed. These steps have been referred to by Partelow (2018) as methodological gaps, because if they are not explicitly defined by authors, they can lead to a lack of transparency for future comparability and interpretability by other scholars. The methodological gaps include: the (1) variable definition gap, (2) variable to indicator gap, (3) the measurement gap, (4) the data transformation gap.

Focusing on methodologies is important for two reasons. First, synthesis research to build theoretical insights across SES applications has been a challenge because the full spectrum of methodological designs and concept definitions are often not fully published or are simply too heterogeneous for making contextually meaningful comparisons (Thiel et al. 2015, Partelow 2018, Cumming et al. 2020, Cox et al. 2021). For example, Villamayor-Tomas et al. (2020) found that the majority of reviewed models from 30 SESF studies were lacking detail regarding what methods or approaches were used to identify the relationships between variables that the authors were presenting. Second, the SESF itself does not provide any explanation of the factors or causal relationships that are shaping the observed SES problem or phenomena. The framework only provides a common vocabulary and a diagnostic conceptual organization of 1st-tier component interactions, not a procedure regarding how or which methods should be applied with the SESF to investigate these factors.

The methodological guide proposed from this review is applicable, in our view, to all future applications of the framework, both quantitative and qualitative. Nonetheless, quantitative methods were used as the basis for the review because they typically follow systematic procedures for data collection and analysis through the discipline of statistics, which in the data collection phase, translates empirical observations into comparable sets of numbers that can be analyzed with standardized analytical techniques. Specifically defined indicators and variables are needed for quantification along with specific steps to appropriately transform and analyze data, in contrast to qualitative studies, in which reproducibility and generalizable measurement may not be possible or is not the goal of the research. Reproducible criteria for how variables are measured in qualitative studies is by nature more difficult because a primary objective in many qualitative contexts is the rich analysis of data, contexts, and processes not easily reduced to individual variables (Queirós et al. 2017) and often focused on broader knowledge transferability than specific data comparability (Guba 1981).

Previous studies have outlined sets of questions or procedures for applying the framework more specifically, such as for conceptualizing and defining the case SES and action situation (Hinkel et al. 2014, 2015, Partelow 2016). However, there is no systematic or procedural guide with a focus on outlining different methodological strategies and choices. As such, this review aims to make two major contributions. First, to review current applications of the SESF to compile a multi-step guide of methodological steps for applying the SESF framework. Second, to use these results as a base for constructively analyzing current trends, inconsistencies, and challenges in applying the framework to date and to highlight needed methodological advancements and paths forward in SESF research. Through a systematic review of SESF methodologies, we explored the methodological heterogeneity and gaps across the literature and discuss how this heterogeneity can lead to ambiguity for synthesis work. Combined with feedback from a survey regarding ongoing SESF challenges from 22 co-authors of publications included in this review, we identified methodological strategies at each step of study design, data collection, and analysis and then we provide a synthetic methodological guide to inform future applications, while also positing critical reflections on the limitations of current approaches.

FRAMEWORK AND REVIEW METHODOLOGY

Social-ecological systems framework

The SESF was developed to conduct institutional analyses on natural resource systems and diagnose related collective action challenges. The core of the framework provides a decomposable list of variables situated around an “action situation” in which actors make decisions and actions based on the available information within their positions, which enables researchers to structure diagnostic inquiry and compare findings. Although most empirical applications of the SESF have established some theoretical ties to the study of the commons and collective action (Partelow 2018), the SESF was conceived and gained traction as a useful tool for the broader characterization and analysis of SES sustainability (Ostrom 2009) and as a “theory-neutral” framework that can be used with other theories or to build new theories (McGinnis and Ostrom 2014, Cox et al. 2016). For a more complete history of the SESF and its connection to the institutional analysis and design (IAD) framework, see its foundational publications (Ostrom, 2007, 2009, McGinnis and Ostrom 2014) as well as previous syntheses and reviews (Thiel et al. 2015, Partelow 2018, 2019).

The SESF is divided into several 1st-tier components representing social and ecological as well as external factors and system interactions and outcomes, each divided into multiple 2nd-tier variables (Ostrom 2009; Table 1). By breaking down an SES into a set of decomposable, nested, and generalizable concepts, the SESF aims to achieve a dual purpose, (1) facilitating an understanding of the specific and contextual factors influencing SES outcomes at a fine local scale and (2) also sharing a common general vocabulary of variables to facilitate the identification of commonalities across cases to build policy recommendations and theory at varying levels of generalizability (Basurto and Ostrom 2009, Ostrom and Cox 2010).

Although the existing literature suggests that the SESF is being successfully applied as a contextually adaptable tool for local SES case analysis, synthetic analysis remains a critical challenge, and the goal of comparability across studies has arguably not been fully realized (Partelow 2018). Scholars applying the SESF have been innovative and exploratory in how their data are collected, analyzed, and reused, leading to methodological pluralism, heterogeneity, and often ambiguity in how the SESF is or should be applied, such as the lack of clarity in how case-relevant variables should be selected and measured (Partelow 2018), as well as difficulties with ambiguous or abstract variable definitions (Hinkel et al. 2014, Thiel et al. 2015). Existing SES and commons database synthesis efforts exist but are made more difficult by the broad range of methodological approaches and inconsistencies with how the framework is applied and variables measured (Cox et al. 2020, 2021). Recent synthesis work of the SESF has noted challenges including both the lack and heterogeneity of information on variable relationships and causal inferences across publications, limiting analysis to only the co-occurrence of variables across SESF studies (Villamayor-Tomas et al. 2020). Social-ecological systems framework applications are taking different approaches to selecting, justifying, measuring, and analyzing SESF variables and lack precision in concepts and measurements (Cumming et al. 2020). We therefore identify methodological inconsistencies in applying the SESF as one major ongoing hurdle to comparable and synthetic SES research, and thus the primary focus of our review.

Methods

This study applied systematic review methods to peer-reviewed literature collected from SCOPUS, Web of Science Core Collection, and Google Scholar between August to September 2020 (with a follow-up search in January 2021) to identify any literature applying the SESF with some degree of quantitative data analysis (Appendix 1, Fig. A1.1). The initial SCOPUS and Web of Science title/abstract search used search terms (TITLE-ABSTRACT ("social-ecological system* framework" OR “social ecological system* framework”) OR "SES framework") OR TITLE-ABSTRACT ("social-ecological system*" AND "framework" AND Ostrom")) OR TITLE-ABSTRACT ("social-ecological system*" AND "SESF")) and a follow-up search with Google Scholar to identify any additional publications, which after removing duplicates resulted in an initial set of 330 peer-reviewed publications. Because a key focus of this review is on the heterogeneity of explicit methodological procedures and variable measurements affecting generalizability, comparability, and reproducibility of results, we chose to focus on completely or mixed-methods quantitative applications of the SESF, which are more likely to face limitations in these regards. These criteria included all publications that applied the SESF and analyzed any amount of quantitative raw or transformed data. Publications with any ambiguities with regard to these criteria were discussed between co-authors to reach consensus on inclusion in the review. A title/abstract scan removed all publications not applying the SESF, followed by a full-text review to identify those applying a quantitative analysis, which identified 46 publications. A follow-up search in January 2021 identified 4 additional publications and 1 additional publication was identified during peer-review, resulting in a total of 51 publications for final review. Each article was evaluated using a standardized coding form that was pre-tested by the authors for consistency. The review followed two guiding questions: (1) How is the SESF being applied with quantitative/mixed-methods quantitative approaches (sectors, research aims, and analytical methods)? (2) How are the 2nd-tier SESF variables being applied (variable selection criteria, data collection, measurable indicator selection criteria)?

To answer these questions, we coded the following data from each publication: purpose for applying the SESF, focal SES analyzed, data analysis methods, challenges in applying the SESF, 2nd-tier variable selection and inclusion criteria, measurable indicator selection, data collection methods, and data type. We make an important distinction between “variables,” or the generally defined 2nd-tier concepts of the SESF, and “indicators” referring to how the variables are actually measured. Any ambiguities during the coding and evaluation process were flagged and discussed between co-authors to reach consensus. Initial coding was completed in February 2021. To gather more explicit reflections from researchers regarding SESF methodological challenges, critiques, and reflections, a researcher survey was also conducted. The survey questionnaire was distributed to all corresponding authors of the reviewed publications starting in February 2021 and consisted of Likert scale and full-text response questions about their experiences with the SESF. The full list of reviewed publications can be found in Appendix 2, 2nd-tier SESF variable indicators from reviewed publications in Appendix 3, and the evaluation forms, procedure, and author survey questionnaire in Appendix 4. The guide steps were developed based on gaps and trends in the SESF literature, in particular the previously noted methodological gaps in the SESF (Partelow 2018) and were further iterated based on the results of the review, researcher survey, experiences in planning our own research with the SESF, and on-going discussions between novice and experienced SESF researchers in our working group.

RESULTS

A multi-step methodological guide for applying the social-ecological systems (SES) framework

Our findings indicate that researchers applying the SESF make a series of methodological choices that can be organized into a multi-step guide that includes all the aggregated choice options across studies at each step. We present this as a 10-step methodological guide and decision tree (Fig. 1). The steps are arranged in what we identified as a generally logical order, but the specific order of operations is likely to vary based on specific research aims. The branches within the decision tree for each numbered step are not all-encompassing, but instead represent, for each step, the categories that were identified and coded in the reviewed SESF publications, with a handful of potential additional categories identified by the authors. A total of 22 complete responses to the SESF researcher survey were received from co-authors of the 51 reviewed publications. Likert-scale survey responses are presented in Figure 2, and Appendix 1 (Table A1.1) summarizes categories of responses to the short answer survey questions.

(1) What is the primary purpose for applying the SESF?

The SESF is generally positioned as a tool to guide diagnostic SES inquiry, but how it is actually applied varies substantially. One application may develop theoretically derived hypotheses on how 2nd-tier variables are linked to collective action and self-organization in a case (e.g., Klümper and Theesfeld 2017, Su et al. 2020). Others might take an inductive approach, using the SESF to code and compare local perceptions of the SES (e.g., Ziegler et al. 2019, Partelow et al. 2021), or use the SESF basis to develop a model of individual actor behavior in an SES (e.g., Cenek and Franklin 2017, Lindkvist et al. 2017).

Most respondents to the researcher survey stated that it was clear how to apply the SESF to their research, how to use the SESF to support theory building and testing, and how to identify relevant variables for a given case. The SESF was typically chosen by respondents because of its clear and coherent organizational structure and comprehensive coverage of a wide range of social and ecological dimensions, however, nearly a third (n = 7) of respondents chose the SESF at least in part due to its origins in the study of the commons and collective action theory. In our synthetic review, we broadly categorized the purpose for applying the framework as extracted from introduction and methods sections of reviewed publications. Although most studies incorporate multiple objectives, the majority of reviewed publications applied the framework with the primary aim of predicting explanatory social-ecological drivers of (typically a small number of) measured dependent variables representing SES outcomes (e.g., Fujitani et al. 2020, Okumu and Muchapondwa 2020; n = 31). The remaining publications were divided between characterization of SESs through descriptive or diagnostic measurements of the important variables (e.g., Leslie et al. 2015, Rocha et al. 2020; n = 10), testing or projecting potential future SES scenarios through simulations or models of system behavior (e.g., Baur and Binder 2015, Cenek and Franklin 2017; n = 5), or social learning aimed at understanding or better integrating local SES user knowledge and perspectives (e.g., Delgado-Serrano et al. 2015, Oviedo and Bursztyn 2016; n = 5). This broader purpose or goal in applying the SESF informs a wide heterogeneity of methodological decisions and considerations leading to the final study outcome.

(2) Is inter- or transdisciplinary research needed to appropriately conduct the study?

Research with the SESF often requires the integration of concepts and data from a wide array of disciplines. Researchers must consider whether adequately analyzing, describing, or diagnosing an SES may require the integration of diverse knowledge types and formats. This integration can take place across multiple dimensions, levels, and scales (Guerrero et al. 2018). Common criticisms of the SESF, for instance, note that the framework itself developed from disciplinary roots in the social sciences, and it is lacking an equivalent depth of consideration of ecological processes and theories (Epstein et al. 2013, Vogt et al. 2015). Our review found that ecological variables are underrepresented compared to social variables in SESF studies (Table 2), and SESF researchers are also more likely to rely on secondary data for ecological variables than for social variables (Fig. 3).

Integration of different scientific disciplinary expertise (interdisciplinary; Hicks et al. 2010, Bennett et al. 2016) or of scientific and non-scientific expertise (transdisciplinary; Caniglia et al. 2021, Lam et al. 2021) can influence how and to what extent all social and biophysical components and dynamics of the SES are investigated, as well as for whom the study outcomes are relevant and meaningful (Guerrero et al. 2018). Many reviewed publications included stakeholders in the research through household surveys or interviews, but only 12 studies were identified that actually integrated stakeholders into the study co-design process, either by influencing the research questions or objectives, or by playing a direct role in the selection and evaluation of relevant SESF variables. Including relevant non-scientific stakeholders at multiple stages in the research can increase knowledge exchange and research influence (Reyers et al. 2015) and the SESF has been demonstrated as a tool to enhance communication between actors in SES governance (Gurney et al. 2019, Partelow et al. 2019). Reflecting on the appropriate type and level of integration should be an important early methodological consideration in SESF research design.

(3) What is the focal SES(s) of analysis and factors determining its boundaries?

Defining the SES and its boundaries is essential for determining how the individual variables are analyzed in relation to what the internal and external influences on those variables are. The focal sector will also determine the degree to which the analysis could be compared to another study or the practical implications of the findings. Most studies are still applying the SESF to classic common pool resource problems (van Laerhoven et al. 2020) in sectors such as forestry and fisheries (Appendix 1, Table A1.2), providing a larger library of sector-specific comparable studies and variables for authors studying these SESs to reference in designing their own research. The SESF is place-based in design, and researchers should also consider what is within the study system and what is external to its context, and this justification should be established based on the research objectives. For example, SESs often have fuzzy social and ecological boundaries that are not easily delineated and often do not align with each other, and how a researcher bounds the system in their study can have implications for the study findings. The focal SESs in the reviewed literature were described or analyzed with boundaries based on social (n = 29), ecological (n = 8), or mixed or fuzzy factors (n = 12; Appendix 1, Table A1.2). A study might have increased clarity or relevance to policymakers by bounding their analysis by administrative borders but fail to adequately capture important ecological processes not conforming to these social boundaries. We have included defining scope and SES boundary clarification as a key step in our guide because of its methodological implications for the rest of the study, but direct researchers to an existing detailed procedure for conceptualizing and defining the focal SES and institutional action situation of analysis (Hinkel et al. 2015).

(4) What are the primary unit(s) of analysis, number of units, and scales of analysis?

Who or what does the study hope to specifically inform? What is the best spatial fit for the SES phenomena being studied? Although most SESF studies are situated within the case context of one or more SESs, actual units of analysis might range from individual aquaculture ponds (Partelow et al. 2018) to residential neighborhoods (Schmitt-Harsh and Mincey 2020) to administrative provinces (Dressel et al. 2018). The selection of unit of analysis, including number of units compared and spatial and temporal levels of analysis, all impact the granularity and types of generalizations that can be made by the study findings and may also reflect certain practical considerations in terms of data collection. We coded units of analysis at the individual (e.g., individual survey respondent), local (e.g., community), or regional (e.g., geographic region or administrative level encompassing multiple communities or governance units) spatial level. Local and individual units were the most common, followed by regional units ranging from political districts (Dressel et al. 2018, Rocha et al. 2020) to large social-ecological regions (Leslie et al. 2015; Table 3). We categorized studies comparing 30 or more units as large-N, following the central limit theorem (with some studies comparing multiple units of analysis). Large-N comparisons of individual or local units were the most common in the reviewed literature, with only two large-N studies comparing regional units. Additionally, although we identified eight publications analyzing cases across multiple countries, only three cross-national studies collected empirical data (including two studies from the same project: Aaron MacNeil and Cinner 2013; Cinner et al. 2012), with the rest reliant entirely on existing secondary data sources. Although our review focused primarily on coding the number and spatial level of units of analysis, we also emphasize the importance of a wide range of critical scales or dimensions for SES analysis. See Glaser and Glaeser 2014 for further reflections on these dimensions.

(5) Which 2nd-tier SESF variables are being examined and what are the inclusion or exclusion criteria?

No empirical studies examine all of the 2nd-tier variables in the framework. Clearly communicating which 2nd-tier variables were selected, and why or why not, improves understandability and comparability. Ambiguities regarding interpreting, selecting, and defining relevant 2nd-tier variables for a given case were the most frequently reported negative aspect of applying the SESF in our survey. Respondents noted the subjectivity in how variables can be defined, allowing for great flexibility but diminishing comparability. Challenges also exist with interpreting whether high or low “states” of a variable may lead to favorable or unfavorable outcomes (e.g., variable hypotheses). Of the 51 reviewed publications, 26 provided clear documentation of all 2nd-tier variables being examined (Fig. 4). The remaining 25 publications were excluded from 2nd-tier variable and indicator analysis because they were either opting not to apply the 2nd-tier variables or lacked clarity regarding which (if any) 2nd-tier variables were being examined. For example, some studies were merging parts of the SESF with other conceptual frameworks, and others provided only a list of indicators categorized by the 1st-tier components, without conclusive indication of which (if any) 2nd-tier variables they aligned with. In some studies, there was a purposive decision to not to apply the 2nd-tier variables by study authors, such as in modeling approaches focused on individual unit behavior within the SES rather than broader SES components. However, in many studies the reasoning was unclear. Some of the 25 excluded publications included alternative 2nd-tier variable definitions or numbering schemes without specifying if these alterations were intended to be interpreted as unmodified, modified, or entirely new 2nd-tier variables (Roquetti et al. 2017, Okumu and Muchapondwa 2020). Modifications to the framework, including adding variables, should be justified while noting the theoretical inclusion criteria that the included variables were based on (Frey and Cox 2015, Partelow 2018). Because journal word counts are often a limiting factor, authors might consider including a clearly formatted 2nd-tier variable appendix as supplementary material (Leslie et al. 2015, Foster and Hope 2016, Dressel et al. 2018, Osuka et al. 2020).

Each study selects this subset of variables based on criteria such as expected relevance to the study. Was a variable excluded because it was not empirically meaningful for the case, because it was potentially relevant but not easily empirically measurable, or because it was not in the authors’ interest to examine it? Was a variable included because the authors have formulated a clear hypothesis for its case relevance or because an abundance of secondary data are readily available to measure it? In the reviewed publications, existing literature and theory was the most common reported criteria, followed by local SES actor expert knowledge, as well as data availability and scarcity influencing variable selection (Table 2). Most studies reported only a general list of inclusion/exclusion criteria (e.g., “our variables were selected based on literature review and expert knowledge”), rather than specific criteria for every included variable in either the main text or supplementary material. Additionally, in five studies we could find no basis for why the selected variables were chosen. Clearly formulated hypotheses for why each included variable was relevant to a case were only identified in eight studies (Leslie et al. 2015, Foster and Hope 2016, Dressel et al. 2018, Partelow et al. 2018, Haider et al. 2019, Rana and Miller 2019a, Osuka et al. 2020, Rocha et al. 2020). Inclusion and exclusion criteria are not always clear-cut and might be based on multiple theoretical, methodological, or logistical aspects. Particularly for quantitative approaches, 2nd-tier variable inclusion and exclusion is likely to also be influenced by statistical factors. In many cases, adding additional variables may need to be weighed against the potential loss of statistical power that this may entail. Similarly, some otherwise relevant variables might be omitted from a study because preliminary data exploration shows high multi-collinearity in their measurements (e.g., Gurney et al. 2016). Documenting not only inclusion criteria but also exclusion criteria should be strongly considered by authors, particularly when 2nd-tier variables may have been omitted for reasons beyond solely a lack of case relevance.

(6) How are selected 2nd-tier variables being measured?

Can the variable be directly measured empirically, given the study design and data collection method? Most of the 2nd-tier variables are concepts and are not directly measurable (at least quantitatively) without specifying one or multiple indicators to represent the concept empirically or to specify its empirical meaning, thus these indicators often form the true unit of comparison in many SESF studies. Even if studies examine the same 2nd-tier variable, they likely select different indicators to specify and measure them. In such cases, what indicators are selected, how many, and why should be considered. Almost half (n = 10) of survey respondents disagreed that it was clear how to identify relevant measurable indicators, and respondents also noted subjectivity and inconsistencies regarding where a given indicator might be coded into the SESF. Our findings suggest heterogeneous and context-dependent indicator selection decisions, with most publications collecting indicators from a wide range of sources and data types. Examples of this indicator diversity for variables RS5 and A2 are shown in Table 4. Study-specific interpretations of 2nd-tier variables and related choice of measurable indicators were highly varied, and reviewed publications were inconsistent in documenting which measurable indicators were applied. Because existing SESF case studies are likely to be an important resource and reference point when identifying appropriate measurable indicators, specificity in documentation of this step when publishing SESF research is critical to improve interpretability and comparability of findings. A selection of all 2nd-tier variable indicators that could be clearly identified in our synthesis can be found in Appendix 3.

(7) What data collection methods are used for the selected indicators?

Social-ecological systems framework studies are likely to rely on a range of different data collection methods and both primary and secondary sources in collecting data for a heterogeneous range of variables in often data-scarce contexts, and researchers should carefully consider the implications for their study design and analysis. Primary data collection ensures complete researcher control over how variables and indicators are measured but is often not feasible across a wide and mixed range of variables. Secondary data collection is often more feasible but may have issues of ambiguity regarding the data quality and clarity of data collection and measurement. Almost all primary data are being collected via social science methods such as questionnaires, interviews, and focus groups (Table 5). Across the 26 studies with clearly articulated 2nd-tier variable selections, primary ecological or biophysical survey data were collected to measure only 9 indicators. Overall, primary data collection is more common than reliance on secondary data. Comparing data collection methods by 1st-tier SESF components suggests that researchers using the SESF are collecting a higher proportion of their social variable data from primary sources compared to their ecological variable data (Fig. 3). However, this trend is highly heterogeneous at the 2nd-tier level (Fig. 4). Thirteen studies relied only on primary data, 20 studies on only secondary data, and 15 studies collected data from a mixture of primary and secondary sources. Our findings indicate that data collection methods across the reviewed literature are wide-ranging with most individual studies applying multiple data collection methods and mixed data types.

(8) What type of data is measured for the selected indicators?

Heterogeneity in data sources and collection methods in SESF studies is likely to result in a range of data types or formats. Schmitt-Harsh and Mincey 2020, for example, combined continuous quantitative indicators calculated from GIS data with ordinal indicators from a multiple-choice survey and binary presence/absence classifications of residential properties. Measuring indicators with a range of mixed data types (e.g., continuous, ordinal, categorical) might facilitate the inclusion of more SESF variables but limits the types of statistical analyses available or requires extensive data processing and transformation. Documentation regarding which indicators were data transformed for analysis was not consistent enough across publications to evaluate in full, however min-max normalization was the most frequent transformation identified. The type or format of the collected data can also add a further layer of abstraction to interpreting or comparing SESF variables in a given study and should be made transparent. For example, two studies seemingly defining the same indicator, e.g. "Kilograms of fish catch," may measure it in different ways, such as from a numeric value (e.g., 37 kg) to a qualitative ordinal scale (e.g., below average, average, above average). These differences in measurement may lead to notable differences in interpretation.

(9) What data analysis methods are being applied?

Data analysis methods broadly encompass the techniques for collection and analysis of data to draw insights. Because the SESF is to an extent only a selection of potentially relevant variables, it can be applied to any number of analysis methods that are determined by the research objectives. The choice of analysis method influences (or is influenced by) overall study design, sample sizes, variable selection, data collection, as well as the inferences that can be made regarding the SESF variables being evaluated and external validity of the study findings. In some regard then, the choice of analysis method encompasses all the previous steps in this methodological guide. We coded the data analysis methods used in the reviewed literature into 11 general categories, provided in Table 6, including potential advantages and disadvantages that researchers might have to weigh with each approach, as well as example studies that exemplify each category.

Studies generally applied multiple analysis methods, but the most frequently coded approach included explanatory/dependent variable analyses (n = 31). Fourteen studies focused on characterizing one or multiple SESs through descriptive or comparative assessments of SESF variables rather than explicitly analyzing causal mechanisms or dependent variables. We further differentiated these SES characterization studies into “descriptive” characterization studies (n = 7), which assess and compare variable measures without a normative value judgement, and “evaluative” characterization studies (n = 7), which provide a normative score (such as from 0-1), alongside supporting theory or literature, for how high or low measures for each variable relate to the evaluative criteria, e.g., potential for sustainability or collective action. Twelve studies utilized modeling and simulation-based analyses (n = 12) to investigate SES structure and behavior, including agent-based and system dynamics models. Seven studies used participatory modeling and evaluation techniques, exploring local expert knowledge and perceptions of the SES as a key source of scientific insight in what are often otherwise data-scarce SES contexts. An additional seven publications applied meta-analyses of the published literature or other existing aggregated case databases. Notably, only one of these studies specifically synthesized empirical SESF literature (Villamayor-Tomas et al. 2020), while the rest used the SESF as a coding tool for existing aggregated cross-case data. We labeled another category as mixed-conceptual (n = 6), representing studies that drew from other conceptual or theoretical frameworks, typically adapting only certain components, or heavily modified versions, of the SESF. Although the results of such studies may be less directly comparable to other SESF applications, they represent one way in which the SESF is being adapted to explore new theoretical insights and lines of inquiry beyond its original design.

(10) Is study SES data publicly available?

Data transparency, including data sharing as well as other contextual information such as how the data were generated or limitations regarding the data, is a critical component of creating more comparable SES knowledge. Eight of the reviewed publications identified an available data source, evaluated by the criteria of whether the publication, journal page, or linked supplementary material explicitly identified a publicly available source for the study data. Although the majority of survey respondents agreed that using the SESF made it more likely that their empirical data can be compared with other SESF studies, this question also had the largest number of neutral responses (7) of all of the questions. Response comments noted the diversity of SES case contexts and uniqueness of each case as challenges. Supplementary publication materials, synthetic databases, and open-source repositories are examples of useful strategies for increasing comparability across heterogeneous SES studies. Several databases have been developed in an attempt to facilitate data synthesis and comparison across SES cases, such as the Dartmouth SESMAD project (Cox et al. 2020; https://sesmad.dartmouth.edu/), SES Library (https://seslibrary.asu.edu/), and more context specific databases such as the International Forestry Resources and Institutions (IFRI; http://ifri.forgov.org) and Nepal Irrigation Institutions and Systems (NIIS; https://ulrichfrey.eu/en/niis/). How well a given case dataset “fits” to the content structure of these databases may vary depending on how the SESF was applied for a given study. Open-source data repositories provide more flexibility for authors regarding how or in what format they share their SES case data but may be less immediately comparable to other cases.

DISCUSSION

The SESF partly aims to provide a common language of variables to coordinate and compare findings, while simultaneously allowing for adaptability by not specifying which variables or methods should be applied to case-specific contexts (McGinnis and Ostrom 2014). It has become increasingly clear that there is a tension between these two goals (Thiel et al. 2015, Partelow 2018). The contextual adaptability of the SESF has been empirically demonstrated (Partelow 2018) and is arguably its core strength, but so far there has been little progress in building synthetic and cumulative SES knowledge from across empirical SESF cases (Schlager and Cox 2018, Villamayor-Tomas et al. 2020). Social-ecological systems frameworks’ study comparability has been challenged by inconsistent applications, interpretations, definitions, and measures (Cumming et al. 2020), which may be exacerbated by the lack of clear procedures or guidance for how to actually apply the SESF (Partelow 2018). Our methodological guide attempts to address this by providing a set of steps or decisions that encourage researchers to critically reflect upon and provide transparency regarding these methodological decisions, which can improve both contextualized study designs while enabling cross-study comparability without limiting flexibility. In the following sections, we discuss the above trends and gaps in the reviewed literature and reflect on how they have influenced our presentation of the guide, which emphasizes transparency over rigid procedure. Transparency emerged as the key issue during the review and coding process when we noted inconsistencies in documenting what we viewed as key methodological decisions in applying the SESF.

Methods used in the SESF literature are highly heterogeneous

Quantitative applications of the SESF are highly heterogeneous. Two non-mutually exclusive perspectives can be considered. The SESF applications generally require interdisciplinary knowledge to operationalize the many variables, i.e., variable selection, data collection, data transformation, analysis, etc. The framework is also applied to understand different contextual problems. Thus, researchers will choose different methodological strategies because there is no current guide or template. More applications may be needed until a reasonable saturation point of studies applying similar methods can be meaningfully compared within contexts.

Using quantitative data is typically employed to facilitate hypothesis testing, prediction, and forecasting. The majority of reviewed publications relied heavily on explanatory/outcome variable analysis methods such as linear and logistic regression techniques. However, several publications in this review noted the limitation of these methods in narrowing analyses of SESs to a series of linear pairwise relationships that often involve investigating the explanatory power of a wide range of social-ecological indicators on only a single or small number of dependent variables representing overall outcomes. Development of more experimental methods and large time-scale studies are needed to advance research into SES causal mechanisms (Table 6; Cumming et al. 2020). Methodological transparency is critically important when making theoretical jumps to generalizability, necessitating clarity and transparency regarding the causal inferences and variable relationships being reported (Villamayor-Tomas et al. 2020).

Social-ecological systems research and the SESF itself draw heavily from complex systems theory, conceptualizing SESs as components with a high degree of interaction or connections, forming a network with often nonlinear, dynamic, and emergent properties (Berkes et al. 2003, Ostrom 2009, Preiser et al. 2018). Despite this, previous critical reflections have identified a lack of SES research that empirically applies these concepts of complexity, such as modeling approaches that explore the connections, dynamics, and feedback effects within SESs rather than simply analyses of pairwise relations between variables (Pulver et al. 2018, Cumming et al. 2020, Gomez-Santiz et al. 2021). To be certain, the often data-scarce and open nature of many SES contexts can obscure attempts to explore the interdependent and interactive effects in more detail, and the SESF’s focus on variables rather than connections adds further ambiguity as to how researchers should conceptualize an SES (Pulver et al. 2018). Still, if we accept that complex systems have emergent properties, then it is clear that our SES methodological toolkit needs to explore ways to expand beyond sums of variable-outcome interactions and into methods that focus on capturing, rather than reducing, complexity. Several publications in our review explore promising analytical techniques in these directions, including agent-based modeling to test the emergent properties of individual actor and resource unit behavior on SES outcomes (Cenek and Franklin 2017, Lindkvist et al. 2017), supervised and unsupervised machine learning to analyze policy impacts on SESs (Rana and Miller 2019b) and assess spatial SES archetypes (Rocha et al. 2020), and system dynamics modeling to simulate SES dynamics under various scenarios (Baur and Binder 2015).

Integrative participatory methods, those which involve local actors in knowledge co-production and study design, are some of the most promising and feasible approaches for improving our understanding of SES complexity in information-scarce contexts. They can further lead to better forecasting and scenario building that inform policy and actionable change because of the embedded nature of knowledge creation and learning with those actors directly involved in social-ecological change processes (Eelderink et al. 2020, Caniglia et al. 2021). Notable approaches from our review include participatory fuzzy cognitive mapping to create SES dynamics models based on stakeholder knowledge (Ziegler et al. 2019) and prospective structural analysis to support SES scenario building (Delgado-Serrano et al. 2015). Such strategically designed integration may come at the cost of time and resources and may require a shared learning process to integrate differing knowledge systems and epistemologies (e.g., transdisciplinarity; Tengö et al. 2014, Norström et al. 2020). Nonetheless, it can promote stakeholder ownership and local study relevance while providing scientists with improved knowledge of important social and ecological components and processes within the SES (Reed et al. 2014, Fischer et al. 2015, Guerrero et al. 2018).

In calling for more transdisciplinary SES research, it is pertinent to consider the tension between case specificity and the need for comparability. This is because transdisciplinary and other knowledge co-production methods have been more often associated with case-specific research than that designed to allow generalizability across multiple cases. However, recent literature demonstrates that knowledge co-production approaches are increasingly being applied with decision makers working across multiple regions or even countries (Gurney et al. 2019). We do not view the need for broadly comparable SES research as being diametrically opposed to case-focused and problem-driven or action-oriented research. Although empirical applications are growing, published SESF research is still relatively scarce, and the sample becomes smaller still when subdivided into more granular categories such as methodological approach or sector (Appendix 1, Table A1.2; Partelow 2018). Although recent literature rightfully pushes for SES research to move beyond the exploration and into theory development (Cumming et al. 2020, Cox et al. 2021), we particularly emphasize the need for more (and more diverse) empirical SESF applications to identify patterns of both more broadly comparable, as well as more context specific, SES variables and interactions across cases. In their post-Ostrom agenda, Cumming et al. 2020 charted a path forward for theory-oriented SES research via “middle-range” theory development in which building explanations of highly complex SES phenomena might entail building partial theories with a bounded or contextual applicability rather than one all-encompassing SES theory. More highly detailed case-specific SES studies play an important building block in developing new hypotheses and theories to test (Guerrero et al. 2018), and “filling out” the SESF literature with more wide-ranging cases is needed for these bounded explanations to emerge. This will likely lead to not only bounded theories but also more bounded SES frameworks covering a more specific and comparable range of contexts, such as SES frameworks for specific resource sectors (Partelow 2018), governance arrangements, or geographic or social-cultural contexts.

The SES literature has made note of a number of gaps that limit the accumulation of knowledge from individual case studies to broader theoretical generalizations (Cox et al. 2021). Both syntheses of diverse case studies and large-scale comparative research projects are key for enabling empirically robust theory building, but current SESF literature struggles to do both (Partelow 2018). Additionally, although we identified 21 large-N comparative studies, most units of analysis were at the individual or local level (rather than, e.g., comparisons of multiple SES cases) and sampled within a limited spatial context (e.g., within one district), likely reducing the external validity beyond that context (Poteete et al. 2010). Only two reviewed studies applied large-N analyses to regional units of analysis, which has been identified as a critical and under-represented focal level of SES analysis (Rounsevell et al. 2012, Glaser and Glaeser 2014), suggesting that researchers are facing a challenge in creating broadly comparative SES research at larger spatial levels. To some extent this may reflect a collective action problem in scientific research itself, in which the collective goal of large-scale SES research may be offset by costs of coordination and collaboration, incentivizing smaller projects at the individual level (Cox et al. 2021). However, it also reflects trade-offs in study design between comparability and case-specificity, in which comparing a wider and more diverse range of SES contexts may necessitate measuring a more general list of broadly relevant variables, risking overgeneralization or missing key variables that are highly relevant but not to all cases (Gurney et al. 2019). Because the SESF itself is decomposable into multiple levels of generalization, one approach for large-N SES analyses is to compare a range of broad, universally relevant 2nd-tier variables across all SES cases, while also including more bounded and decomposed (e.g., 3rd-tier variables), which might be highly influential but only within a subset of cases (Gurney et al. 2019). Still, these approaches are likely to have high resource and coordination costs, suggesting the need for continued synthetic analysis of case-specific SESF research. Several reviewed studies synthesized secondary case databases to assess patterns across multiple SESs, however only one specifically synthesized patterns across existing empirical SESF studies, and this meta-analysis noted challenges regarding methodological transparency that limited the level of detail for case comparison (Villamayor-Tomas et al. 2020). It is evident from these patterns in the literature that further attention to methodological transparency and documentation in SESF studies is needed.

Methodological transparency issues: two main challenges

We identified continued ambiguity regarding 2nd-tier variable and measurable indicator selection as perhaps one of the most critical methodological challenges facing between-study SESF comparability and middle-range theory development. Methodological transparency is a broader academic challenge but should not necessarily be attributed to carelessness or negligence. A variety of reasons exist, ranging from scientific publishing standards regarding short and concise methods, journal word counts and formatting requirements, and procedural doubt or the “fear” of showing too much. Or, publications may simply have enough documentation to support the findings being presented, only lacking in certain explicit details at the meta-analytical level. Furthermore, many SESF publications are interdisciplinary, and methodological assumptions regarded as common knowledge in one field or discipline may need to be explained to scholars in another field in interdisciplinary journals. Regardless, we encourage SESF researchers to be as transparent as possible regarding the methodological steps we have outlined, such as making full use of supplementary materials to share these extra layers of methodological procedure (i.e., the choices at each step of the guide). Below we reflect on two specific transparency challenges identified in this review:

Transparency challenge 1: which 2nd-tier variables are being applied and why?

The SESF 2nd-tier variables lack clarity in how to conceptualize and measure them for a given case, and many researchers are finding it difficult and subjective to link their case SES data to the generalized concepts, which are the SESF 2nd-tier variables. Although the majority of surveyed authors stated that they understood how to identify relevant variables for a case, both publications and survey respondents noted recurring challenges regarding how to conceptualize or define the 2nd-tier variables within their specific case context, or how to categorize existing empirical and secondary data to specific variables. Importantly, the variable selection criteria in many studies is often unclear, which hinders learning in the research community, interpretability, and cross-case comparisons. One critical building block to SESF research is identifying which 2nd-tier variables are relevant or generalizable across specific SES contexts (McGinnis and Ostrom 2014). However, it is often unclear if the inclusion or exclusion of variables is deductive and theory driven (e.g., hypothesis-based), inductive (e.g., participatory evaluation), or because available secondary data aligns with particular variables. It could also be that certain variables are relevant across a larger number of cases, or that they are less abstract and easier to conceptualize and measure than others. Criteria for variable modifications including the inclusion of new variables are also often unclear and lacking justification (Partelow 2018). We argue that although there is no specifically right or wrong approach to applying the SESF variables, it is clear from our review that the lack of consistency and transparency limits both the ability to compare and contrast study findings with others.

Transparency challenge 2: how are 2nd-tier variables being measured?

To quantitatively measure abstract concepts, such as many of the 2nd-tier SESF variables, one or more empirically measurable indicators are required. Nearly all the variables could have many different possible indicators, such as RS5 - System productivity, in which indicators range from coastal chlorophyll levels, to kilograms of production of a resource unit, to average park visitation (Table 4). The context of those indicators presumably matters in each case, and the role that each plays in the case when abstracted to the broader concept of “system productivity”, may not mean the same thing outside of those contexts. Even indicators that appear similar on the surface may be representing different conceptual phenomena in the SES, such as A1, i.e., number of actors; different studies measure the number of relevant actors in terms of a raw population value, or as population density in a given spatial unit, or as a ratio of another population. Each measure informs us about the same concept in ways that might confer different insights or highlight different phenomena. Most surveyed researchers found it unclear how to select appropriate measurable indicators for the variables in their research (Fig. 2) and documentation of indicator selection was inconsistent in the reviewed literature. Indeed, indicator selection is an often messy process driven by data availability and feasibility. Numerous publications noted challenges in data scarcity (Budiharta et al. 2016, Lindkvist et al. 2017, Filbee-Dexter et al. 2018, Rana and Miller 2019b, Rocha et al. 2020), and studies are often relying on a wide range of primary and secondary sources to collect indicator data (Table 5), which may vary in structure, comprehensiveness, feasibility, and quality (Neumann and Graeff 2015). As such, research with the SESF is often by practical necessity relying on incomplete or low-quality data sources or using certain available data as proxies for other indicators. Transparency regarding how these decisions were made will help future researchers learn how to deal with those issues and enhance the interpretability of study findings.

Standardizing SES indicators is not a feasible or arguably desirable approach given the range of case contexts and research objectives across individual SESF studies. We rather encourage continued empirical applications so that patterns of context specific indicator measures may emerge, even when generalizability is not the core objective. Increased transparency regarding SESF variable and empirical indicator selection can aid in this cumulative accumulation of knowledge. As existing SESF studies are one of the most important references for researchers operationalizing the SESF variables in their work, we further suggest the development of a more comprehensive and accessible database of SESF variables and measurable indicators, such as the wiki-type format proposed by Cox et al. 2021 as an important path forward.

Applying the multi-step methodological guide to the SESF

This review builds on the methodological gaps identified by Partelow 2018, by providing a full methodological guide to the SESF. We see this guide as being supplemental to existing SESF guides in the literature, including guides for conceptualizing a case SES and related institutional and collective action challenges (Hinkel et al. 2015), for characterizing an SES at the local level (Delgado-Serrano and Ramos 2015), and for coevolving SESF research with sustainability science (Partelow 2016).

Our guide should be considered a multi-step, rather than step-by-step, procedure. We recognize that different research goals and researcher interests will align with different methodological trajectories. For example, a theory-driven researcher might first select the 2nd-tier variables and the hypotheses they expect to be important for collective action in their case SES, after which they might identify a set of measurable indicators, whereas another researcher applying a more inductive approach might apply participatory modeling methods to identify important SES factors and only in the analysis stage code these to the SESF variables. We see this flexibility as a strength of the framework, and although we present our methodological steps in what we interpret as a broadly logical order, we encourage researchers using this guide to answer these questions in the order that makes sense for their own research. The steps of this guide may best be interpreted as key “decision points” and questions that a researcher should be able to answer and clearly document with the long-term goal of building and improving comparable research with the SESF.

Although this guide was specifically developed around a review of quantitative applications of the SESF, we believe it is applicable to all future applications of the framework including qualitative approaches, and it may be able to inform SES studies beyond the SESF. Both quantitative and qualitative studies are critical for progressing the field. For example, descriptive SESF analyses have been found to often include case descriptions of a large range of variables that are then ignored in explanations of case outcomes, leading to confusion about which variables are actually relevant (Villamayor-Tomas et al. 2020). This also warrants some reflection by researchers on the anticipated level of generalizability of the research, where, in many cases, a more in-depth case study may simply be less focused on generalizability in lieu of a richer descriptive analysis of a specific context. Still, clear and formal narrative summaries answering the questions in this guide (even simple visual diagrams of the variable relationships identified, as suggested by Villamayor-Tomas et al. 2020) could improve generalizability and accessibility of SES findings for synthetic analysis even in cases where creating generalizable findings is not a priority, without compromising the depth of the overall analysis. Our guide was developed with an understanding of this current state of the SESF literature, and we expect more context-specific and potentially more standardized procedures to eventually develop based out of these more specialized versions of the SESF, similar to existing SESF modifications for marine aquaculture (Johnson et al. 2019), lobster and benthic small-scale fisheries (Basurto et al. 2013, Partelow and Boda 2015), urban stormwater management (Flynn and Davidson 2016) and food systems research (Marshall 2015).

CONCLUSION

Our review analyzed the step-by-step decisions scholars have made when applying the SESF with quantitative methods. With this review data, we have developed a multi-step methodological guide for new applications of the SESF, while also examining current trends and discussing challenges. Our guide and discussion aim to promote methodological transparency as the basis for enhancing comparability across publications and making diagnostic place-based research more meaningfully tailored to context. Still, our review found that researchers are finding it unclear how to apply the SESF to create comparable research, particularly in the areas of variable and indicator selection, and the methodological decisions being made within studies are often ambiguous. Although we noted a high degree of methodological heterogeneity in quantitative SESF applications, analyses are still skewed toward certain methods and case sectors. We call for more empirical applications of the SESF and encourage both methodological plurality and case diversity, alongside enhanced methodological transparency. In doing so, comparability and synthesis can emerge across varying methodological, theoretical, sector-specific, and other dimensions. We argue that this can move our understanding of SESs as complex adaptive systems forward and help resolve tensions between the need for contextual adaptability and the need for comparison.

LITERATURE CITED

Aaron MacNeil, M., and J. E. Cinner. 2013. Hierarchical livelihood outcomes among co-managed fisheries. Global Environmental Change 23(6):1393-1401. https://doi.org/10.1016/j.gloenvcha.2013.04.003

Aswani, S., G. G. Gurney, S. Mulville, J. Matera, and M. Gurven. 2013. Insights from experimental economics on local cooperation in a small-scale fishery management system. Global Environmental Change 23(6):1402–1409. https://doi.org/10.1016/j.gloenvcha.2013.08.003

Basurto, X., S. Gelcich, and E. Ostrom. 2013. The social-ecological system framework as a knowledge classificatory system for benthic small-scale fisheries. Global Environmental Change 23(6):1366-1380. https://doi.org/10.1016/j.gloenvcha.2013.08.001

Basurto, X., and E. Ostrom. 2009. The core challenges of moving beyond Garrett Hardin. Journal of Natural Resources Policy Research 1(3):255-259. https://doi.org/10.1080/19390450903040447

Baur, I., and C. R. Binder. 2015. Modeling and assessing scenarios of common property pastures management in Switzerland. Ecological Economics 119:292-305. https://doi.org/10.1016/j.ecolecon.2015.09.019

Bennett, N. J., R. Roth, S. C. Klain, K. M. A. Chan, D. A. Clark, G. Cullman, G. Epstein, M. P. Nelson, R. Stedman, T. L. Teel, R. E. W. Thomas, C. Wyborn, D. Curran, A. Greenberg, J. Sandlos, and D. Veríssimo. 2016. Mainstreaming the social sciences in conservation. Conservation Biology 31(1):56-66. https://doi.org/10.1111/cobi.12788

Berkes, F., J. Colding, and C. Folke. 2003. Navigating social-ecological systems: building resilience for complexity and change. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511541957

Budiharta, S., E. Meijaard, J. A. Wells, N. K. Abram, and K. A. Wilson. 2016. Enhancing feasibility: incorporating a socio-ecological systems framework into restoration planning. Environmental Science and Policy 64:83-92. https://doi.org/10.1016/j.envsci.2016.06.014

Caniglia, G., C. Luederitz, T. von Wirth, I. Fazey, B. Martín-López, K. Hondrila, A. König, H. von Wehrden, N. A. Schäpke, M. D. Laubichler, and D. J. Lang. 2021. A pluralistic and integrated approach to action-oriented knowledge for sustainability. Nature Sustainability 4(2):93-100. https://doi.org/10.1038/s41893-020-00616-z

Cenek, M., and M. Franklin. 2017. An adaptable agent-based model for guiding multi-species Pacific salmon fisheries management within a SES framework. Ecological Modelling 360:132-149. https://doi.org/10.1016/j.ecolmodel.2017.06.024

Christou, M., V. Sgardeli, A. C. Tsikliras, G. Tserpes, and K. I. Stergiou. 2020. A probabilistic model that determines the social ecological system (SES) attributes that lead to successful discard management. Reviews in Fish Biology and Fisheries 30(1):109–119. https://doi.org/10.1007/s11160-020-09593-0

Cinner, J. E., T. R. McClanahan, M. Aaron MacNeil, N. A. J. Graham, T. M. Daw, A. Mukminin, D. A. Feary, A. L. Rabearisoa, A. Wamukota, N. Jiddawi, S. J. Campbell, A. H. Baird, F. A. Januchowski-Hartley, S. Hamed, R. Lahari, T. Morove, and J. Kuange. 2012. Comanagement of coral reef social-ecological systems. Proceedings of the National Academy of Sciences 109(14):5219-5222. https://doi.org/10.1073/pnas.1121215109

Cox, M., G. G. Gurney, J. M. Anderies, E. Coleman, E. Darling, G. Epstein, U. J. Frey, M. Nenadovic, E. Schlager, and S. Villamayor-Tomas. 2021. Lessons learned from synthetic research projects based on the Ostrom Workshop frameworks. Ecology and Society 26(1):17. https://doi.org/10.5751/ES-12092-260117

Cox, M., S. Villamayor-Tomas, N. C. Ban, G. Epstein, L. Evans, F. Fleischman, M. Nenadovic, G. A. Garcia-Lopez, F. van Laerhoven, C. Meek, I. P. Ibarra, and M. Schoon. 2020. From concepts to comparisons: a resource for diagnosis and measurement in social-ecological systems. Environmental Science and Policy 107:211-216. https://doi.org/10.1016/j.envsci.2020.02.009

Cox, M., S. Villamayor-Tomas, G. Epstein, L. Evans, N. C. Ban, F. Fleischman, M. Nenadovic, and G. Garcia-Lopez. 2016. Synthesizing theories of natural resource management and governance. Global Environmental Change 39:45-56. https://doi.org/10.1016/j.gloenvcha.2016.04.011

Cumming, G. S., G. Epstein, J. M. Anderies, C. I. Apetrei, J. Baggio, Ö. Bodin, S. Chawla, H. S. Clements, M. Cox, L. Egli, G. G. Gurney, M. Lubell, N. Magliocca, T. H. Morrison, B. Müller, R. Seppelt, M. Schlüter, H. Unnikrishnan, S. Villamayor-Tomas, and C. M. Weible. 2020. Advancing understanding of natural resource governance: a post-Ostrom research agenda. Current Opinion in Environmental Sustainability 44:26-34. https://doi.org/10.1016/j.cosust.2020.02.005

Dancette, R., and L. Sebastien. 2019. The actor in 4 dimensions: a relevant methodology to analyze local environmental governance and inform Ostrom’s social-ecological systems framework. MethodsX 6:1798–1811. https://doi.org/10.1016/j.mex.2019.07.025

Delgado-Serrano, M. del M., E. Oteros-Rozas, P. Vanwildemeersch, C. Ortíz-Guerrero, S. London, and R. Escalante. 2015. Local perceptions on social-ecological dynamics in Latin America in three community-based natural resource management systems. Ecology and Society 20(4):24. https://doi.org/10.5751/ES-07965-200424

Delgado-Serrano, M. del M., and P. Ramos. 2015. Making Ostrom’s framework applicable to characterise social ecological systems at the local level. International Journal of the Commons 9(2):808-830. https://doi.org/10.18352/ijc.567

Dressel, S., G. Ericsson, and C. Sandström. 2018. Mapping social-ecological systems to understand the challenges underlying wildlife management. Environmental Science and Policy 84:105-112. https://doi.org/10.1016/j.envsci.2018.03.007

Eelderink, M., J. M. Vervoort, and F. van Laerhoven. 2020. Using participatory action research to operationalize critical systems thinking in social-ecological systems. Ecology and Society 25(1):16. https://doi.org/10.5751/ES-11369-250116

Epstein, G., J. M. Vogt, S. K. Mincey, M. Cox, and B. Fischer. 2013. Missing ecology: integrating ecological perspectives with the social-ecological system framework. International Journal of the Commons 7(2):432-453. https://doi.org/10.18352/ijc.371

Filbee-Dexter, K., C. C. Symons, K. Jones, H. A. Haig, J. Pittman, S. M. Alexander, and M. J. Burke. 2018. Quantifying ecological and social drivers of ecological surprise. Journal of Applied Ecology 55(5):2135-2146. https://doi.org/10.1111/1365-2664.13171

Fischer, J., T. A. Gardner, E. M. Bennett, P. Balvanera, R. Biggs, S. Carpenter, T. Daw, C. Folke, R. Hill, T. P. Hughes, T. Luthe, M. Maass, M. Meacham, A. V. Norström, G. Peterson, C. Queiroz, R. Seppelt, M. Spierenburg, and J. Tenhunen. 2015. Advancing sustainability through mainstreaming a social-ecological systems perspective. Current Opinion in Environmental Sustainability 14:144-149. https://doi.org/10.1016/j.cosust.2015.06.002

Flynn, C. D., and C. I. Davidson. 2016. Adapting the social-ecological system framework for urban stormwater management: the case of green infrastructure adoption. Ecology and Society 21(4):19. https://doi.org/10.5751/ES-08756-210419

Foster, T., and R. Hope. 2016. A multi-decadal and social-ecological systems analysis of community waterpoint payment behaviours in rural Kenya. Journal of Rural Studies 47(A):85-96. https://doi.org/10.1016/j.jrurstud.2016.07.026

Frey, U. J., and M. Cox. 2015. Building a diagnostic ontology of social-ecological systems. International Journal of the Commons 9(2):595-618. https://doi.org/10.18352/ijc.505

Fujitani, M. L., C. Riepe, T. Pagel, M. Buoro, F. Santoul, R. Lassus, J. Cucherousset, and R. Arlinghaus. 2020. Ecological and social constraints are key for voluntary investments into renewable natural resources. Global Environmental Change 63:102125. https://doi.org/10.1016/j.gloenvcha.2020.102125

Glaser, M., and B. Glaeser. 2014. Towards a framework for cross-scale and multi-level analysis of coastal and marine social-ecological systems dynamics. Regional Environmental Change 14(6):2039-2052. https://doi.org/10.1007/s10113-014-0637-5

Gomez-Santiz, F., M. Perevochtchikova, and D. Ezzine-de-Blas. 2021. Behind the scenes: scientific networks driving the operationalization of the social-ecological system framework. Science of the Total Environment 787:147473. https://doi.org/10.1016/j.scitotenv.2021.147473

Guba, E. G. 1981. Criteria for assessing the trustworthiness of naturalistic inquiries. ECTJ 29(2):75-91. https://doi.org/10.1007/BF02766777

Guerrero, A. M., N. J. Bennett, K. A. Wilson, N. Carter, D. Gill, M. Mills, C. D. Ives, M. J. Selinske, C. Larrosa, S. Bekessy, F. A. Januchowski-Hartley, H. Travers, C. A. Wyborn, and A. Nuno. 2018. Achieving the promise of integration in social-ecological research: a review and prospectus. Ecology and Society 23(3):38. https://doi.org/10.5751/ES-10232-230338

Gurney, G. G., J. E. Cinner, J. Sartin, R. L. Pressey, N. C. Ban, N. A. Marshall, and D. Prabuning. 2016. Participation in devolved commons management: multiscale socioeconomic factors related to individuals’ participation in community-based management of marine protected areas in Indonesia. Environmental Science and Policy 61:212-220. https://doi.org/10.1016/j.envsci.2016.04.015

Gurney, G. G., E. S. Darling, S. D. Jupiter, S. Mangubhai, T. R. McClanahan, P. Lestari, S. Pardede, S. J. Campbell, M. Fox, W. Naisilisili, N. A. Muthiga, S. D’agata, K. E. Holmes, and N. A. Rossi. 2019. Implementing a social-ecological systems framework for conservation monitoring: lessons from a multi-country coral reef program. Biological Conservation 240:108298. https://doi.org/10.1016/j.biocon.2019.108298

Haider, L. J., B. Neusel, G. D. Peterson, and M. Schlüter. 2019. Past management affects success of current joint forestry management institutions in Tajikistan. Environment, Development and Sustainability 21(5):2183-2224. https://doi.org/10.1007/s10668-018-0132-0

Hicks, C. C., C. Fitzsimmons, and N. V. C. Polunin. 2010. Interdisciplinarity in the environmental sciences: barriers and frontiers. Environmental Conservation 37(4):464-477. https://doi.org/10.1017/S0376892910000822

Hinkel, J., P. W. G. Bots, and M. Schlüter. 2014. Enhancing the Ostrom social-ecological system framework through formalization. Ecology and Society 19(3):51. https://doi.org/10.5751/ES-06475-190351

Hinkel, J., M. E. Cox, M. Schlüter, C. R. Binder, and T. Falk. 2015. A diagnostic procedure for applying the social-ecological systems framework in diverse cases. Ecology and Society 20(1):32. https://doi.org/10.5751/ES-07023-200132

Hoque, S. F., R. Hope, S. T. Arif, T. Akhter, M. Naz, and M. Salehin. 2019. A social-ecological analysis of drinking water risks in coastal Bangladesh. Science of the Total Environment 679:23–34. https://doi.org/10.1016/j.scitotenv.2019.04.359

Johnson, T. R., K. Beard, D. C. Brady, C. J. Byron, C. Cleaver, K. Duffy, N. Keeney, M. Kimble, M. Miller, S. Moeykens, M. Teisl, G. P. van Walsum, and J. Yuan. 2019. A social-ecological system framework for marine aquaculture research. ustainability 11(9):2522. https://doi.org/10.3390/su11092522

Kelly, R. P., A. L. Erickson, L. A. Mease, W. Battista, J. N. Kittinger, and R. Fujita. 2015. Embracing thresholds for better environmental management. Philosophical Transactions of the Royal Society B: Biological Sciences 370(1659):20130276. https://doi.org/10.1098/rstb.2013.0276

Klümper, F., and I. Theesfeld. 2017. The land-water-food nexus: expanding the social-ecological system framework to link land and water governance. Resources 6(3):28. https://doi.org/10.3390/resources6030028

Lam, D. P. M., M. E. Freund, J. Kny, O. Marg, M. Mbah, L. Theiler, M. Bergmann, B. Brohmann, D. J. Lang, and M. Schäfer. 2021. Transdisciplinary research: towards an integrative perspective. GAIA - Ecological Perspectives for Science and Society 30(4):243-249. https://doi.org/10.14512/gaia.30.4.7

Leslie, H. M., X. Basurto, M. Nenadovic, L. Sievanen, K. C. Cavanaugh, J. J. Cota-Nieto, B. E. Erisman, E. Finkbeiner, G. Hinojosa-Arango, M. Moreno-Báez, S. Nagavarapu, S. M. W. Reddy, A. Sánchez-Rodríguez, K. Siegel, J. J. Ulibarria-Valenzuela, A. H. Weaver, and O. Aburto-Oropeza. 2015. Operationalizing the social-ecological systems framework to assess sustainability. Proceedings of the National Academy of Sciences 112(19):5979-5984. https://doi.org/10.1073/pnas.1414640112

Lindkvist, E., X. Basurto, and M. Schlüter. 2017. Micro-level explanations for emergent patterns of self-governance arrangements in small-scale fisheries—A modeling approach. PLOS ONE 12(4):e0179439. https://doi.org/10.1371/journal.pone.0175532

Marshall, G. R. 2015. A social-ecological systems framework for food systems research: accommodating transformation systems and their products. International Journal of the Commons 9(2):881. https://doi.org/10.18352/ijc.587

McGinnis, M. D., and E. Ostrom. 2014. Social-ecological system framework: initial changes and continuing challenges. Ecology and Society 19(2):30. https://doi.org/10.5751/ES-06387-190230

Naiga, R., and M. Penker. 2014. Determinants of users’ willingness to contribute to safe water provision in rural Uganda. Lex localis - Journal of Local Self-Government 12(3):695–714. https://doi.org/10.4335/12.3.695-714(2014)

Neumann, R., and P. Graeff. 2015. Quantitative approaches to comparative analyses: data properties and their implications for theory, measurement and modelling. European Political Science 14(4):385-393. https://doi.org/10.1057/eps.2015.59

Norström, A. V., C. Cvitanovic, M. F. Löf, S. West, C. Wyborn, P. Balvanera, A. T. Bednarek, E. M. Bennett, R. Biggs, A. de Bremond, B. M. Campbell, J. G. Canadell, S. R. Carpenter, C. Folke, E. A. Fulton, O. Gaffney, S. Gelcich, J.-B. Jouffray, M. Leach, M. Le Tissier, B. Martín-López, E. Louder, M.-F. Loutre, A. M. Meadow, H. Nagendra, D. Payne, G. D. Peterson, B. Reyers, R. Scholes, C. I. Speranza, M. Spierenburg, M. Stafford-Smith, M. Tengö, S. van der Hel, I. van Putten, and H. Österblom. 2020. Principles for knowledge co-production in sustainability research. Nature Sustainability 3(3):182-190. https://doi.org/10.1038/s41893-019-0448-2

Okumu, B., and E. Muchapondwa. 2020. Determinants of successful collective management of forest resources: evidence from Kenyan community forest associations. Forest Policy and Economics 113:102122. https://doi.org/10.1016/j.forpol.2020.102122

Ostrom, E. 2007. A diagnostic approach for going beyond panaceas. Proceedings of the National Academy of Sciences 104(39):15181-15187. https://doi.org/10.1073/pnas.0702288104

Ostrom, E. 2009. A general framework for analyzing sustainability of social-ecological systems. Science 325(5939):419-422. https://doi.org/10.1126/science.1172133

Ostrom, E., and M. Cox. 2010. Moving beyond panaceas: a multi-tiered diagnostic approach for social-ecological analysis. Environmental Conservation 37(4):451-463. https://doi.org/10.1017/S0376892910000834

Osuka, K., S. Rosendo, M. Riddell, J. Huet, M. Daide, E. Chauque, and M. Samoilys. 2020. Applying a social-ecological systems approach to understanding local marine management trajectories in Northern Mozambique. Sustainability 12(9):3904. https://doi.org/10.3390/su12093904

Oviedo, A. F. P., and M. Bursztyn. 2016. The fortune of the commons: participatory evaluation of small-scale fisheries in the Brazilian Amazon. Environmental Management 57(5):1009-1023. https://doi.org/10.1007/s00267-016-0660-z

Partelow, S. 2016. Coevolving Ostrom’s social-ecological systems (SES) framework and sustainability science: four key co-benefits. Sustainability Science 11(3):399-410. https://doi.org/10.1007/s11625-015-0351-3

Partelow, S. 2018. A review of the social-ecological systems framework: applications, methods, modifications, and challenges. Ecology and Society 23(4):36. https://doi.org/10.5751/ES-10594-230436

Partelow, S. 2019. Analyzing natural resource governance with the social-ecological systems framework. Pages 65-93 in F. Nunan, editor. Governing renewable natural resources: theories and frameworks. Routledge, London, UK. https://doi.org/10.4324/9780429053009-4

Partelow, S., and C. Boda. 2015. A modified diagnostic social-ecological system framework for lobster fisheries: case implementation and sustainability assessment in Southern California. Ocean and Coastal Management 114:204-217. https://doi.org/10.1016/j.ocecoaman.2015.06.022

Partelow, S., M. Fujitani, V. Soundararajan, and A. Schlüter. 2019. Transforming the social-ecological systems framework into a knowledge exchange and deliberation tool for comanagement. Ecology and Society 24(1):15. https://doi.org/10.5751/ES-10724-240115

Partelow, S., A. Jäger, and A. Schlüter. 2021. Linking fisher perceptions to social-ecological context: mixed method application of the SES framework in Costa Rica. Human Ecology 49(2):187-203. https://doi.org/10.1007/s10745-021-00228-x

Partelow, S., P. Senff, N. Buhari, and A. Schlüter. 2018. Operationalizing the social-ecological systems framework in pond aquaculture. International Journal of the Commons 12(1):485-518. https://doi.org/10.18352/ijc.834

Poteete, A. R., M. A. Janssen, and E. Ostrom. 2010. Broadly comparative field-based research. Pages 64-88 in A. R. Poteete, M. A. Janssen, and E. Ostrom, editors. Working together: collective action, the commons and multiple methods in practice. Princeton University Press, Princeton, New Jersey, USA. https://doi.org/10.1515/9781400835157.64

Preiser, R., R. Biggs, A. De Vos, and C. Folke. 2018. Social-ecological systems as complex adaptive systems: organizing principles for advancing research methods and approaches. Ecology and Society 23(4):46. https://doi.org/10.5751/ES-10558-230446

Pulver, S., N. Ulibarri, K. L. Sobocinski, S. M. Alexander, M. L. Johnson, P. F. McCord, and J. Dell’Angelo. 2018. Frontiers in socio-environmental research: components, connections, scale, and context. Ecology and Society 23(3):23. https://doi.org/10.5751/ES-10280-230323

Queirós, A., D. Faria, and F. Almeida. 2017. Strengths and limitations of qualitative and quantitative research methods. European Journal of Education Studies 3(9):369-387.

Rana, P., and D. C. Miller. 2019a. Explaining long-term outcome trajectories in social-ecological systems. PLoS ONE 14(4):e0215230. https://doi.org/10.1371/journal.pone.0215230

Rana, P., and D. C. Miller. 2019b. Machine learning to analyze the social-ecological impacts of natural resource policy: insights from community forest management in the Indian Himalaya. Environmental Research Letters 14(2):024008. https://doi.org/10.1088/1748-9326/aafa8f

Reed, M. S., L. C. Stringer, I. Fazey, A. C. Evely, and J. H. J. Kruijsen. 2014. Five principles for the practice of knowledge exchange in environmental management. Journal of Environmental Management 146:337-345. https://doi.org/10.1016/j.jenvman.2014.07.021

Reyers, B., J. L. Nel, P. J. O’Farrell, N. Sitas, and D. C. Nel. 2015. Navigating complexity through knowledge coproduction: mainstreaming ecosystem services into disaster risk reduction. Proceedings of the National Academy of Sciences 112(24):7362-7368. https://doi.org/10.1073/pnas.1414374112

Rocha, J., K. Malmborg, L. Gordon, K. Brauman, and F. DeClerck. 2020. Mapping social-ecological systems archetypes. Environmental Research Letters 15(3):034017. https://doi.org/10.1088/1748-9326/ab666e

Roquetti, D. R., E. M. Moretto, and S. M. P. Pulice. 2017. Dam-forced displacement and social-ecological resilience: The Barra Grande hydropower plant in southern Brazil. Ambiente y Sociedade 20(3):115-134. https://doi.org/10.1590/1809-4422asoc153r2v2032017

Rounsevell, M. D. A., B. Pedroli, K.-H. Erb, M. Gramberger, A. G. Busck, H. Haberl, S. Kristensen, T. Kuemmerle, S. Lavorel, M. Lindner, H. Lotze-Campen, M. J. Metzger, D. Murray-Rust, A. Popp, M. Pérez-Soba, A. Reenberg, A. Vadineanu, P. H. Verburg, and B. Wolfslehner. 2012. Challenges for land system science. Land Use Policy 29(4):899-910. https://doi.org/10.1016/j.landusepol.2012.01.007

Schlager, E., and M. Cox. 2018. The IAD framework and the SES framework: an introduction and assessment of the Ostrom workshop frameworks. Pages in C. M. Weible and P. A. Sabatier, editors. Theories of the policy process. Routledge, New York, New York, USA. https://doi.org/10.4324/9780429494284-7

Schmitt-Harsh, M. L., and S. K. Mincey. 2020. Operationalizing the social-ecological system framework to assess residential forest structure: a case study in Bloomington, Indiana. Ecology and Society 25(2):14. https://doi.org/10.5751/ES-11564-250214

Sharma, D., I. Holmes, G. Vergara-Asenjo, W. N. Miller, M. Cunampio, R. B. Cunampio, M. B. Cunampio, and C. Potvin. 2016. A comparison of influences on the landscape of two social-ecological systems. Land Use Policy 57:499–513. https://doi.org/10.1016/j.landusepol.2016.06.018

Su, Y., E. Araral, and Y. Wang. 2020. The effects of farmland use rights trading and labor outmigration on the governance of the irrigation commons: evidence from China. Land Use Policy 91:104378. https://doi.org/10.1016/j.landusepol.2019.104378

Tengö, M., E. S. Brondizio, T. Elmqvist, P. Malmer, and M. Spierenburg. 2014. Connecting diverse knowledge systems for enhanced ecosystem governance: the multiple evidence base approach. Ambio 43(5):579-591. https://doi.org/10.1007/s13280-014-0501-3

Thiel, A., M. E. Adamseged, and C. Baake. 2015. Evaluating an instrument for institutional crafting: how Ostrom’s social-ecological systems framework is applied. Environmental Science and Policy 53:152-164. https://doi.org/10.1016/j.envsci.2015.04.020

van Laerhoven, F., M. Schoon, and S. Villamayor-Tomas. 2020. Celebrating the 30th Anniversary of Ostrom’s Governing the Commons: traditions and trends in the study of the commons, revisited. International Journal of the Commons 14(1):208-224. https://doi.org/10.5334/ijc.1030

Villamayor-Tomas, S., C. Oberlack, G. Epstein, S. Partelow, M. Roggero, E. Kellner, M. Tschopp, and M. Cox. 2020. Using case study data to understand SES interactions: a model-centered meta-analysis of SES framework applications. Current Opinion in Environmental Sustainability 44:48-57. https://doi.org/10.1016/j.cosust.2020.05.002

Vogt, J. M., G. B. Epstein, S. K. Mincey, B. C. Fischer, and P. McCord. 2015. Putting the “E” in SES: unpacking the ecology in the Ostrom social-ecological system framework. Ecology and Society 20(1):55. https://doi.org/10.5751/ES-07239-200155

Yandle, T., D. S. Noonan, and B. Gazley. 2016. Philanthropic support of aational parks: analysis using the social-ecological systems framework. Nonprofit and Voluntary Sector Quarterly 45(4). https://doi.org/10.1177/0899764016643612

Ziegler, J. P., S. E. Jones, and C. T. Solomon. 2019. Local stakeholders understand recreational fisheries as social-ecological systems but do not view governance systems as influential for system dynamics. International Journal of the Commons 13(2):1035-1048. https://doi.org/10.5334/ijc.945