INTRODUCTION

Complexity and uncertainty are inherent in social-ecological systems. Complex systems tend to have a large number of components and a high level of interconnectedness among them often involving nonlinear interactions and feedbacks that ultimately lead to complex behavior. Importantly, observable behaviors emerge as a result of these interactions and are not contained within the individual components alone (Waltner-Toews et al. 2008, Cilliers et al. 2013). Uncertainties are also pervasive. There may be an imperfect knowledge about the structure and behavior of a system, large natural variation across locations and years, errors in human observation, varying assumptions in the way data are used to provide broader inferences, and a lack of clarity about how a system should be managed (Suter et al. 1987, Morgan and Henrion 1990). For scientists and decision makers, these realities create challenges for understanding social-ecological systems and managing actions within their control by making it difficult to reliably identify pathways of cause and effect in the presence of large unknowns and confounding factors that are hard to isolate and to which it is hard to assign attribution. Despite these challenges, they cannot be a cause for delaying research and management given the urgent need for making decisions today.

Over recent decades, two strategies have become more widespread, in part, to address these challenges. Though framed in different ways for different purposes, a first strategy is systems thinking, which represents a more holistic approach to improve understanding and management of complex systems (Grant et al. 1997, Meadows 2008). It is based on the premise that the behavior of parts of a system can be better understood by examining it as a whole. Such a perspective underlies the growing interest in cumulative effects when undertaking environmental assessments (Duinker and Greig 2006, Canter and Ross 2010), moving away from single species management of fisheries toward ecosystem-based approaches (Browman and Stergiou 2004) and increasing consideration of current environmental stressors in frameworks for assessing vulnerability to future climate change (Staudt et al. 2013). Expert elicitation represents a second strategy that has seen growing use in situations in which complexities and uncertainties are high and there is the urgent need to make decisions despite these realities (Donlan et al. 2010, McDaniels et al. 2010, Teck et al. 2010, Martin et al. 2012, Wittmann et al. 2015). There are concerns that the judgments of experts can be biased or poorly calibrated (Tversky and Kahneman 1974); however, when these concerns are addressed, expert opinions can prove valuable for estimating model parameters, characterizing uncertainty, validating results from other studies, and filling knowledge gaps for decision making (Martin et al. 2012). In the context of complexity and uncertainty, systems thinking encourages a greater diversity of considerations to represent complex systems, whereas the consideration of expert judgments enables the inclusion of both “hard” forms of knowledge, i.e., based on scientific methods of observation and analysis, and “soft” forms of knowledge, i.e., based on individual judgments, priorities, and values, when faced with imperfect information.

Analytical approaches that explicitly involve systems thinking and expert judgments are embodied in various participatory modeling techniques (Lynam et al. 2007, Gray et al., in press). These approaches have in common a tendency to use conceptual models coupled with existing evidence and expert/stakeholder input to characterize the components and linkages in the particular system of interest. Individual approaches vary in the way they characterize relationships with varying levels of sophistication. Approaches range from conceptual models that provide qualitative representations of ecosystem interactions (DiGennaro et al. 2012), to semiquantitative approaches such as fuzzy cognitive mapping, which uses fuzzy logic to assign directions of influence and weights to relationships (Papageorgiou and Kontogianni 2012), to Bayesian belief networks, which use empirically or expert-derived probability distributions around interacting variables and their states of nature to provide probabilistic predictions of outcomes (Marcot et al. 2001).

In our experience, there can be limitations in applying such techniques in some situations. For instance, although these approaches provide a strong framework for encouraging a mechanistic representation of how a system functions, they can be limited in their ability to allow experts to quantify linkages based on a more holistic understanding of pathways within a system. Many field studies and experts’ mental models are based on knowledge derived from empirical relationships that tend to embed many individual linkages into more fulsome pathways of effect (Rastetter et al. 2003, Shochat et al. 2006). Thus, it can be helpful to elicit expert judgments in a more holistic way rather than a more reductionist approach that forces consideration of only a few individual components in isolation of interactions with other components. One approach to address this limitation is to use structured exercises in combination with a clear foundation of technical information. Through closed-ended questions, researchers can elicit standardized information from participants, thereby facilitating the combination of responses from several individuals (Hofer 1986) while still allowing them to make intuitive judgments about the system as a whole. In addition, multiattribute methods derived from the field of choice modeling allow analysts to jointly estimate parameters for individual components from more holistic evaluations (Louviere et al. 2000), adding value to box-arrow conceptual models by supplying numeric weights on the same interval scale (Flynn et al. 2007).

We explore the approaches and limitations, alongside the opportunities and challenges, of applying conceptual modeling and expert elicitation as ways of addressing complexity and uncertainty in a specific natural resource setting. In particular, we discuss the strategies and outcomes around conceptual model development that helped clarify understanding of the multiple and interacting influences of different human stressors on a guild of interest, migratory forest birds. We then discuss the elicitation strategies and information outputs derived from technical experts that helped strengthen the understanding of importance of different influences in the conceptual models. The ultimate objective of this research was to guide design and expansion of monitoring programs in northern Alberta, Canada, where it is important to understand the impacts of oil sands development on biodiversity (Gosselin et al. 2010, Environment Canada 2011, GOC 2012).

METHODS

The area of interest for this case study represents a complex social-ecological system in the boreal forest of northern Alberta, Canada (Fig. 1). Human activities include approximately 142,000 km² of oil sands deposits, as well as conventional oil and gas deposits, commercial forestry, agriculture, urbanization, a transportation network to support those industries, and other smaller economic interests (see Table A1.1 in Appendix 1). Large-scale and natural drivers such as an active fire regime, insect disturbance, and climate change also have an overarching influence. A rich biodiversity of terrestrial and freshwater species live in this landscape.

Conceptual model development

We began development of conceptual models with a literature review to identify key features and types of approaches to suit our needs. We selected model types that would best serve the various audiences, systems and processes of interest, levels of specificity, and information availability. To represent the complexity of the study area, we developed conceptual models at a hierarchy of scales ranging from ecosystem to landscape to guild to species levels. There was an intentional decrease in breadth and increase in specificity in these models moving from highest to lowest levels in the hierarchy. Models were developed with the intention of being both independent and interdependent with others such that higher level models informed lower level models.

Conceptual models were developed in five steps based on guidance from others (see Fig. 2; Grant et al. 1997, Noon 2002, Fischenich 2008). First, model objectives were clarified based on intended uses and audiences, which ranged from knowledgeable decision makers to ecologists and avian specialists. Next, models were bounded according to subsystems of interest and related spatial/temporal boundaries, which required clarifying the focus and level of specificity for each model. This clarity included understanding the development sectors, human activities, stressors, natural drivers, and valued ecosystem components represented in each model. A third step required specifying model components, in particular, summarizing the evidence pertaining to the drivers, outcomes, and linkages at relevant scales. Drivers included natural influences and human stressors that affect the behavior or state of the ecosystems’ components. Outcomes included the direct and indirect results, impacts, or consequences of particular drivers. Linkages represented the connections between drivers and outcomes, such that each linkage was associated with an “effect,” and a series of linkages from an initial driver to a final outcome was considered a “pathway of effect.” The fourth step included illustrating relationships among the drivers, outcomes, and linkages at each level in the hierarchy to develop graphical representations of each system. All models were mechanistic in nature to illustrate the sequence of causal linkages or pathways of effect between a driver and an outcome of interest. Finally, models were evaluated in several stages for comprehensiveness, consistency, robustness, and importance of components. These evaluation stages involved a combination of internal and external peer review, resulting in many iterations of the models to address the breadth and/or depth required. More details around conceptual model development are available in Appendix 1.

Engagement of technical experts

The first stages of conceptual model development were based on a review of readily available evidence describing the study area, in other words “hard” forms of knowledge. To effectively leverage and value the collective intelligence of experts, or “soft” forms of knowledge, we intentionally designed a process that engaged scientists in the advanced stages of model development to review, validate, refine, and develop “weightings” of importance around different components in the conceptual models.

To do so, we employed a variety of engagement strategies across model development stages (Fig. 2). A first point of input involved an internal review to refine and validate the models by experts who were avian ecologists internal to the government agency providing input to the broader monitoring program, yet naive to our model-building process. Next, a 2-day in-person workshop was convened to provide a “peer review” to improve the accuracy, relevance, and clarity of the models, as well as to provide guidance on developing a structured survey for eliciting judgments from scientists. Approximately 30 avian experts and ecologists from across North America were initially considered to participate in this research. These experts were from academia and government and had familiarity with conceptual modeling, monitoring, migratory birds, oil and gas development, other development impacts relevant to forest birds, or a combination thereof. We excluded industry representatives purposely given criticisms of influence noted by previously published reviews that have highlighted monitoring shortcomings in the oil sands area. The workshop was attended by 12 external experts. A background document that described the context for the work and conceptual models was distributed in advance of the workshop. Finally, a 1-day remote workshop was convened using a conference call, desktop sharing, and live online survey tools to engage 8 experts. These participants included avian experts and ecologists from academia, government, and nongovernmental organizations, many of whom had participated in the earlier in-person workshop. A financial incentive was offered to encourage participation and compensate experts for their time. A background document describing the meeting task process, forest bird model, and its components was distributed prior to the workshop. An overview of materials and an explanation of survey questions were provided at the meeting.

In advance of the remote workshop, a survey was developed as a structured process for eliciting expert judgments on the importance of different components and pathways of effect in the forest bird conceptual model. Prior to its deployment, the survey was piloted with two independent experts to test for clarity and accuracy in framing the questions. The survey was deployed using a uniquely designed website, given the need to provide access to and present a variety of complex technical information, as well as to quickly summarize results for immediate discussion during the workshop.

The survey was completed in three parts and consisted of five sections, incorporating several tools to enable participation, for instance closed-ended questions, the Delphi method, and “fist-to-five” voting. It was deliberately organized in a way that required respondents to consider increasing levels of specificity from broad to specific influences on migratory birds. The first part required participants to indicate their familiarity with boreal ecology, both across Canada and in the oil sands area; the sectors of development impacting the region; and five major guilds of forest birds, using a five-point scale from not at all familiar to extremely familiar. These questions were posed to assess the experience of the group and were deployed before engaging the group in the remote workshop.

The second part of the survey was completed during the remote workshop and focused on understanding the relative importance of breeding, migrating, and overwintering life stages to support regional population abundance and distribution of forest birds in the study area. These questions were posed to provide a better context for understanding the influence of impacts on the breeding range, the area of focus for monitoring and over which agencies have some degree of management control, relative to other life stages where no monitoring would occur and over which agencies have little influence. The relative importance of life stages was assessed using the analytical hierarchy process (AHP), which required presenting all pairwise comparisons of life stages and then asking experts to indicate which are more important and the degree of importance using a standardized five-point scale, ranging from equally important to extremely more important (Pavlikakis and Tsihrintzis 2003). Responses from multiple experts were combined by calculating the geometric mean of importance for each pair across all respondents. Expert ratings were then entered into a pairwise comparison matrix to facilitate calculation of weightings for each individual life stage. This part of the survey also included assessing the relative importance of 19 stressors that were identified as having a potential impact on quality of breeding habitats. These ratings were made using a 5-point importance scale, ranging from not at all important to extremely important.

The last part of the survey asked participants to rank the relative importance of each of 16 pathways of effect. Given the complexity of this task, we applied a maximum difference conjoint approach (MDC; Finn and Louviere 1992). Like AHP, the MDC is an iterative approach; however, it relies on experimental design principles to reduce the burden on respondents. With this approach, respondents were asked to identify the pathways they considered to be most and least important from 16 sets of 4 pathways. To ensure the independence of each parameter estimate, each combination of pathways was determined by an experimental design (Raktoe et al. 1981). To reduce the possibility of biases associated with learning or fatigue effects (Louviere et al. 2000), the order of questions was randomly assigned to each participant.

Statistical analysis of MDC surveys is grounded in random utility theory (McFadden 1974), which assumes that people choose the single option that maximizes their benefits. Under this assumption, the probability of an individual choosing one option (i) from a set of alternatives may follow a multinomial logit (MNL) function (Louviere and Woodworth 1983). In the case of the MDC, the choice probability of the least important pathway is assumed to be inversely related to its benefits (Cohen 2003). The resulting statistical model estimates underlying preferences by comparing each particular attribute value relative to a specified base. In our case, effects coding was used to center the resulting parameters around a mean of zero (Bech and Gyrd-Hansen 2005). The resulting model provided preference estimates on an interval scale for each pathway with constants accounting for the effect of list order (Cohen 2003).

The MDC portion of the survey was administered using the Delphi method (Linstone and Turoff 2002), in which experts were first asked to independently respond to a set of structured questions. Data were analyzed in real time with the group convened on the phone shortly afterward to review results and encouraged to share with others why and how they answered the questions. They were then asked to break away again to review their initial responses and allowed to adjust them based on what was heard from others.

Through this process, the experts identified two distinct types of pathways in the conceptual models that were not originally framed as such: pathways by which forest bird habitats are disturbed by various forms of human development and causal mechanisms related to the ecological responses of forest birds to habitat pressures. To address this feedback, additional questions were developed with responses completed approximately two weeks later. These additional questions required experts to rate the strength of influence and certainty of evidence related to the causal mechanisms underlying each of eight habitat pathways of effect. For each pathway, causal mechanisms were rated using five-point scales to represent the strength of influence, ranging from none to dominant, and certainty of evidence related to understanding the causal mechanism, which was rated as theoretically a concern, evidence is ambiguous, evidence is preliminary, evidence is strong, or widespread agreement. Across all parts of the survey, a fist-to-five voting system was used to gauge the level of agreement around the emerging responses from the group, with a fist representing a vote of no support and five fingers representing the strongest level of support.

RESULTS

Conceptual models

Fourteen conceptual models were developed at a hierarchy of scales to represent the complexity of the system: one ecosystem, two landscape, two bird guild, and nine species models. A systems model was used for the ecosystem level to illustrate the breadth of human stressors and natural drivers that influence the study area. A state and transition model was used for the landscape level to represent habitat states and transitional processes that influence habitat dynamics, whereas a life cycle model was used to represent population dynamics for the migratory and resident terrestrial species occupying the study area. Life cycle models were also used for the guild and species levels to represent interactions between the environment and all forest and wetland dependent birds that migrate annually from or through the study area. We focus on results related to the forest bird, i.e., guild-level, model because it provided the foundation for the structured survey exercises and determination of model weightings. Other models are provided in Appendix 1.

The forest bird model is provided in Figure 3. The top left portion shows an annual life cycle separated into key life stages: breeding, fall migration, overwintering, and spring migration. An inner ring represents the period until juveniles start breeding. An outer ring represents adult years of life with seven distinct aspects: fecundity, summer growth/condition, summer survival, fall migration survival, overwinter condition, overwinter survival, and spring migration survival. Survival and fecundity have direct influences on populations (arrows to the center), whereas changes in condition have indirect influences through their effect on subsequent life stages (arrows to other life stages). For instance, summer growth/condition affects summer survival, fall migration survival, and ultimately overwinter condition (denoted by arrow from summer growth to overwinter condition). Alternatively, fecundity is influenced both by the condition that birds are in when they return from overwintering and conditions on the breeding grounds themselves. Regional population outcomes, as opposed to continental outcomes, are the end points of interest because many migratory species have wide summer breeding ranges that are subject to different stressors in the boreal forest. Human stressors and natural drivers are illustrated in the right portion of the model. These influences have the potential for impacts on summer breeding, migration, and overwintering as mediated by pathways resulting from impacts on habitat, health, and behavior.

Broadscale monitoring is intended to provide information on the role and relative influence of various forcings on migratory birds: mining, forestry, agriculture, urbanization, and conventional oil and gas development, alongside oil sands development and an extensive transportation network to support these sectors. To facilitate this understanding, we adapted the International Union for Conservation of Nature threats classification system (Salafsky et al. 2008) to develop a list of 19 stressors as the fundamental agents of change in the study area (right side of Fig. 3, described in Table A1.3 in Appendix 1). An alignment of development sectors with these stressors provided a way of clarifying the many overlapping interactions with migratory birds (see Table A1.4 in Appendix 1). For example, “patch clearing” and “linear clearing” include the removal of surface vegetation in either a polygonal or linear pattern from forest harvesting, right-of-way development, and seismic lines. Major sources of these stressors include forest harvesting, oil sands development, conventional oil and gas, and the transportation network, with minor sources from mining, agriculture, and other forms of human use.

These stressors have direct and indirect impacts on habitats and/or individuals, which ultimately affect migratory birds (middle portion of Fig. 3). Impacts on habitat include loss, transformation, and degradation. Impacts on behavior and health include interference with a species’ ability to conduct its normal activities, especially foraging and movement among habitats, or may cause stress, injury, disease, malnutrition, and toxicity. Impacts on survival include different sources of human-induced mortality, e.g., intentional and incidental take, and natural mortality, e.g., predation, senescence/loss of vigor, disease, and starvation, which can also be altered by human-induced changes in habitat. Impacts on nesting are influenced by the availability of habitat and processes that interfere with successful nesting. Underlying mechanisms of influence include predation or parasitism of nests, destruction of nests by human actions or severe weather, detrimental changes in habitat quality, or disturbance as a result of human intrusion.

Because of its breadth, the forest bird model lacked the specificity required to inform the model weighting process with experts. Using this generic model, we identified 8 distinct habitat pathways of effect (Table 1) and 11 related causal mechanisms (Table 2) affecting life-stage and population-level responses. The habitat pathways of effect were distinguished according to their spatial scale of effect, type of habitat impact, and form of habitat disturbance based on the sector of origin. The full set of habitat pathways is summarized in more detail in Table A1.5 of Appendix 1.

Model weightings

Results from a series of structured exercises with experts were used to derive weightings of importance of different components in the forest bird model to inform priorities for monitoring. On a linear scale between 0 and 1, the weighting of 3 life stages indicated that experts, in aggregate, placed relatively more importance on pathways influencing breeding bird habitats (0.44) compared with those affecting overwintering habitats (0.35) and migratory life stages (0.21; see Fig. 4). A fist-to-five vote revealed a somewhat divergent and moderate level of support for these results with an average support of 3.1 out of 5 and votes ranging from 1 to 5. The support of experts was related to how strongly the combined weightings aligned with an individual’s original responses. This finding led to reanalysis of the data that revealed 2 contrasting opinions of experts, with group 1 weighting the breeding life stage as most important (0.63) and group 2 weighting overwintering habitats as most important (0.57).

On a linear 5-point scale from not at all important to extremely important, 19 stressors were rated by experts based on their anticipated impact on the quality of breeding habitats in the study area (Fig. 5). The most important stressor, patch clearing, was consistently rated by experts as extremely important because of the potential for direct loss of nesting sites. The next important stressors, rated as more than somewhat important, included vegetation extraction, water management, and linear clearing because of their impact on availability of nesting sites; pesticides, soil disturbance, and water pollution because of their impact on foraging activities and food sources; and linear infrastructure and traffic because of their direct impact on survival. All other stressors were rated as less than somewhat important, specifically human intrusion, introductions, animal extraction, noise, dust, light, air emissions, soil contamination, solid waste, and structures. For several stressors, there was a wide variation in responses across experts, which included vegetation extraction, linear clearing, pesticides, soil disturbance, and water management. A vote revealed a relatively strong and consistent level of support for these rankings across experts, with an average of 3.6 out of 5 and votes ranging from 2 to 4.

An MNL model was developed using data from the best–worst scaling exercise to estimate the importance of each pathway’s influence on abundance and distribution of migratory birds in the study area. Parameter estimates in the model were rescaled to derive an interval scale of importance ranging from 1 (least important) to 8 (most important). A process of ranking was repeated on 3 occasions to encourage experts to share information and learn from each other. Variation (standard error) in responses across experts declined, and the relative ranking of pathways changed with each iteration of the exercise (Fig. 6). For the final iteration, habitat alterations as a result of agriculture and forestry were rated as having the strongest relative importance, with effects at the stand level (pathways D and F) being more important than the landscape level (pathways E and G). Pathways associated with industrial development (pathways A and B) were rated as slightly less important than top-rated pathways, but significantly higher than stand-level impacts on habitat quality (pathway C) and landscape-level transformations as a result of changing fire dynamics (pathway H).

Despite minor variations, patterns in the relative strength of influence of different causal mechanisms were largely similar across the eight habitat pathways. Expert judgments on the most influential causal mechanisms underlying stand-level conversion as a result of industrial development (pathway A) are shown in Figure 7. Habitat-mediated mechanisms leading to displacement and either complete (mechanism a) or partial (mechanism b) loss were consistently rated as having a major influence. A reduction in fecundity (mechanism k) was seen as being moderately influential, whereas dispersal (mechanism c), food (mechanism i), direct mortality (mechanism j), and predation at varying ages (mechanisms d and e) were rated as having less influence. Parasitism (mechanism f), weather (mechanism g), and climate (mechanism h) were rated as having the weakest influence on this pathway.

DISCUSSION

Representing complexity

Developing visualizations to represent complexity of the study area was extremely challenging and time consuming. Although systems thinking encourages a more holistic representation of a system, the level of model completeness and complexity had to be balanced against the need for comprehension and simplicity to support clear communication. Technical experts demonstrate this dichotomy by advocating for simple conceptual models, yet simultaneously requiring that such models reflect observed complexities. More broadly, the realities of natural resource management also suggest that complex systems should be managed in a more holistic and integrated way but require simplification and prioritization to identify key variables that account for most changes (Mitchell et al. 2014). In our research, conceptual model development provided a clear framework for representing the ecosystem using a solid foundation of scientific evidence and expert opinion. As highlighted by the tenet “simplicity often lies on the other side of complexity” (E. Berlow, TED Talk, http://www.ted.com/talks/eric_berlow_how_complexity_leads_to_simplicity?language=en), the complexity of models evolved over these stages, initially increasing in complexity as the breadth and depth of models evolved and then decreasing as our understanding and artistry with which to represent the system improved (Fig. 2). No level of complexity was seen as best serving all audiences and purposes. Rather, multiple models with a range of complexities were developed to provide a strong foundation from which to structure the elicitation of expert judgments and to serve the communication needs of other audiences, meeting the primary purpose of using the models to inform the monitoring program design.

A hierarchy of conceptual models proved useful for characterizing the ecosystem because it forced a decrease in breadth and an increase in specificity at each level in the model and allowed for the use of different models at different scales. For instance, species-level models were the most detailed but also the easiest to develop because they were constrained in scope, grounded in the structure of the guild-level models, and supported with readily available summaries of evidence. The simultaneous development of our suite of models, from ecosystem level to species level, created several advantages. Internal coherence was the principle design advantage, especially the obvious linkages from the landscape- to the species-level models. This coherence helped enforce comprehensiveness and rigor across all models by ensuring a common typology and framework to accurately represent the many species, habitats, drivers, and stressors within the ecosystem.

The relatively detailed mechanistic structure of the models clarified the cumulative and interactive nature of factors influencing migratory birds both within and outside the study area. Within the study area, this approach emphasized how multiple industries produce the same types of stressors. As is required for most scientific investigations of cause and effect, these interactions emphasize the need to partition monitoring between affected and unaffected areas to attribute changes in population distribution and/or abundance to individual industries. However, the spatial extent of development, size of the study area, and the sometimes overlapping and sometimes spatially clustered land uses mean it will be difficult to find suitable reference areas to make regional population inferences. The approach also enabled a clear representation of carryover effects across life stages (Norris 2005) and the significant influence of factors outside of the oil sands area for one-half to three-quarters of the annual calendar for most migratory birds, including significant mortality during migration periods (Sillet and Holmes 2002). Although the mechanistic approach facilitated development of individual pathways of effect within the breeding season, which was needed to engage experts in the near term and inform priorities for the monitoring program in the longer term, a disadvantage is that few data are available to support partitioned modeling of the system in detail. Therefore, direct links with existing landscape simulation models incorporating birds are limited but are growing as an emerging property of the monitoring efforts achieved in the area (L. Mahon, personal communication).

Integrating expert judgments

Engagement of technical experts at selective points in our research facilitated and strengthened conceptual model development (Fig. 2). This approach was based on a view that the collective intelligence of many is superior to that of a few (Fisher and Fisher 1998). Compared with other audiences (Nelitz and Beardmore, in press), the perspectives of scientists required unique considerations given their strong level of technical understanding, the detail that is often required for them to provide meaningful input, and their sometimes poorly calibrated or self-serving opinions (Tversky and Kahneman 1974). These considerations affected our approach to engagement, the form of knowledge being elicited, and the way in which expert judgments were used. When combined with the conceptual models, these judgments were compatible with more traditional forms of scientific evidence. The participation of scientists helped improve and validate conceptual models in the early phases of development as expected from a traditional peer review. In the later stages, there was some initial reluctance among experts to use structured elicitation techniques to represent their judgments, in part because of some of the inherent challenges and biases that can be introduced into the results (Martin et al. 2012). Although efforts were made to address their concerns, experts tended to view the ecosystem in highly uncertain and in slightly different ways, meaning that it was not possible to frame the survey in a way that satisfied all perspectives. We were able to overcome one unanticipated challenge in the initial framing in which experts had difficulty disentangling the causal mechanisms from the pathways of effect that influence migratory birds. This difficulty led to adjustments in the design and iterative deployment of the MDC exercise. Despite these challenges, there were value-added benefits of formally integrating expert judgments because they provided additional information beyond the research and synthesis undertaken by the research team.

The use of structured survey exercises, i.e., pairwise comparisons, Likert scaling, and the MDC, provided significant advantages for integrating expert judgments. Foremost, we were able to tailor each task to the unique information inputs such that a quantitative weighting and measure of variance could be generated as an output. This approach allowed for the development of a common framework for discussing and analyzing the varied opinions of experts, each of whom brought different mental models and reference points to each task. For instance, despite similar access and knowledge of the available evidence, experts held divergent opinions about the importance of different life stages on population-level production of migratory birds in the study area (Fig. 4). A quantitative representation of this dichotomy was more potent than might have been revealed through less structured, i.e., dialogic, forms of engagement. A rating of stressors helped separate items into three bins of importance, although this was less informative for a finer level of discrimination given the variation in responses across experts (Fig. 5). An iterative MDC exercise was helpful in improving the level of agreement across the group and separating high-influence pathways as a result of forestry, agriculture, and industrial development from pathways leading to impacts on habitat quality and pathways related to natural processes such as wildfire (Fig. 6). Finally, a rating task to identify the strength of influence and certainty of evidence helped confirm the key mechanisms underlying many of the pathways of effect and illustrate a correlation in which mechanisms with more influence also tended to be rated as those with greater certainty of evidence (Fig. 7). Collectively, these exercises were helpful for quantifying the average opinion and variation in opinion across experts and providing measures of the relative weightings of importance of different influences on migratory birds.

In addition to the structured exercises, an innovative combination of supporting tools helped us effectively engage the group. A virtual meeting process, i.e., a conference call, online polling, desktop sharing, and a web-deployed survey, encouraged participation by limiting the travel required of a geographically distributed group of individuals from across North America. A financial incentive was also provided. Although its importance was largely seen as symbolic, and in some cases was not accepted, it helped individuals to prioritize this effort among the many other activities that impose constraints on their time. Enablers of participation were important given the level of specialization of experts required and the relatively limited number of individuals who could potentially contribute to this exercise. Such considerations are important when compared with public opinion surveys, in which the potential sample population is much larger. Deployment of the survey using a uniquely designed web platform provided us the capability of presenting an abundance of complex technical information and developing a common understanding of the study area with the group. This platform also provided us the flexibility to easily overcome an initial problem with the framing of one set of questions, leading to a reframing and rapid redeployment of the survey. The use of the Delphi method for one exercise harnessed the learning potential of the group and helped avoid individual biases by encouraging information sharing among experts and researchers. This approach led to less variation in responses across iterations of the exercise (Fig. 6). Finally, a consensus voting system was more informative than a simpler yes-no voting system because it facilitated expression of independent opinions and illuminated areas of agreement and disagreement across experts (Fig. 4). This combination of additional tools and techniques proved powerful for drawing on the collective intelligence of the group.

CONCLUSION

To date, this work has influenced the design and collection of hypothesis-driven research on cause-and-effect questions in the study area. Specifically, researchers have indicated that the models highlighted the importance of examining multisector and multistressor systems that overlap in space and time, articulating the nature of cumulative effects across sectors, rather than trying to isolate their investigations to oil and gas. Researchers have also focused on habitat disturbance on the breeding grounds because it was identified as a priority influence. The variation in expected responses highlighted that both species and community responses need to be investigated simultaneously. The models served an additional and important function of explaining to managers many aspects of the generic nature of disturbances from multiple industries, e.g., linear features, that cannot necessarily be compartmentalized as an “oil sands” effect versus other active industries. These conceptual models have been as much a communication tool as a touchstone to guide research.

The application of our research to the oil sands of northern Alberta, Canada, provides interesting insights that are relevant to other complex social-ecological settings. The study area features a naturally dynamic and biodiverse landscape overlaid with a high level of human development and economic intensity from a wide range of sectors, alongside contentious political and stakeholder oversight. Although there is a large quantity of specific information about the study area, there remains a relatively weak understanding about how the system functions, as highlighted by criticisms of past monitoring that was unable to detect significant effects (Dowdeswell et al. 2010, Kelly et al. 2010, Kvisle et al. 2011) and unresolved disputes around the impacts of oil sands on biodiversity in the region (Wasser et al. 2011, 2012, Boutin et al. 2012). Some combination of these factors has led to commitments by government to provide “a world class monitoring program for the oil sands to provide assurance of environmentally responsible development of the resource” with specific objectives to “support sound decision-making,” “ensure transparency,” and “enhance science-based monitoring,” which are ideal aspirations for any effective monitoring program (GOC 2012:2).

Resource development in Canada and elsewhere is being more thoroughly scrutinized, requiring scientists to provide strong evidence and managers to make difficult decisions in increasingly complex settings where “facts are uncertain, values in dispute, stakes high, and decisions urgent” (Funtowicz and Ravetz 1992, as cited in Jones 2011:15). In such situations, there is an unavoidable need to address challenges of complexity and uncertainty. With this case study, addressing these challenges to inform priorities for monitoring required representing the complexity of the system to provide a more fulsome view of the breadth and depth of influences affecting migratory birds, as well as integrating expert judgments to help address the incomplete understanding about system behavior. In our view, this context enabled the use of conceptual modeling and expert elicitation as ways of improving transparency and increasing the diversity of input to support development of a more scientifically defensible monitoring program. The combination of these strategies was innovative and proved useful.

LITERATURE CITED

Bech, M., and D. Gyrd-Hansen. 2005. Effects coding in discrete choice experiments. Health Economics 14(10):1079-1083. http://dx.doi.org/10.1002/hec.984

Boutin, S., M. S. Boyce, M. Hebblewhite, D. Hervieux, K. H. Knopff, M. C. Latham, A. D. M. Latham, J. Nagy, D. Seip, and R. Serrouya. 2012. Why are caribou declining in the oil sands? Frontiers in Ecology and the Environment 10(2):65-67. http://dx.doi.org/10.1890/12.WB.005

Browman, H. I., and K. I. Stergiou, editors. 2004. Perspectives on ecosystem-based approaches to the management of marine resources. Marine Ecology Progress Series 274:269-303. http://dx.doi.org/10.3354/meps274269

Canter, L., and B. Ross. 2010. State of practice of cumulative effects assessment and management: the good, the bad and the ugly. Impact Assessment and Project Appraisal 28(4):261-268. http://dx.doi.org/10.3152/146155110x12838715793200

Cilliers, P., H. C. Biggs, S. Blignaut, A. G. Choles, J. S. Hofmeyr, G. P. W. Jewitt, and D. J. Roux. 2013. Complexity, modeling, and natural resource management. Ecology and Society 18(3): 1. http://dx.doi.org/10.5751/es-05382-180301

Cohen, S. H. 2003. Maximum difference scaling: improved measures of importance and preference for segmentation. Research Paper Series. Sawtooth Software Inc., Sequim, Washington, USA.

DiGennaro, B., D. Reed, C. Swanson, L. Hastings, Z. Hymanson, M. Healey, S. Siegel, S. Cantrell, and B. Herbold. 2012. Using conceptual models and decision-support tools to guide ecosystem restoration planning and adaptive management: an example from the Sacramento–San Joaquin Delta, California. San Francisco Estuary & Watershed Science 10(3):1-15.

Donlan, C. J., D. K. Wingfield, L. B. Crowder, and C. Wilcox. 2010. Using expert opinion surveys to rank threats to endangered species: a case study with sea turtles. Conservation Biology 24(6):1586-1595. http://dx.doi.org/10.1111/j.1523-1739.2010.01541.x

Dowdeswell, L., P. Dillon, S. Ghoshal, A. Miall, J. Rasmussen, and J. P. Smol. 2010. A foundation for the future: building an environmental monitoring system for the oil sands. Environment Canada, Ottawa, Ontario, Canada.

Duinker, P. N., and L. A. Greig. 2006. The impotence of cumulative effects assessment in Canada: ailments and ideas for redeployment. Environmental Management 37(2):153-161. http://dx.doi.org/10.1007/s00267-004-0240-5

Environment Canada. 2011. Integrated monitoring plan for the oil sands, terrestrial biodiversity component. Government of Canada, Ottawa, Ontario, Canada.

Finn, A., and J. J. Louviere. 1992. Determining the appropriate response to evidence of public concern: the case of food safety. Journal of Public Policy & Marketing 11:12-25.

Fischenich, J. C. 2008. The application of conceptual models to ecosystem restoration. Technical Note EMRRP-EBA-01. U.S. Army Engineer Research and Development Center, Vicksburg, Mississippi, USA.

Fisher, K., and M. D. Fisher. 1998. The distributed mind: achieving high performance through the collective intelligence of knowledge work teams. AMACOM, New York, New York, USA.

Flynn, T. N., J. J. Louviere, T. J. Peters, and J. Coast. 2007. Best–worst scaling: what it can do for health care research and how to do it. Journal of Health Economics 26:171-189. http://dx.doi.org/10.1016/j.jhealeco.2006.04.002

Gosselin, P., S. E. Hrudey, M. A. Naeth, A. Plourde, R. Therrien, G. Van Der Kraak, and Z. Xu. 2010. The Royal Society of Canada expert panel: environmental and health impacts of Canada’s oil sands industry. Royal Society of Canada, Ottawa, Ontario, Canada.

Government of Canada (GOC). 2012. Joint Canada-Alberta implementation plan for oil sands monitoring. GOC, Ottawa, Ontario, Canada.

Grant, W. E., E. K. Pedersen, and S. L. Marin. 1997. Ecology and natural resource management: systems analysis and simulation. John Wiley and Sons, New York, New York, USA.

Gray, S., M. Paolisso, R. Jordan, and S. Gray, editors. In press. Environmental modeling with stakeholders. Springer International, Cham, Switzerland.

Hofer, E. 1986. On surveys of expert opinion. Nuclear Engineering and Design 93(2-3):153-160. http://dx.doi.org/10.1016/0029-5493(86)90214-1

Jones, H. 2011. Taking responsibility for complexity: how implementation can achieve results in the face of complex problems. Working Paper 330. Overseas Development Institute, London, UK.

Kelly, E. N., D. W. Schindler, P. V. Hodson, J. W. Short, R. Radmanovich, and C. C. Nielsen. 2010. Oil sands development contributes elements toxic at low concentrations to the Athabasca River and its tributaries. Proceedings of the National Academy of Sciences of the United States of America 107:16178-16183. http://dx.doi.org/10.1073/pnas.1008754107

Kvisle, H., J. Doucet, W. Kindierski, A. Miall, D. Pryce, J. Rasmussen, G. Taylor, H. Tennant, R. Wallace, H. Wheater, and D. Williams. 2011. A world class environmental monitoring, evaluation and reporting system for Alberta: the report of the Alberta Environmental Monitoring Panel. Alberta Environmental Monitoring Panel, Edmonton, Alberta, Canada.

Linstone, H. A., and M. Turoff, editors. 2002. The Delphi method: techniques and applications. Addison-Wesley, Reading, Massachusetts, USA.

Louviere, J. J., D. A. Hensher, and J. D. Swait. 2000. Stated choice methods: analysis and applications. Cambridge University Press, Cambridge, UK. http://dx.doi.org/10.1017/cbo9780511753831

Louviere, J. J., and G. Woodworth. 1983. Design and analysis of simulated consumer choice or allocation experiments: an approach based on aggregate data. Journal of Marketing Research 20:350-367. http://dx.doi.org/10.2307/3151440

Lynam, T., W. De Jong, D. Sheil, T. Kusumanto, and K. Evans. 2007. A review of tools for incorporating community knowledge, preferences, and values into decision making in natural resources management. Ecology and Society 12(1): 5. [online] URL: http://www.ecologyandsociety.org/vol12/iss1/art5/

Marcot, B. G., R. S. Holthausen, M. G. Raphael, M. M. Rowland, and M. J. Wisdom. 2001. Using Bayesian belief networks to evaluate fish and wildlife population viability under land management alternatives from an environmental impact statement. Forest Ecology and Management 153:29-42. http://dx.doi.org/10.1016/S0378-1127(01)00452-2

Martin, T. G., M. A. Burgman, F. Fidler, P. M. Kuhnert, S. Low-Choy, M. McBride, and K. Mengersen. 2012. Eliciting expert knowledge in conservation science. Conservation Biology 26(1):29-38. http://dx.doi.org/10.1111/j.1523-1739.2011.01806.x

McDaniels, T., S. Wilmot, M. Healey, and S. Hinch. 2010. Vulnerability of Fraser River sockeye salmon to climate change: a life cycle perspective using expert judgements. Journal of Environmental Management 91:2771-2780. http://dx.doi.org/10.1016/j.jenvman.2010.08.004

McFadden, D. 1974. Conditional logit analysis of qualitative choice behaviour. Pages 105-142 in P. Zarembka, editor. Frontiers in econometrics. Academic Press, New York, New York, USA.

Meadows, D. H. 2008. Thinking in systems: a primer. Chelsea Green, White River Junction, Vermont, USA.

Mitchell, B., K. Bellette, and S. Richardson. 2014. ‘Integrated’ approaches to water and natural resources management in South Australia. International Journal of Water Resources Development, in press. http://dx.doi.org/10.1080/07900627.2014.979399

Morgan, M. G., and M. Henrion. 1990. Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, New York, New York, USA. http://dx.doi.org/10.1017/cbo9780511840609

Nelitz, M. A., and B. Beardmore. In press. Eliciting judgments, priorities, and values using structured survey methods. In S. Gray, M. Paolisso, R. Jordan, and S. Gray, editors. Environmental modeling with stakeholders. Springer International, Cham, Switzerland.

Noon, B. R. 2002. Conceptual issues in monitoring ecological resources. Pages 27-72 in D. E. Busch and J. C. Trexler, editors. Monitoring ecosystems: interdisciplinary approaches for evaluating ecoregional initiatives. Island, Washington, D.C., USA.

Norris, D. R. 2005. Carry-over effects and habitat quality in migratory populations. Oikos 109:178-186. http://dx.doi.org/10.1111/j.0030-1299.2005.13671.x

Papageorgiou, E., and A. Kontogianni. 2012. Using fuzzy cognitive mapping in environmental decision making and management: a methodological primer and an application. Pages 427-450 in S. S. Young and S. E. Silvern, editors. International perspectives on global environmental change. InTech, Rijeka, Croatia. http://dx.doi.org/10.5772/29375

Pavlikakis, G. E., and V. A. Tsihrintzis. 2003. A quantitative method for accounting human opinion, preferences and perceptions in ecosystem management. Journal of Environmental Management 68(2):193-205. http://dx.doi.org/10.1016/s0301-4797(03)00067-7

Raktoe, B. L., A. Hedayat, and W. T. Federer. 1981. Factorial designs. Wiley, New York, New York, USA.

Rastetter, E. B., J. D. Aber, D. P. C. Peters, D. S. Ojima, and I. C. Burke. 2003. Using mechanistic models to scale ecological processes across space and time. BioScience 53(1):68-76. http://dx.doi.org/10.1641/0006-3568(2003)053[0068:ummtse]2.0.co;2

Salafsky, N., D. Salzer, A. J. Stattersfield, C. Hilton-Taylor, R. Neugarten, S. H. M. Butchart, B. Collen, N. Cox, L. L. Master, S. O’Connor, and D. Wilkie. 2008. A standard lexicon for biodiversity conservation: unified classifications of threats and actions. Conservation Biology 22(4):897-911. http://dx.doi.org/10.1111/j.1523-1739.2008.00937.x

Shochat, E., P. S. Warren, S. H. Faeth, N. E. McIntyre, and D. Hope. 2006. From patterns to emerging processes in mechanistic urban ecology. Trends in Ecology & Evolution 21(4):186-191. http://dx.doi.org/10.1016/j.tree.2005.11.019

Sillett, T. S., and R. T. Holmes. 2002. Variation in survivorship of a migratory songbird throughout its annual cycle. Journal of Animal Ecology 71:296-308. http://dx.doi.org/10.1046/j.1365-2656.2002.00599.x

Staudt, A., A. K. Leidner, J. Howard, K. A. Brauman, J. S. Dukes, L. J. Hansen, C. Paukert, J. Sabo, and L. A. Solórzano. 2013. The added complications of climate change: understanding and managing biodiversity and ecosystems. Frontiers in Ecology and the Environment 11(9):494-501. http://dx.doi.org/10.1890/120275

Suter, G. W., II, L. W. Barnthouse, and R. V. O’Neill. 1987. Treatment of risk in environmental impact assessment. Environmental Management 11(3):295-303. http://dx.doi.org/10.1007/bf01867157

Teck, S. J., B. S. Halpern, C. V. Kappel, F. Micheli, K. A. Selkoe, C. M. Crain, R. Martone, C. Shearer, J. Arvai, B. Fischhoff, G. Murray, R. Neslo, and R. Cooke. 2010. Using expert judgment to estimate marine ecosystem vulnerability in the California Current. Ecological Applications 20(5):1402-1416. http://dx.doi.org/10.1890/09-1173.1

Tversky, A., and D. Kahneman. 1974. Judgment under uncertainty: heuristics and biases. Science 185(4157):1124-1131. http://dx.doi.org/10.1126/science.185.4157.1124

Waltner-Toews, D., J. J. Kay, and N.-M. E. Lister, editors. 2008. The ecosystem approach: complexity, uncertainty, and managing for sustainability. Columbia University Press, New York, New York, USA.

Wasser, S. K., J. L. Keim, M. L. Taper, and S. R. Lele. 2011. The influences of wolf predation, habitat loss, and human activity on caribou and moose in the Alberta oil sands. Frontiers in Ecology and the Environment 9:546-551. http://dx.doi.org/10.1890/100071

Wasser, S. K., J. L. Keim, M. L. Taper, and S. R. Lele. 2012. To kill or not to kill – that is the question. Frontiers in Ecology and the Environment 10:67-68. http://dx.doi.org/10.1890/12.wb.006

Wittmann, M. E., R. M. Cooke, J. D. Rothlisberger, E. S. Rutherford, H. Zhang, D. M. Mason, and D. M. Lodge. 2015. Use of structured expert judgment to forecast invasions by bighead and silver carp in Lake Erie. Conservation Biology 29(1):187-197. http://dx.doi.org/10.1111/cobi.12369