Marmorek, David; Peters, Calvin

doi:doi:10.5751/ES-00310-050208

Finding a PATH toward Scientific Collaboration: Insights from the Columbia River Basin

Observed declines in the Snake River basin salmon stocks, listed under the U.S. Endangered Species Act (ESA), have been attributed to multiple causes: the hydrosystem, hatcheries, habitat, harvest, and ocean climate. Conflicting and competing analyses by different agencies led the National Marine Fisheries Service (NMFS) in 1995 to create the Plan for Analyzing and Testing Hypotheses (PATH), a collaborative interagency analytical process. PATH included about 30 fisheries scientists from a dozen agencies, as well as independent participating scientists and a technical facilitation team. PATH had some successes and some failures in meeting its objectives. Some key lessons learned from these successes and failures were to: (1) build trust through independent technical facilitation and multiple levels of peer review (agency scientists, independent participating scientists and an external Scientific Review Panel); (2) clarify critical uncertainties by developing common data sets, detailed sensitivity analyses, and thorough retrospective analyses of the weight of evidence for key alternative hypotheses; (3) clarify advice to decision makers by using an integrated life cycle model and decision analysis framework to evaluate the robustness of potential recovery actions under alternative states of nature; (4) involve key senior scientists with access to decision makers; (5) work closely with policy makers to clearly communicate analyses in nontechnical terms and provide input into the creation of management alternatives; and (6) recognize the trade-off between collaboration and timely completion of assignments.

KEY WORDS: adaptive management, analytical framework, collaborative process, Columbia River, decision analysis, endangered species, hydrosystem, multi-agency research, salmon management, Snake River.

Salmon populations in the Snake River sub-basin of the Columbia River in Oregon and Idaho, USA (Fig. 1) have been in decline since the late 1800s (National Marine Fisheries Service 1995), with particularly dramatic declines occurring since the mid-1970s (Schaller et al. 1999). Snake River spring, summer, and fall chinook (Oncorhynchus tshawytscha), and steelhead (Oncorhynchus mykiss) are all listed as threatened under the Endangered Species Act (ESA; see Table 1 for definitions of acronyms), and Snake River sockeye (Oncorhynchus nerka) are listed as endangered. Accelerating declines have been attributed to various factors: habitat degradation in freshwater and estuary rearing areas; tributary and ocean harvest (particularly for fall chinook and steelhead); interaction with hatchery steelhead and chinook smolts; changes in ocean conditions; completion in 1976 of the eight dams and reservoirs that comprise the Federal Columbia River Power System (FCRPS); and initiation of large-scale transportation of smolts in trucks and barges from the upper dams past the FCRPS to below Bonneville Dam (the last dam encountered by smolts as they migrate out of the Columbia River).

Simultaneous changes in the system confound attempts to determine the importance of the four H’s (habitat, harvest, hatchery, and hydro) and ocean conditions in causing historic stock declines (Fig. 2), and, consequently, the best course of action for recovering stocks. Different beliefs about which factors contributed most to observed declines resulted in the development of three different modeling systems for evaluating recovery strategies for Columbia River salmon stocks. These systems were developed by State and Tribal fishery managers, federal hydropower operating agencies (Bonneville Power Administration (BPA), the U.S. Army Corps of Engineers, and scientists at the Northwest Power Planning Council (NPPC), a political body that oversees research programs in the Columbia River. The models used the same basic kinds of information, but had different underlying assumptions that generally reflected the policy favored by the sponsoring agency. As a result, they provided different, often-conflicting management advice about the relative performance of alternative recovery strategies for Snake River populations.

In 1995, the National Marine Fisheries Service (NMFS), after three years of comparing these models and subjecting them to external peer review, issued a Biological Opinion on the FCRPS (a document that summarizes the status of listed stocks and prescribes hydropower system operations to avoid extinction), in which they concluded that the emphasis should shift to identifying and evaluating the models’ assumptions (NMFS 1995:124, Rec. 17). This recommendation was augmented by a 1994 court ruling (IDFG vs. NMFS, D.Or 1994) that determined that NMFS must consult with State and Tribal biologists. The new, collaborative process that was formed in response to the NMFS and court recommendations became known as the Plan for Analyzing and Testing Hypotheses (PATH). At its formation, PATH was intended to help reduce uncertainties in NMFS's future hydrosystem decisions.

PATH operated from September 1995 to May 2000 at a total cost of U.S.$7 million, funded from BPA power revenues through the NPPC’s Fish and Wildlife Program. We had the challenging tasks of facilitating and coordinating the PATH process. Our purpose in this paper is to summarize the overall PATH process, the complex institutional context in which it operated, and lessons learned that might apply to other collaborative research processes. Although we have considered the perceptions of others working within PATH or using its results, the conclusions presented here are our own.

Columbia River agencies faced many institutional and technical challenges to selecting the best long-term recovery strategy (see column 1 of Table 2). Many of these challenges are common in resource management problems. Less common are analytical processes with the necessary attributes to meet these challenges and help managers make difficult decisions (see column 2 of Table 2). Although no process is likely to meet all of these requirements, the degree to which these requirements were met in PATH forms a useful template for assessing its strengths and weaknesses.

PATH adopted six tools and approaches to meet these scientific and policy challenges (see column 3 of Table 2).

The structure of PATH resulted from discussions with senior policy makers and scientists in 1995. PATH’s immediate “client” was the Regional Forum for the Implementation of NMFS’s 1995 Biological Opinion, or “Implementation Team” (I. T.) (Fig. 3). The Implementation Team included Federal and State fish and wildlife agencies and FCRPS operating agencies. Although the I. T. represented a broad range of interests and objectives, its mandate was to focus entirely on hydrosystem operations and associated long-term decisions. About four times per year, the I. T. received detailed presentations on PATH’s findings and helped to prioritize its activities.

PATH consisted of about 30 scientists who contributed directly to workshops, analyses, and products (Appendix 1). Specialties included fisheries biology and management, analytical and modeling skills, field studies, experimental design, and dam operations. Although most scientists were employees or consultants from 12 regional institutions (Fig. 3), PATH also included an independent facilitation team and three independent scientists. The three independent scientists, who had expertise in the areas of conservation biology, quantitative methods, fisheries management, and decision analysis, assisted participants in developing and critiquing analytical methods and results. PATH activities were coordinated by a six-member Planning Group representing the facilitation team, State, Tribal, and Federal fishery agencies, the power system operating agencies, and the NPPC (Northwest Power Planning Council).

The role of the facilitation team was to stimulate and organize PATH analyses, provide clarity to decision makers, maintain the creativity and commitment of PATH scientists, and ensure that analyses met the highest scientific standards. Given the turbulent history, a major challenge was to build trust (or at least mutual respect) among scientists from competing agencies. We used five methods to do this: (1) establish ground rules for interactions (e.g., “be hard on the problem, easy on the people” Fisher et al. 1991); (2) redirect personal attacks into examinations of evidence for alternative hypotheses; (3) develop structured approaches that permit meaningful participation by all PATH scientists; (4) formally document evidence for and against alternative hypotheses; and (5) demonstrate qualities we wished others to display (i.e., respect, objectivity, integrity, creativity, and humor).

The PATH process was deliberately iterative, with four levels of peer review (Fig. 4). Small analytical work groups generally completed initial drafts of analyses, which were then reviewed and refined by larger work groups. Ultimately, all PATH participants reviewed draft reports, and the PATH Scientific Review Panel (SRP) and other regional scientists provided external review of final products. The SRP consisted of four independent scientists providing arm’s length peer review. They spent 150 days over four years reviewing about two thousand pages of PATH reports, and provided valuable direction on methods and priorities for future work.

PATH products were available to the public through open presentations to the I. T. (Implementation Team) and NPPC, and a dozen PATH scientists gave a special joint presentation to the public in February 1999. PATH's presentations were frequently reported by regional print and electronic media, and all reports were made available on the BPA web site http://www.bpa.gov/Environment/PATH. This provided an opportunity for review by other regional scientists.

Retrospective analyses followed from the requirement to understand the fundamental differences between models, and set the foundation for prospective analyses (Fig. 5).

The retrospective analyses:
      1) identified the assumptions underlying different modeling systems, and their management implications;
      2) expressed these assumptions in terms of unambiguous hypotheses about historical stock trends and causal factors;
      3) assessed the level of support for alternative hypotheses using existing data; and
      4) identified information gaps that constrained the ability to distinguish among alternative hypotheses.

Hypotheses for PATH retrospective analyses of Columbia River spring/summer chinook stocks were developed using a three-level framework to address observed patterns in specific life history stages and stressors that act on those stages (Fig. 6; Appendix 2). Assessments for Snake River steelhead and fall chinook were less comprehensive than for spring/summer chinook because of data limitations.

PATH prospective analyses evaluated the biological consequences of three primary hydrosystem recovery actions for ESA (U.S. Endangered Species Act)-listed Snake River salmonids: continue current operations (option A1); maximize transportation of salmon smolts down the migration corridor (A2); or natural river drawdown of four Snake River dams (A3). Information gained from the retrospective analyses was incorporated into the prospective analyses using a decision analysis approach (Fig. 5), which had been recommended by both the SRP and PATH independent scientists. The prospective/decision analyses of spring/summer and fall chinook employed models to simulate outcomes for each hydrosystem action under a range of hypotheses about various uncertainties, expressed as a decision tree (Fig. 7). This allowed us to: (1) look systematically at the outcomes of management actions across a range of hypotheses; (2) conduct a detailed sensitivity analysis of results to determine “key uncertainties” (Appendix 3); and (3) determine which actions performed well over a broad range of uncertainties (i.e., were most robust). The simulations also considered the range of uncertainty in stock productivity parameters and climate conditions, as estimated from retrospective analyses (Deriso, in press). We calculated weighted average outcomes for each action and also examined the distribution of outcomes over all combinations of hypotheses. Initially, we weighted all hypotheses equally, but later developed a “Weight of Evidence process” for spring/summer chinook to elicit probabilities on hypotheses from the PATH SRP (Fig. 8, Appendix 4).

Experimental management reduces key uncertainties by deliberately implementing spatial and temporal contrasts in treatments and associated monitoring. Although the SRP had recommended in 1996 that PATH design and evaluate experimental management approaches, this exploration only began in December 1998. The goal was to find management approaches that met conservation and recovery objectives, while also generating information to help select long-term management actions. PATH generated a set of seven experimental management and three research activities to distinguish among critical alternative hypotheses that implied different long-term decisions (Table 3). For each candidate activity, we described their spatial and temporal components, associated monitoring, potential learning benefits, risks to stocks, and practical constraints. In October 1999, this list was pared down by the I. T., who selected five experimental management actions plus a base case of current operations for further study of the quantitative trade-off between learning and conservation objectives (Appendix 5). This evaluation was completed in April 2000.

In this section, we assess the extent to which PATH was able to meet each of the challenges facing decision makers on the Columbia River, and some of the lessons learned from PATH’s successes and failures. We focus on some of the key lessons here and provide a longer list of lessons learned in Table 4.

PATH generally received clear direction from the Implementation Team on management objectives (largely dictated by the Endangered Species Act) and what hydrosystem actions to analyze. The I. T. focused exclusively on hydrosystem actions, consistent with its mandate from NMFS’s 1995 Biological Opinion on the FCRPS, and because other agencies (e.g., U.S. Forest Service, U.S. Fish and Wildlife Service) were reluctant to allow the I. T. to venture into their jurisdictional territories by specifying non-hydrosystem (habitat, harvest, and hatchery) actions. PATH considered only hydrosystem decisions as actions, but we did include habitat, harvest, and hatchery effects as uncertain states of nature in the decision analysis or as sensitivity analyses. Ultimately, though, PATH was criticized because the actions that we evaluated did not adequately capture the complete range of management alternatives and objectives, and thus did not fully address the challenge of differing management objectives among agencies.

PATH scientists recognized the potential pitfalls of focusing only on hydrosystem actions, but were not involved in developing the alternatives and thus had no opportunity to affect the choice of actions. This suggests that analytical processes should allow scientists more input to the development of alternative actions and should broaden the mandate of policy groups like the I. T.

2. Lack of trust among scientists and agencies over past disputes and court cases with dueling models

Over PATH’s lifetime, interactions among PATH scientists became much more respectful and constructive. Communication of disagreements progressed from personal attacks to identification of alternative hypotheses and examination of evidence. This was due to several reasons. First, the use of decision analysis did not require consensus, but instead considered all tenable hypotheses and assessed their implications for management decisions. Sensitivity analyses showed that decisions were insensitive to several alternative hypotheses that had been proposed, saving many hours of purposeless debate. Second, it was critical to have independent facilitators, who emerged as the only ones trusted to prepare, revise, and present final PATH products. In an analytical process such as PATH, the facilitation team must be technically proficient so that they can understand the analyses and its assumptions, mediate technical arguments between parties, and present findings to nontechnical audiences. Third, the rigorous internal review process within PATH gave everyone an opportunity to see each other’s analyses before they became public. This eliminated the possibility of surprise attacks characteristic of previous court cases. Fourth, the three independent scientists who actively participated in PATH analyses played a critical role in keeping debates within PATH scientifically grounded and ensuring that arguments over analyses and results were based on facts, not agency positions. Finally, the PATH SRP played a key role as an external arbiter of alternative hypotheses put forward by different scientists. Their involvement throughout PATH’s entire duration gave them an intimate understanding of the issues, models, and analyses. The long-term engagement of the PATH SRP differs from most Blue-Ribbon Panels of Experts, which generally do not have the time or resources to gain as in-depth an understanding of the models and data that they review.

PATH was partially successful in involving a representative range of scientists. Oversight by the multi-agency Implementation Team ensured involvement of a wide range of agencies and interests, and prevented any single agency from overly influencing PATH. Independent funding for PATH participants through the NPPC Fish and Wildlife Program also helped to ensure a level playing field. However, PATH was ultimately unsuccessful in actively involving key senior-level scientists from the NMFS Northwest Fisheries Science Center in Seattle, Washington, USA. In the Pacific Northwest, the Science Center has a very strong influence on NMFS decisions on endangered species. Although three scientists from the Science Center occasionally participated in PATH, they each had numerous other responsibilities, and as a result had less comprehension and ownership of PATH’s methods and conclusions. NMFS was represented in PATH by three very competent scientists from their Portland, Oregon office, but ultimately these scientists had less influence within their agency than did Science Center staff. Had NMFS required senior scientists from their Science Center to participate more directly in PATH starting in 1996, the process and methods probably could have evolved to incorporate their concerns, while still retaining an integrative framework. Instead, the NMFS Science Center eventually developed their own tools and analyses through their Cumulative Risk Initiative (CRI), which ultimately had more influence than PATH on NMFS’s decisions on the hydrosystem. This experience emphasizes the need to involve influential key scientists with strong links to ultimate decision makers.

3. Lack of understanding of differences and similarities in the models’ underlying assumptions

PATH retrospective analyses successfully elucidated differences between models, brought substantial empirical information to bear on alternative hypotheses to explain recent declines, produced considerable convergence on historical data sets (some of which were previously thought to be unusable, such as the spawner-recruit data), and made a significant contribution to the regional data inventory. The retrospective analyses also identified major uncertainties in past and current conditions that were unresolvable because of incomplete data and historical confounding (Fig. 2); these uncertainties were carried forward into the prospective decision analysis.

The decision analysis framework permitted rigorous sensitivity analyses of how each uncertainty affected the ranking and performance of actions. Importantly, we found that the different models’ estimate of the survival rate of in-river migrants through the hydropower system, a hotly debated value, was NOT an important determinant of overall life cycle survival. Rather, the key uncertainties that emerged from these sensitivity analyses were related to the cause of mortality in the estuary and ocean (Appendix 3). Key uncertainties were assessed through the Weight of Evidence process (Appendix 4), in which the PATH SRP judged arguments for and against alternative hypotheses and assigned probabilistic weights.

PATH had only limited success in identifying research, monitoring, and adaptive management actions to resolve uncertainties. Although the SRP highlighted the importance of experimental management work in 1996, the I. T. did not authorize PATH to focus seriously on this issue until 1999. We believe that the I. T. placed low priority on experimental management for a number of reasons. First, 1999 deadlines for important regulatory decisions pressured agencies to make firm long-term decisions (with supposed omniscience), rather than acknowledging uncertainty and encouraging deliberate management experiments. Second, discussions of experimental management actions made policy makers uneasy because of their inherent uncertainty. They correctly perceived the risk that management experiments may not deliver desired survival improvements, but were not swayed by arguments that implementing long-term decisions now would also have uncertain results and generate less information than a well-designed experiment.

In the end, PATH’s experience with experimental management was similar to that of others who have run into barriers to the implementation of adaptive management (Gunderson et al. 1995, Walters 1997, and MacDonald et al. 1999), and continues a tradition of ultra-passive approaches to adaptive management within the Columbia Basin (McConnaha and Paquet 1996). Perhaps testimonials from other jurisdictions where experimental management approaches were successfully used would have helped. However, the main lesson here is that experimental management is much easier to sell to policy makers, and to implement, before populations are placed on the endangered species list. Afterward, statutory restrictions on management actions, political pressure for immediate action, public attention on policy decisions, and the precarious status of the stocks make it almost impossible to implement experimental management.

PATH had mixed success in providing clear management advice to decision makers. We were successful in developing a single integrative data and modeling framework by melding three different life cycle models into a single Bayesian Simulation Model (Deriso in press; Appendix 6). By developing common data sets, an integrated modeling framework, and detailed sensitivity analyses, PATH greatly clarified the overall effects of different assumptions and provided a common understanding of the factors that determined the performance of recovery actions. This was a major advance over pre-PATH approaches, which had to reconcile results of different modeling systems using different data and assumptions.

PATH was less successful in clearly communicating technical analyses to nontechnical audiences. Part of this was because of the relatively complex structure of the integrative modeling framework, which had to be flexible enough to accommodate multiple hypotheses. The integrative life cycle model and the decision analysis, which incorporated multiple hypotheses, were critical approaches for building trust among PATH participants and understanding the factors affecting model results and performance of actions. However, these methods were hard to explain to nonscientists having little experience with these types of risk assessment methods. PATH’s performance measures (the 1995 Biological Opinion Jeopardy Standards) were also complex. It often proved to be difficult to communicate to nontechnical audiences “the probability of the geometric mean number of spawners exceeding a recovery threshold over 4000 simulations.” PATH reports were carefully worded to ensure that the views of all participants were fairly represented. As a result, PATH reports became comprehensive and technically demanding documents that were ill suited for nonspecialists.

A lesson here is that scientists need to think creatively about how to communicate summaries of risk assessments to nontechnical audiences, and to allocate sufficient time and personnel (e.g., technical editors, graphic designers) for producing such summaries. Thoughtfully designed interactive models can also provide nontechnical audiences with insights into the nature of variability and uncertainty (Walters 1994). Another lesson relates to the trade-off between the scientific relevance of model performance measures and their complexity. The Jeopardy Standards were meaningful to PATH scientists, but were difficult to explain to the public. It may have been wise to develop a simpler set of performance measures that, although perhaps less comprehensive, were more readily understood by nonscientists.

PATH was usually several months late in delivering results. This was largely because of the time required for participating scientists and agencies to agree on basic data sets, integrate models into a decision analysis framework, complete four levels of peer review, and agree on precise wording of Executive Summaries and presentations. As technical facilitators and synthesizers, we had no formal authority to pressure participants to deliver data and analyses on time, but instead relied on moral suasion to enforce deadlines. A more efficient approach might be to have scientists seconded from their agencies to serve on a task force under some independent institution, and/or give the leadership of such processes strong authority to enforce deadlines (scientific committees of the Intergovernmental Panel on Climate Change do the latter).

In December 2000, NMFS released their final Biological Opinion on the FCRPS (NMFS 2000). This plan largely maintains status quo operation of the hydropower and transportation systems, and relies on mitigation through habitat, harvest, and hatchery management to generate necessary survival improvements. In their decision, NMFS relied almost exclusively on analyses conducted by its Northwest Fisheries Science Center. Although some elements of the PATH analyses contributed to the structure of NMFS’s internal analyses (e.g., spawner-recruit data set, emphasis on uncertainties in the estuary and ocean life stage), NMFS did not accept PATH’s primary conclusions.

Why did PATH ultimately have less influence on NMFS’s decision than was originally envisioned? Given the issues at stake, NMFS was probably reluctant to commit to a collaborative scientific process whose methods, outcome, and schedule they could not control. Both NMFS and the NPPC faced political pressure from various groups to avoid or defer a drawdown decision, which PATH’s results generally supported. Other factors include PATH’s failure to actively involve senior, influential scientists within the NMFS Northwest Science Center, and an insufficient effort to communicate its results to senior policy makers within NMFS and the NPPC. Without a high level of involvement in, or ownership of, PATH analyses, these influential agencies perceived that they could have more control over the analyses and the decision by conducting their own internal analyses. These weaknesses ultimately undermined PATH’s funding support.

The PATH experience thus demonstrates some of the difficulties in implementing collaborative analytical approaches, but it also demonstrates the value of such an approach. Prior to PATH in 1993-1994, NMFS was faced with half a dozen lawsuits. While PATH was operating from 1995 to 2000, there were no lawsuits initiated against NMFS by the participating agencies. Since the termination of PATH in May 2000, many state, tribal, and environmental groups have expressed their dissatisfaction with the lack of collaboration by NMFS in the development of their December 2000 Biological Opinion. It therefore appears unlikely that the parties will remain outside of the courtroom for long in the absence of an established collaborative process. We believe that the courts are not the best place for scientists to do their work of testing alternative hypotheses and assessing alternative actions in the context of multiple uncertainties.

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.

The authors thank PATH participants, the SRP (Scientific Review Panel), and members of the I.T. for their commitment, creativity, and hard work over five years. Randall Peterman and two anonymous reviewers provided helpful comments on an earlier draft. This research was funded by the Bonneville Power Administration, as part of the NPPC (Northwest Power Planning Council) Fish and Wildlife Program.

Level-3 hypotheses sought to explain the specific mechanisms associated with observed trends in each life history stage identified at Level 2. These hypotheses link directly to key management decisions, which are affected by the quantitative magnitude of various effects (e.g., changes in survival with increased flow) rather than whether or not an effect merely exists. PATH completed a wide range of Level-3 analyses for spring/summer chinook and fall chinook, and a limited set of analyses for steelhead and sockeye. These analyses focused on the effects of specific hydrosystem actions on survival through the juvenile migratory corridor. For example, a PATH subgroup developed a detailed flowchart of the expected response of juvenile spring/summer chinook salmon to various operations and configurations of dams in the Snake and Columbia River (Toole et al. 1996). An important data set for Level-3 analyses was the mark–recapture experiments with PIT (Passive Induced Transponder) —tagged juvenile salmon conducted by NMFS (e.g., Muir et al. 1996). These data were useful for estimating reach- and project-specific survival rates of migrating salmon smolts, and for estimating smolt-adult survival rates of transported and nontransported fish. PATH conclusions from Level-3 hypotheses were that modifications to the existing hydropower system were not likely to improve juvenile survival rates. Transportation of smolts improves the direct survival of smolts, but there was insufficient information about delayed effects of transportation to say whether transporting smolts improves overall spawner-to-recruit survival rates. In addition, drawdown of Snake River dams can compensate for effects of the hydrosystem and improve juvenile survival rates.

The results of these retrospective analyses were ultimately condensed into a clearly written, 30-page Conclusions Document (Marmorek et al. 1996b). The tough internal and external review process led to 10 drafts of this document, but ensured that the strength of the conclusions was consistent with the available evidence.

FALL CHINOOK

Retrospective analyses for fall chinook were less comprehensive than for spring/summer chinook because of time constraints and limitations in available data (Peters et al. 1999, in press). For example, for fall chinook there is a shorter time series of spawner-recruit and juvenile passage survival data. Retrospective analyses for fall chinook generally focused on the consistency of different structures of stock-recruit models with spawner-recruit data developed for four fall chinook stocks from the Snake, Lewis, and Deschutes Rivers, and from the Hanford reach of the Columbia River. These models embodied different hypotheses about the importance of transportation, hatchery supplementation, and climate effects on historical trends in survival rates (these analyses were roughly analogous to Level-2 analyses in the spring/summer hypothesis framework). We also explored overall trends in spawner-recruit survival (analogous to Level-1 analyses), and reviewed evidence on the survival of smolts through various components of the hydrosystem such as survival through turbines and in spillways (analogous to Level-3 analyses).

APPENDIX 3
Key Uncertainties in the PATH Decision Analyses

KEY UNCERTAINTIES

PATH conducted extensive sensitivity analyses to determine which uncertainties were most influential in determining model outcomes. This involved comparing outcomes for all of the runs (i.e., all combinations of hypotheses) with the outcomes under a subset of runs associated with a particular hypothesis, or a particular combination of hypotheses. These comparisons looked at both the ability to meet survival and recovery standards and the ranking of decisions (i.e., which action had the higher probability of meeting a given standard). We considered a range of differences in these probabilities (i.e., 0.02, 0.06, and 0.1) to assess how robust our conclusions were. To independently check the inferences we drew from these methods, we applied Categorical Regression Tree analysis to the complete data set of decision analysis outcomes. The ‘CART’ trees clearly showed the relative importance of each action and hypothesis to the computed probabilities of survival and recovery. Sensitivity analyses for spring/summer chinook are described in Marmorek et al. (1998b, c, d) and Peters and Marmorek (in press). Sensitivity analyses for fall chinook are described in Peters et al. (1999, in press).

Using this approach, PATH scientists identified two key uncertainties that have the strongest effects on survival and recovery of Snake River spring/summer and fall chinook: extra mortality of nontransported fish and the relative post-Bonneville survival of transported fish compared to post-Bonneville survival of nontransported fish. Bonneville Dam is the last of eight dams that smolts pass on their way to the ocean (“BON” in Fig. 1).

1. Extra mortality of nontransported fish

Extra mortality is defined as any mortality occurring outside the juvenile migration corridor that is not accounted for by the other terms in the life cycle model used for retrospective and prospective modeling (i.e., terms for stock productivity and carrying capacity, mortality in dams and reservoirs, and estuarine/ocean mortality affecting all salmonid populations). Because many of the changes that may account for historical patterns in extra mortality all happened around the same time (e.g., Fig. 2) there is uncertainty about which of these factors (or mix of factors) influences extra mortality. Therefore, PATH formulated three alternative hypotheses about the source of this extra mortality:

a. Hydro – extra mortality is related to the experience of smolts that pass through the hydropower system (e.g., delayed effects of stress).

b. Regime shift – extra mortality follows a 60-yr cycle that is related to long-term cycles in ocean conditions. There are no actions that can be taken to reduce extra mortality, but extra mortality will eventually go down when ocean conditions improve.

c. Stock viability (here to stay) – extra mortality is due to some phenomenon that will not be affected by any hydrosystem action or regime shift (i.e., interaction with hatchery fish, presence of diseases such as Bacterial Kidney Disease, or reduction in nutrients associated with historical declines in spawning stock).

Extra mortality can only be inferred from other measured quantities; it cannot be directly measured. This makes it difficult to monitor changes in extra mortality resulting from an experimental action, and thus to test alternative hypotheses. Nevertheless, extra mortality is still an important construct because (a) it helps to design experimental management actions that address its potential causes; and (b) it is needed to simulate the range of effects of alternative experimental actions to assess their relative risks and benefits.

2. The relative post-Bonneville survival of transported fish compared to post-Bonneville survival of nontransported fish

In the PATH modeling framework, the ratio of these two values is known as “D”. Like extra mortality, D cannot be directly measured, but must be inferred from other measured quantities (e.g., Transport:Control ratios and in-river survival estimates from transportation studies for spring/summer chinook). Differences in the assumptions used to estimate D led to alternative hypotheses about both historical and future D values, for both spring/summer and fall chinook.

IMPLICATIONS FOR SELECTING A LONG-TERM MANAGEMENT ACTION

In general, the ability of transportation to recover stocks depends directly on D (i.e., more likely to recover stocks when D is high, less likely when D is low). Drawdown actions were forecast to recover stocks over a wider range of D values (Marmorek et al. 1998b, Peters et al. 1999, in press, Peters and Marmorek in press). The ability of both drawdown and transportation to recover stocks also depends on the extra mortality hypothesis – both actions were more likely to recover stocks with the hydro hypothesis than with the regime shift or stock viability hypotheses.

Reducing these key uncertainties can help to identify the long-term management action that is best able to recover the stocks. There is an interaction between extra mortality hypotheses and the D value: forecasts of recovery are generally more sensitive to the extra mortality hypotheses if D is a high value. If D is high, fewer transported fish die below Bonneville Dam. Other factors causing extra mortality of all fish are then required to explain historical declines in overall survival. If D is low, post-Bonneville mortality of transported fish is sufficient to explain most of the observed historical declines in overall survival rates, and extra mortality factors affecting all fish become less important. This suggests that we should not measure D without also narrowing down the extra mortality hypotheses, and vice versa.

APPENDIX 4
Description of Weight of Evidence Process

This appendix provides more detail on the Weight of Evidence Process developed for spring/summer chinook (Fig. 8). We did not have time to initiate an analogous approach for fall chinook, steelhead, or sockeye. We first narrowed the problem down by sensitivity analysis to find the key uncertainties that affected the choice of management action (see Appendix 3). These analysis indicated that only seven of 14 uncertainties had a significant effect on outcomes, and three of these seven were particularly critical. The next steps involved an iterative series of written submissions, workshops, and syntheses to examine the evidence for and against alternative hypotheses for the seven key uncertainties. There were 25 submissions from PATH scientists (about 350 pages), which we synthesized into a 150-page document (Marmorek et al. 1998c). We used four criteria to evaluate alternative hypotheses:

    1) the clarity of the hypothesis (i.e., clear specification of stressors affecting survival, without confounding);
    2) existence of a reasonable mechanism or set of mechanisms by which the hypothesis operates;
    3) consistency with empirical evidence (i.e., Do stock survival indices and hypothesized stressors vary across space and time in a manner consistent with the hypothesis? How well do different hypotheses fit empirical data such as reach survivals and recruits per spawner?); and
    4) validity of the method of projecting the hypothesis into the future (i.e., are mathematical methods consistent with hypotheses and mechanisms that they were meant to represent? Are projections under current operations reasonable given recent measurements not used in model calibration?).

Arguments and counter-arguments under each of these criteria were laid out systematically, with reference to supporting evidence in the submissions and other literature. All of the documentation (i.e., submissions and synthesis document) was provided to the Scientific Review Panel (SRP) for review. Three weeks later, the four SRP members attended a workshop in Vancouver at which experts in elicitation (people not involved in the PATH process) led them through the following steps (Peters et al. 1998):

    1) training regarding process and judgmental biases;
    2) clear definition of the judgments to be obtained, and explicit exclusion of any recommendations of specific management actions;
    3) independent elicitation of the relative probability of alternative hypotheses, and the rationale for each SRP member’s conclusions;
    4) aggregation and discussion of differences among experts; and
    5) documentation.

Step 4 led the SRP members to recommend that PATH explore strong management experiments as an alternative approach to determining the probability of alternative hypotheses.

Subsequent to the workshop, the facilitation team applied the individual judgements of SRP members to the hypotheses in the decision analysis, calculated weighted-average outcomes, and compared those weighted averages to the situation in which all hypotheses were weighted equally (Peters et al. 1998). We found that applying the weights did not change the relative ranking of actions (A3 generally performed best), and had only a small effect on weighted average results, primarily because all four of the SRP members assigned similar weights to the passage/transportation models (which had large effects on the outcomes). These results were presented to the Implementation Team and a large contingent of interested public and media at a meeting in October 1998.

APPENDIX 5
Challenges in Designing Management Experiments to Resolve Critical Uncertainties

Experimental management requires spatial/temporal contrasts in treatments. In our initial scoping of experimental management actions (Marmorek et al. 1999), we determined that for many actions affecting the mainstem migration corridor (e.g., changes in flow, methods or amount of smolt transportation, magnitude of hatchery releases, harvest rates), one can only create temporal contrasts, not true spatial contrasts (i.e., an independent control system). Partial spatial contrasts are possible by comparing the performance of Snake River stocks with stocks in the Lower or Mid-Columbia region (Fig. 1), using the approach developed by Deriso et al. (1996). However some of the Snake River treatments could also affect Lower Columbia stocks, confounding the ‘control’ stocks. For example, upstream changes in flow, transportation, or hatchery releases could all affect conditions for Lower Columbia River stocks during the critical period of entry into the estuary and ocean. Because the four Snake River projects are not used as storage reservoirs, drawdown actions are less likely to have confounding impacts on downriver stocks. PIT-tags offer an opportunity to create control groups within the same year for different transportation methods, but the total number of tags required depends exactly on what one defines as a control (e.g., smolts passing down river and never undetected at any project, or simply smolts that are never put in a barge or truck), as well as the level of marine survival of both treatment and control groups.

In our more detailed analyses of the five priority experimental management actions, we used relatively simple models to simulate “true” future survival changes associated with the candidate actions. We then assessed the ability to learn by seeing how well the experiment and future monitoring could estimate the “true” survival change in a fluctuating environment (Peters et al. 2000; C. M. Paulsen and R. A. Hinrichsen, unpublished data). The main metrics of how much can be learned from an action were expressed in terms of the probability of estimating effects of an action over various time frames, or, conversely, how long it would take to estimate an effect with a certain level of confidence. Various criteria can be applied to determine how long an experiment needs to be run to estimate effect sizes that reflect the risk preferences of decision makers. The models also estimated various conservation metrics for each action, which were compared to the ability to learn to assess the trade-off between learning and conservation objectives. To avoid the confounding of gradual changes in climate or ocean conditions, we found that the most effective temporal contrast is switching an action on and off in alternate years. Sub-basin actions such as carcass fertilization could utilize more efficient experimental designs with both spatial and temporal contrasts, and hence obtain a faster rate of learning. However, their survival benefits are expected to be only modest, and insufficient on their own to recover the stocks.

APPENDIX 6
Limitations of PATH's Models

PATH's escapement projections under the current operations scenario (option A1) were significantly higher that what was actually observed for recent years. This is primarily because the models were designed to make relative comparisons of actions over 100 years, and not short-term predictions. For example, year effects were sampled from the last 40 years of stock performance, not only from the poorer conditions that have generally prevailed since 1977. Although we have no reason to suspect differences in the relative performance of actions from this bias, the forecasted probabilities of survival and recovery are probably over-optimistic for all actions. Recent model projections have confirmed that most stocks would be likely to go extinct under current management if the poor post-1985 year effects were assumed to continue into the future.

As with any model, certain processes were deliberately excluded because of both a lack of understanding and a need to keep the scale of complexity reasonable. The PATH life cycle models did not consider interactions between populations (e.g., straying) that could affect productivity through changes in genetic diversity. The impacts of upstream storage projects were only considered in the models in terms of changes in smolt travel time to the estuary; other mechanisms of impact (e.g., changes in salinity) could also be important. The models also did not consider changes in the quantity and quality of mainstem river habitat other than those caused by the hydrosystem.

At a higher level, the PATH decision analysis only considered biological impacts. A logical next step would have been to combine biological, social, and economic performance measures into an integrated decision analysis that could examine all uncertainties and trade-offs. However, the social, economic, tribal, cultural, and recreation impacts of the actions under consideration were examined in a separate effort from PATH (the Lower Snake River Feasibility Study, under the direction of the U.S. Army Corps of Engineers), and it proved to be infeasible to complete an integrated decision analysis. The Water Use Planning Process developed to evaluate hydrosystem operations at 20 facilities in British Columbia offers a promising approach to integrating social, economic, and environmental objectives (McDaniels et al. 1999).

Anderson, J. J. 1996. Review of the Influence of Climate on Salmon. Chapter 12 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Beamesderfer, R. C. P., H. A. Schaller, M. P. Zimmerman, C. E. Petrosky, O. P. Langness, and L.LaVoy. 1997. Spawner-recruit data for spring and summer chinook salmon populations in Idaho, Oregon, and Washington. In D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Retrospective and prospective analyses of spring/summer chinook reviewed in fiscal year1997. Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/1997retro/toc.htm.

Botsford, L. W., and C. M. Paulsen. 2000. Assessing covariability among populations in the presence of intraseries correlation: Columbia River spring-summer chinook salmon (Oncorhynchus tshawytscha) stocks. Canadian Journal of Fisheries and Aquatic Sciences 57:616-627.

Budy, P., C. E. Petrosky, P. Wilson, O. P. Langness, and N. Bouwes. 1998. Hatchery “extra mortality” hypotheses: Response to Submissions 1 and 2. Submission 19 to D. R. Marmorek et al. 21 August 1998. Plan for Analyzing and Testing Hypotheses (PATH): Weight of Evidence Report. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/WOE_Report/.

Chamberlain, T. C. 1890. The method of multiple working hypotheses. Science 15:92.

Deriso, R. B. In press. Bayesian analysis of stock survival and recovery of spring and summer chinook of the Snake River basin. In J. M. Berkson, L. L. Kline, and D. J. Orth, editors. Incorporating uncertainty into fisheries models. American Fisheries Society, Bethesda, Maryland, USA.

Deriso, R. B, D. R. Marmorek, and I. J. Parnell. 1996. Retrospective analysis of passage mortality of spring chinook of the Columbia River. Chapter 5 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Deriso, R. B, D. R. Marmorek, and I. J. Parnell. In press. Retrospective patterns of differential mortality and common year effects experienced by spring chinook of the Columbia River. Canadian Journal of Fisheries and Aquatic Science.

Fisher R., W. Ury, and B. Patton. 1991. Getting to YES: Negotiating agreement without giving in. Penguin, New York, New York, USA.

Gunderson, L. H., C. S. Holling, and S. S. Light. 1995. Barriers and bridges to the renewal of ecosystems and institutions. Columbia University Press, New York, New York, USA.

Hinrichsen, R., J. J. Anderson, G. M. Matthews, and C. C. Ebbesmeyer. 1997. Effects of the ocean and river environments on the survival of Snake River stream-type chinook salmon. In D. R. Marmorek et al. 1998. Plan for Analyzing and Testing Hypotheses (PATH): Retrospective and prospective analyses of spring/summer chinook reviewed in fiscal year1997. Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/1997retro/toc.htm.

Langness, O. P., and H. A. Schaller. 1996. Contrasting stock-recruitment and harvest patterns of the Columbia River stream-type chinook populations. Chapter 13 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Coilumbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

MacDonald, G. B., J. Fraser, and P. Gray. 1999. Adaptive management forum: Linking management and science to achieve ecological sustainability. Available from Ontario Ministry of Natural Resources, Peterborough, Ontario, Canada. E-mail: nricmn@epo.gov.on.ca

Mantua, N. J., S. R. Hare, Y. Zhang, J. M. Wallace, and R. C. Francis. 1997. A Pacific interdecadal climate oscillation with impacts on salmon production. Bulletin of the American Meteorological Society 78:1069-1079.

Marmorek, D. R., et al. (25 co-authors). 1996b. PATH: Plan for Analyzing and Testing Hypotheses. Conclusions of fiscal year 96 Retrospective Analyses. 10 December 1996. Prepared by ESSA Technologies, Vancouver, British Columbia, Canada for Bonneville Power Administration, Portland, Oregon, USA. BPA web site: http://www.efw.bpa.gov/Environment/PATH/cncdft10.html.

Marmorek, D. R., et al. (19 co-authors). 1998 a. Plan for Analyzing and Testing Hypotheses (PATH): Retrospective and prospective analyses of spring/summer chinook reviewed in fiscal year1997. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/1997retro/toc.htm.

Marmorek, D. R., et al. (27 co-authors). 1998b. Plan for Analyzing and Testing Hypotheses (PATH): Preliminary decision analysis report on Snake River spring/summer chinook. Draft Report, March 1998. ESSA Technologies, Vancouver, British Columbia, Canada. [Appendices include descriptions of how various hypotheses were implemented in decision analysis.] BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/pdar/index.html.

Marmorek, D. R., et al. (28 co-authors). 1998c. Plan for Analyzing and Testing Hypotheses (PATH): Weight of Evidence Report. 21 August 1998. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. [116 pages and appendices, plus 25 submissions.] BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/WOE_Report/.

Marmorek, D. R., et al. (19 co-authors). 1999. PATH: Scoping of candidate research, monitoring and experimental management actions: concurrently reducing key uncertainties and recovering stocks. Working draft prepared by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/.

McConnaha, W. E., and P. J. Paquet. 1996. Adaptive strategies for the management of ecosystems: the Columbia River experience. American Fisheries Society Symposium 16:410-421.

McDaniels, T. L., R. S. Gregory, and D. Fields. 1999. Democratizing risk management: Successful public involvement in local water management decisions. Risk Analysis 19(3):497.

Muir, W. D., S. Smith, E. Hockersmith, S. Achord, R. Absolon, P. Ocker, B. Eppard, T. Ruehle, J. Williams, R. Iwamoto, and J. Skalski. 1996. Annual Report: Survival estimates for the passage of juvenile chinook salmon through Snake River dams and reservoirs, 1995. Report from National Marine Fisheries Service to Bonneville Power Administration. Project Number 93-29, Contract Number DE-AI79-93BP10891. Available from Bonneville Power Administration, Portland, Oregon, USA.

National Marine Fisheries Service (NMFS). 1995. Reinitiation of consultation on 1994-1998 operation of the Federal Columbia River Power System and Juvenile Transportation in 1995 and Future years. Northwest Region, National Marine Fisheries Service, Seattle, Washington, USA.

National Marine Fisheries Service (NMFS). 2000. Biological opinion. Reinitiation of consultation on operation of the federal Columbia River Power System, Including the Juvenile Fish Transportation System, and 19 Bureau of Reclamation Projects in the Columbia River. Northwest Region, 21 December 2000. National Marine Fisheries Service web site: http://www.nwr.noaa.gov/1hydrop/hydroweb/docs/Final/2000Biop.html.

Paulsen, C. M. 1996. Level 2 Hypotheses. Chapter 4 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Paulsen, C. M., and T. Fisher. 1997. Update on ocean distribution of coded wire tagged spring/summer chinook. In D. R. Marmorek et al. 1998. Plan for Analyzing and Testing Hypotheses (PATH): Retrospective and prospective analyses of spring/summer chinook reviewed in fiscal year1997. Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/1997retro/toc.htm.

Paulsen, C. M., and R. A. Hinrichsen. 1998. Testing the hypothesis that extra mortality of wild Snake River spring/summer chinook varies with releases of Snake River hatchery fish. Submission 2 to D. R. Marmorek and 28 co-authors. 21 August 1998. Plan for Analyzing and Testing Hypotheses (PATH): Weight of Evidence Report. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/WOE_Report/.

Paulsen, C. M., R. A. Hinrichsen, and T. Fisher. 1997. Effects of climate and land use on index stock recruitment. In D. R. Marmorek et al. 1998. Plan for Analyzing and Testing Hypotheses (PATH): Retrospective and prospective analyses of spring/summer chinook reviewed in fiscal year1997. Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/1997retro/toc.htm.

Peterman, R. M., and J. Anderson. 1999. Decision analysis: a method for taking uncertainties into account in risk-based decision making. Human and Ecological Risk Assessment 5(2):231-244.

Peters, C. N., and D. R. Marmorek. In press. Application of decision analysis to evaluate recovery actions for threatened Snake River spring and summer chinook salmon (Onchorhynchus tshawytscha). Canadian Journal of Fisheries and Aquatic Science.

Peters, C. N., D. R. Marmorek, and R. B. Deriso. In press. Application of decision analysis to evaluate recovery actions for threatened Snake River fall chinook salmon (Onchorhynchus tshawytscha). Canadian Journal of Fisheries and Aquatic Science.

Peters, C. N., I. J. Parnell, D. R. Marmorek, R. Gregory, T. Eppel, S. Carpenter, J. Collie, S. Saila, and C. Walters. 1998. Conclusions and recommendations from the PATH Weight of Evidence Workshop, 8-10 September 1998, Vancouver, British Columbia, Canada. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/981008.weight.pdf.

Peters, C. N., et al. (27 co-authors). 1999. PATH decision analysis report for Snake River fall chinook. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/991006.fall99d6.pdf.

Peters, C. N., et al. (14 co-authors). 2000. PATH preliminary evaluation of the learning opportunities and biological consequences of monitoring and experimental management actions. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/2000_M_and_E.pdf.

Petrosky, C. E. 1998. BKD extra mortality hypothesis. Submission 20 to D. R. Marmoreket al. 21 August 1998. Plan for Analyzing and Testing Hypotheses (PATH): Weight of Evidence Report. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/WOE_Report/.

Petrosky, C. E., and H. A. Schaller. 1996. Evaluation of productivity and survival rate trends in the freshwater spawning and rearing life stage for Snake River spring and summer chinook. Chapter 9 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Petrosky, C. E., H. A. Schaller, and P. Budy. 2001. Productivity and survival rate trends in the freshwater spawning and rearing stage of Snake River chinook salmon (Oncorhynchus tshawytscha). Canadian Journal of Fisheries and Aquatic Science 58:1196-1207.

Schaller, H. A., C. E. Petrosky, and O. P. Langness. 1999. Contrasting patterns of productivity and survival rates for stream-type chinook salmon (Oncorhynchus tshawytscha) populations of the Snake and Columbia rivers. Canadian Journal of Fisheries and Aquatic Sciences 56:1031-1045.

Toole, C., A. Giorgi, E. Weber, and W. E. McConnaha. 1996. Hydro decision pathway and review of existing information. Chapter 6 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Walters, C. 1994. Use of gaming procedures in evaluation of management experiments. Canadian Journal of Fisheries and Aquatic Sciences 51:2705-2714.

Wilson, P. 1996. Hatchery impacts. Chapter 11 in D. R. Marmorek et al. 1996. Plan for Analyzing and Testing Hypotheses (PATH): Final report on retrospective analyses for fiscal year 1996. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/FY96_Retro_Report/.

Williams, J., G. Matthews, J. Myers, S. G. Smith, T. Cooney, and C. Toole. 1998. Hatchery extra mortality hypothesis. Submission 1 to D. R. Marmorek and 28 co-authors. 21 August 1998. Plan for Analyzing and Testing Hypotheses (PATH): Weight of Evidence Report. Compiled and edited by ESSA Technologies, Vancouver, British Columbia, Canada. BPA web site: http://www.efw.bpa.gov/Environment/PATH/reports/ISRP1999CD/PATH%20Reports/WOE_Report/.