Uncertainty as Information: Narrowing the Science-policy Gap

G. A. Bradshaw¹ and Jeffrey G. Borchers²

• Abstract
• Introduction
• Sources of the Science-policy Gap
• The Role of Cognitive Dissonance and Volition
• The Role of Scientific Uncertainty
• Conclusions: Bridging the Science-policy Gap
• Responses to this Article
• Acknowledgments
• Literature Cited

Conflict and indecision are hallmarks of environmental policy formulation. Some argue that the requisite information and certainty fall short of scientific standards for decision making; others argue that science is not the issue and that indecisiveness reflects a lack of political willpower. One of the most difficult aspects of translating science into policy is scientific uncertainty. Whereas scientists are familiar with uncertainty and complexity, the public and policy makers often seek certainty and deterministic solutions. We assert that environmental policy is most effective if scientific uncertainty is incorporated into a rigorous decision-theoretic framework as knowledge, not ignorance. The policies that best utilize scientific findings are defined here as those that accommodate the full scope of scientifically based predictions.

KEY WORDS: adaptive management, decision making, environmental policy, global climate change, monitoring, risk, uncertainty.

Published: March 22, 2000

Amid the diversity of conflicting opinions and special interests that characterize most large-scale environmental issues, there is one phenomenon upon which nearly all agree: the rate at which humans are altering the biosphere has increased dramatically in the past century (Reischauer and Fairbank 1960, United Nations 1997, Vitousek et al. 1997). For scientists, policy makers, and the public at large, the inferences drawn from scientific findings differ greatly. Even unprecedented efforts such as the Intergovernmental Panel on Climate Change (IPCC 1990, 1996) appear to provide insufficient scientific guidance for the formulation of decisive environmental policy. Although the latest report from the IPCC was heralded as an unprecedented international scientific consensus, considerable scrutiny and debate concerning the validity and implications of its findings followed (Shackley and Wynne 1996, Raynor and Malone 1997). This now-familiar pattern wherein policy lags behind science has been characterized as either a cautious response to uncertain predictive capabilities or dangerous procrastination fueled by political and economic exigencies (New York Times 1997, The Oregonian 1998). Critics argue that scientists know too little about global change to warrant anticipatory policy formulation and assert that current information and levels of certainty fall short of scientific standards for decision making. Others maintain that science is not the issue, and that the indecisiveness of policy makers reflects a shortfall of political willpower (Gelbspan 1997).

We discuss the means by which some dysfunctional aspects of the science-policy interface, herein referred to as the "science-policy gap," can be ameliorated. Specifically, we suggest that inaccurate translations from science to policy derive in large part from an improper inference of scientific uncertainty (Funtowicz and Ravetz 1990). Generally speaking, whereas scientists may be familiar with the conditions of scientific uncertainty, the public and policy makers often seek certainty and deterministic solutions. In some cases, the social and cultural standards superimposed on those of science may become critical constraints to effective decision making (Gunderson et al. 1995). As shown in Table 1, the institutions of science and government are generally marked by very distinct behaviors and attributes. These differences contribute to some of the difficulties associated with transmitting and translating scientific information into policy and decisions. They also underscore the need for adaptive management principles and a rigorous decision-theoretic framework as a foundation for robust policy formulation (Walters 1986, 1997, Dovers et al. 1996, Lee 1999).

To better articulate the nature of the science-policy gap, it is useful to outline the life history of a scientific model from the perspective of Kuhn's (1962) paradigm shifts. As shown in Fig. 1, the level of confidence in the model by the scientific community increases with the level of scientific confirmation (i.e., scientific activities that cumulatively corroborate the theory's hypotheses). This relationship is portrayed as linear for the scientific community where the confidence level tracks the rate of confirmation. In contrast, the degree and rate at which social confidence and consensus develops for a given scientific finding may lag behind that of the science community due to a complex of social factors. In reality, the shape of this function will vary with individual scientific findings.

Fig. 1. Schematically, the science-policy gap is defined as the difference in levels of confidence for a given scientific finding expressed by the scientific community and society. Generally speaking, as confirmation of a model or scientific finding increases, the level of confidence in it also increases.

As evidence accumulates to support the underlying hypotheses of the model, confidence in its representations increases (e.g., weather prediction models). In time, the model achieves greater standing as inferences concerning its representations are disseminated and debated in scientific literature and other fora. Publication, citations, and merit awards, such as competitive grants, mark its acceptance. At some threshold of accord within the scientific community, consensus emerges.

However, the emergence of a so-called scientific consensus does not necessarily guarantee the level of certainty demanded by most policy makers (Lemons 1996). Even the constants of physics and chemistry are recognized as potentially inaccurate or imprecise and subject to continual revision (Peterman and Peters 1997). In the case of large-scale simulation models, constants and parameters contain assumptions and uncertainties that propagate in uncertain ways to produce uncertain output. For scientists, this is business as usual (Morgan and Henrion 1990, Raynor and Malone 1998). For society and its decision makers, however, such uncertainty may cast a shadow upon science itself (Shackley and Wynne 1996).

In contrast to the relatively formal process characterizing the scientific community, the acceptance of scientific results by a diverse public sector may differ markedly. We define the science-policy gap as the difference in levels of confidence for a given scientific finding expressed by the scientific community and by society (Fig. 1). In actuality, the broad categories of "public" and "scientific" comprise a vast array of individuals and groups having distinct histories, cultures, and belief systems that influence perceptions of nonhuman and human nature (Nader 1996). For example, because of their position within government, agency scientists may hold very different attitudes toward scientific uncertainty relative to their academic counterparts. An agency scientist has fealty not only to the scientific community but also to a sometimes highly politicized leadership that may be directly involved in defending policy. Paradoxically, the reluctance of scientists in such agencies to reveal ambiguities and uncertainties to the public out of fear of diminishing their credibility serves only to engender greater mistrust in the public (Walters 1997).

The science-policy lag is evidenced by the length of time required for a given scientific finding to assimilate into society. In part, the lag can be attributed to the rate of information dissemination. During this cognition phase, scientific information (e.g., effects of greenhouse gases) is disseminated by various media (e.g., the Internet, science magazines, television). Realistically, the science-policy gap is more than an information gap; the extent to which society's level of confidence in a theory or model lags behind that of the scientific community depends on other significant factors.

Individuals and groups exhibit varied responses when faced with new information. If such information is consistent with extant behaviors and beliefs, it can be readily accepted and integrated. However, if the new information conflicts with behavior and belief, the resulting state is described as "cognitive dissonance" (Festinger 1957, Adams 1973). According to the theory, the inconsistency and psychological discomfort of cognitive dissonance can be reduced by changing one's beliefs, values, or behavior. Dissonance can be avoided by rejecting or avoiding information that challenges belief systems or by interpreting dissonant information in a biased way.

The role of cognitive dissonance can be observed in numerous contexts. One highly publicized case concerning public land use dramatically exemplifies the collision of differing world views. As early as 1976, a landmark report was published forecasting future shortfalls of mature, harvestable timber independent of any consideration for the Northern Spotted Owl, Stryx occidentalis (Beuter et al. 1976, Yaffee 1994). In ensuing years, this shortfall, combined with improved technologies for harvesting and processing and a vigorous raw materials export market, resulted in significant job declines. Yet, despite this information, the issue continued to be misrepresented as an "owls-vs.-jobs" issue, one that failed to acknowledge trends within the timber industry (Yaffee 1994). This type of oversimplification of complex issues and denial of "dissonant" information continues to embroil science in acrimonious public debates (USDA and USDI 1994, USDA 1996).

Dissonance between existing beliefs and new information may be shaped by a host of factors, all of which inhibit the rate at which scientific findings are assimilated into policy. In what we have called the "volition" phase of the science-policy gap, public debate around an emerging scientific consensus may derive from a combination of cultural, psychological, and economic interests threatened by the policy inferences of dissonant scientific findings. One obvious example is the tobacco industry, which is undergoing an onslaught of litigation decades after research confirmed the health risks of smoking tobacco. As seen in Fig. 2, the volition phase of the science-policy gap may be described in many cases as social inertia born not of a paucity of information but of a complex, deep-seated resistance to change derived from numerous social, religious, and cultural sources (Lee 1993, Jasanoff and Wynne 1998).

Fig. 2. The science-policy gap consists of related sets of constraints and sources of uncertainty.

By definition, science is a provider of new information and has always been cast in the dual role of both defending and attacking reigning paradigms (Yearley 1996, Schick 1997). For this reason, science will frequently produce cognitive dissonance, uncomfortable levels of uncertainty, and resistance in the body politic. Acceptance of its findings will be contingent upon attitudes and perceptions toward uncertainty and risk (Dorner 1996). In the case of global climate change, the challenge is to delineate appropriate responses to highly uncertain predictions of ecological and social crises in the absence of reliable estimates of risk (IPCC 1996).

Scientific uncertainty is typically characterized by statistical analysis (e.g., statistical confidence intervals, model output). Decision making in the sciences, such as that accomplished by hypothesis testing based on frequentist statistics, is usually performed according to consistent, though arbitrary standards (e.g., Type I error probability levels of 0.05). In less controlled situations, scientific uncertainty must be ascertained by other means, such as model prediction errors. Although a familiar companion to most scientists, there is little tolerance in the policy arena, as in most organized human activity, for the uncertainty and "ignorance" typically associated with complex systems (Briskin 1998). In contrast to the society that uses science to reduce uncertainty, "[d]oubt is clearly a value in the sciences" (Feynman 1998). Hence, the culture of science ends up in competition with the demanding exigencies of economics and politics, except when its findings are possessed of sufficiently high levels of certainty (Sims and Baumann 1974).

Nowhere is this truer than in the case of global climate change. The large-scale simulations presented in the IPCC reports portray a set of highly uncertain outcomes for various boundary conditions (e.g., global patterns of temperature extremes under fixed scenarios for CO₂ emission controls, which are themselves based on uncertain estimates of model parameters (IPCC 1996). The IPCC reports represent a wealth of both accumulated knowledge and uncertainty. Unlike more tractable, data-rich scientific problems that readily yield understanding from statistical analyses, science in the IPCC report appears to confound policy makers who prefer more "certain," contained estimates of risks. The presence of uncertainty associated with climate change science has been interpreted as an undermining of scientific authority and a hindrance to policy (Martin and Richards 1995, Shackley and Wynne 1996).

The uncertainty (or the lack of confidence in scientific findings) perceived by the public and policy makers can be grouped into two categories. First, there is uncertainty about the uncertainty. The public is puzzled by debate within the scientific community when it surfaces in the media (Risbey et al. 1991, Schlesinger and Jiang 1991, Martin and Richards 1995). For example, in a recent Congressional hearing on global change, when asked about an immediate "act now" versus a "wait and see" policy, one scientist stated that "many would argue that we know more than enough ... to include it at the top of the list of issues deserving serious consideration by policy makers" (U.S. House of Representatives 1995). However, a second scientist in the same hearing wrote of his concern about the continuing increase of CO₂ in the atmosphere, claiming that "we have demonstrated no acceptable scientific basis for predicting catastrophic or near catastrophic effects that would council against a wait, think and see pattern" (U.S. House of Representatives 1995). Such diversity of opinion may signal confusion and ignorance, thereby supporting a rationale for inaction. As one major petroleum corporation states it, "Let's face it: The science of climate change is too uncertain to mandate a plan of action ..." (New York Times 1997).

Uncertainty also plagues the interpretation of science in a second way. For many, the significance of scientific findings is irrelevant to the exigencies of everyday life. A lack of familiarity with scientific methods hinders a ready translation of science into personal choices (Joyce 1995, Smith 1996). Underlying this phenomenon are profound differences in perceptions of space and time of the type that characterize different cultures (Deloria 1995, Abram 1997). For individuals in post-industrial societies, the vast spatial and temporal concerns of science lie far outside their experiential domain of short-term, local events (Caton 1985). Not surprisingly, these differences are reflected in the relatively short cycles of funding and elections that drive policy formulation and decision making and preclude effective treatment of long-term crises in the natural world (Gunderson et al. 1997).

As seen in Fig. 3, the problem is exacerbated by the intricacies and inaccessibility of numeric models, the primary tool for investigating large-scale, complex systems (Oreskes et al. 1994). Science uses a combination of data, theory, and models depending on the particular problem at hand. Increasingly, models are used to address multivariate and large-scale environmental questions such as global climate change. The strength of inference for various scientific activities will differ; generally speaking, there is less confidence in understanding large-scale, complex systems than confined experimental systems described by simple mechanistic hypotheses. Issues such as global change that involve large-scale, complex systems are intrinsically more uncertain. In contrast, traditional experimental science generally retains credibility because it is conducted at scales familiar to most individuals, or at levels of complexity where scientific inference is rarely disputed (e.g., the role of micro-organisms in disease, tidal predictions).

Fig. 3. Uncertainty increases with models of increasing complexity due mainly to the impossibility of testing the hypotheses upon which these models are based.

We propose three general approaches for bridging the science-policy gap. Under the assumption that a shared understanding of science and its implications would help to reconcile the opposing points of view held by science and society, the first and most familiar approach is to directly enhance public confidence by increasing communication (Dovers et al. 1996). There are innumerable examples of the effects of science education and communication on changes in policy via the public. For example, many policy makers and legislators rely upon the views of concerned citizens, scientists, and lobbyists to formulate scientifically valid law and policy (Wynne 1995). Since the 1960s, most national environmental legislation has been prompted, and to a great extent shaped, by increasing public awareness of the scientific aspects of environmental degradation. Citizen groups are increasingly organized and well-versed in the scientific complexities of environmental issues (Dunlap 1992, Steel and Lovrich 1997). As such, they have become increasingly litigious in challenging the practices of government agencies. With the resulting judicial standoff, there are calls for broader participation and collaboration in environmental policy and decision making (e.g., Shindler and Cheek 1999, USDA 1999, USDA Committee of Scientists 1999). When scientists and managers inform and involve their public constituencies in meaningful collaborations, the policy outcome is more likely to be consensus-based and less prone to legal challenge from disaffected stakeholders (Johnson and Campbell 1999; C. Spinos, personal communication).

A second possible approach is to increase confidence by increasing the rate of scientific confirmation. This approach reflects the attitude that scientists can decrease uncertainty sufficiently to allow more precise estimations of risk for policy makers. However, in the case of global climate change, the IPCC (1996) report states that "perhaps the greatest weakness in trying to formulate policy is our many times over demonstrated inability to predict advances in science and technology" (IPCC 1996). This may doom "wait and see" policy options; science, with its large, complex simulation models of possibly chaotic systems may never produce the needed levels of certainty (Casti and Karlquist 1991, Oreskes et al. 1994, Abel 1998).

To account for these seemingly inescapable uncertainties, we propose a third alternative to bridge the science-policy gap: realign the definition of scientific uncertainty as perceived by the public and policy makers with that of the science community. This means that scientific uncertainty must be regarded in the policy arena as it is in scientific circles: as information for hypothesis building, experimentation, and decision making. In effect, the conflicting models and statistical confidence levels that represent the bounds of scientific knowledge would delimit the scope of a flexible science-based policy (Fig. 4). This strategy would recognize that: (1) science and knowledge are intrinsically uncertain, with new information continually altering our perceptions and beliefs; (2) decisions based on scientific information must be made in a context of uncertainty; and (3) faster and better science as an adequate basis for policy formulation is inconsistent with the nature of scientific inquiry and resilient policy formulation.

Fig. 4. Scientific information is best represented by a policy that encompasses the entire envelope of relevant scientific knowledge (including uncertainty). This translates to formulating a policy that spans the range of scientific opinion that has undergone the process of peer review.

This perceptual shift requires policy makers to adopt a rigorous decision-theoretic framework and learning approach to policy formulation in accordance with the tenets of adaptive management (Walters 1986, 1997, Gunderson et al. 1995, Lee 1999). While there are significant obstacles to achieving such a rapprochement between science and policy (Walters 1997, Johnson and Campbell 1999, Lee 1999, Shindler and Cheek 1999), new technologies and approaches for improving environmental planning and decision making are emerging (Reynolds et al. 1996, Lee and Bradshaw 1998, Berg et al. 1999). They will be most effective when used to enhance social learning that is linked with social action (Walters 1986, Gunderson et al. 1995). A corollary implies that scientists need to effectively articulate the true nature of science to the public and policy makers. Moreover, activities such as monitoring, designed and performed in partnership with citizens, science, and managers, can enhance public and institutional learning, especially if integrated into a statistically sound framework for decision making (Lee and Bradshaw 1998).

Finally, as demands for more predictability have increased, the science community has become averse to risk. The charged atmosphere surrounding environmental issues threatens to obfuscate and undermine valid scientific inference (Ludwig et al. 1993). Without the freedom to engage in self-examination and self-doubt, scientific quality and integrity are diminished. This freedom, and the corresponding uncertainty, is the essence of scientific inquiry.

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.

The authors would like to thank D. Lach, P. L. Ringold, K. Ronnenberg, R. Haynes, and H. Weeks for comments on an earlier manuscript and, in particular, the insights of three anonymous reviewers and C. S. Holling.

Abel, T. 1998. Complex adaptive systems, evolutionism, and ecology within anthropology: interdisciplinary research for understanding cultural and ecological dynamics. Georgia Journal of Ecological Anthropology 2: 6-29.

Abram, D. 1997. Spell of the sensuous: perceptual language in a more-than-human world. Vintage, New York, New York, USA.

Adams, R. L. A. 1973. Uncertainty in nature, cognitive dissonance, and the perceptual distortion of environmental information. Economic Geography 49: 287-297.

Berg, J., B. Bradshaw, J. Carbone, C. Chojnacky, S. Conroy, D. Cleaves, R. Solomon, and S. Yonts-Shepard. 1999. Decision Protocol 2.0. U.S. Forest Service FS-634.

Beuter, J. H., K. N. Johnson, and H. L. Scheurman. 1976. Timber for Oregon' s tomorrow: an analysis of reasonably possible occurrences. Research Bulletin 19, Oregon State University, School of Forestry, Forest Research Laboratory, Eugene, Oregon, USA.

Briskin, A. 1998. The stirring of the soul in the workplace. Berret-Koehler, San Francisco, California, USA.

Casti, J. L., and A. Karlquist. 1991. Beyond belief: randomness, prediction, and explanation in science. CRC, Boca Raton, Florida, USA.

Caton, W. 1985. Overshoot. University of Washington Press, Seattle, Washington, USA.

Deloria, V., Jr. 1973. God is red. Grosset & Dunlap, New York, New York, USA.

Deloria, V., Jr. 1995. Red earth, white lies: Native Americans and the myth of scientific fact. Scribner & Sons, New York, New York, USA.

Dorner, D. 1996. The logic of failure: recognizing and avoiding error in complex situations. Addison Wesley, Reading, Massachusetts, USA.

Dovers, S. R., T. W. Norton, and J. W. Handmer. 1996. Uncertainty, ecology, sustainability, and policy. Biodiversity and Conservation 5: 1143-1167.

Dunlap, R. 1992. Trends in public opinion toward environmental issues: 1965-1990. Pages 89-116 in R. Dunlap and A. Mertig, editors. American Environmentalism: The U.S. Environmental Movement, 1970-1990. Taylor and Francis, Philadelphia, Pennsylvania, USA.

Festinger, L. 1957. A theory of cognitive dissonance. Stanford University Press, Stanford, California, USA.

Feynman, R. P. 1998. The meaning of it all: thoughts of a citizen scientist. Perseus, New York, New York, USA.

Funtowicz, S. O., and J. R. Ravetz. 1990. Uncertainty and quality in science for policy. Kluwer Academic, Dordrecht, The Netherlands.

Gelbspan, R. 1997. The heat is on: the high stakes over the earth's threatened climate. Addison-Wesley, Reading, Massachusetts, USA.

Gunderson, L. H., C. S. Holling, and S. S. Light. 1995. Barriers and bridges to the renewal of ecosystems and institutions. Columbia University Press, New York, New York, USA.

IPCC (Intergovernmental Panel on Climate Change). 1990. Climate change: the IPCC scientific assessment. Cambridge University Press, Cambridge, UK.

IPCC. 1996. Climate change 1995: the science of climate change. Cambridge University Press, Cambridge, UK.

Jasanoff, S., and B. Wynne. 1998. Science and decision making. Pages 1-87 in S. Raynor and E.L. Malone, editors. Human choice and climate change. Volume one. The societal framework. Batelle Institute, Columbus, Ohio, USA.

Johnson, B., and R. Campbell. 1999. Ecology and participation in landscape-based planning within the Pacific Northwest. Journal of Policy Studies. 27(3): 502-509.

Joyce, L. A., editor. 1995. Productivity of America's forests and climate change. General Technical Report RM-271. U.S. Forest Service, Rocky Mountain Research Station, Fort Collins, Colorado, USA.

Kuhn, T. 1962. Structure of scientific revolution. University of Chicago Press, Chicago, Illinois, USA.

Lee, K. N. 1993. Compass and gyroscope: integrating science and politics for the environment. Island Press, Washington, D.C., USA.

Lee, K. N. 1999. Appraising adaptive management. Conservation Ecology 3(2): 3. [online] URL: http://www.consecol.org/vol3/iss2/art3

Lee, D. C., and G. A. Bradshaw. 1998. Making monitoring work for managers. U.S. Forest Service. [online] URL: http://www.icbemp.gov/spatial/lee_monitor/begin.html

Lemons, J. 1996. Scientific uncertainty and environmental problem solving. Blackwell Science, Cambridge, Massachusetts, USA.

Ludwig, D., R. Hilborn, and C.J. Walters. 1993. Uncertainty, resource exploitation and conservation: lessons from history. Science 260: 17, 36.

Manning, E.W. 1988. Models and the decision maker. Pages 3-7 in R. Gelinas, D. Bond, and B. Smit. Perspectives on Land Modelling. Workshop Proceedings. Polyscience, Montreal, Quebec, Canada.

Martin, B., and E. Richards. 1995. Scientific knowledge, controversy, and public decision making. Pages 506-525 in S. Jasanoff, E. Markle, J. C. Petersen, and T. Pinch, editors. Handbook of Science and Technology Studies. Sage, Thousand Oaks, California, USA.

Morgan, M. G., and M. Henrion. 1990. Uncertainty: a guide to dealing with uncertainty in quantitiative risk and policy analysis. Cambridge University Press, Cambridge, UK.

Nader, L. 1996. Naked science: anthropological inquiry into boundaries, power, and knowledge. Routledge, New York, New York, USA.

New York Times. 1997. Reset the alarm. New York Times, New York, New York, USA.

The Oregonian. 1998. For Oregon, the global is local. The Oregonian, Portland, Oregon, USA.

Oreskes, N., K. Schrader-Frechetee, and K. Belitz. 1994. Verification, validation, and conformation of numerical models in the earth sciences. Science 263: 641-646.

Peterman, R. M., and C. N. Peters. 1998. Decision analysis: taking uncertainties into account in forest resource management. Pages 105-127 in V. Sit and B. Taylor, editors. Statistical methods for adaptive management studies. Land Management Handbook Number 42. British Columbia Ministry of Forestry, Research Branch, Victoria, British Columbia, Canada.

Raynor, S., and E. L. Malone. 1997. Zen and the art of climate maintenance. Nature 390: 332-334.

Reischauer, E. O., and J. K. Fairbank. 1960. East Asia, the great tradition: a history of East Asian civilization. Volume one. Houghton Mifflin, Boston, Massachusetts, USA.

Reynolds, K., P. Cunnignham, L. Bednar, M. Sanders, M. Foster, R. Olson, D. Schmolt, D. Latham, B. Miller, and J. Steffenson. 1996. A knowledge-based information management system for watershed analysis in the Pacific Northwest U.S. AI Applications 2(10): 9-22.

Risbey, J., M. Handel, and P. Store. 1991. Should we delay responses to the greenhouse issue and do we know what difference a delay means? EOS Trans. AGU. 72(53): 593.

Schick, F. 1997. Making choices: a recasting of decision theory. Cambridge University Press, New York, New York, USA.

Shackley, S., P. Young, S. Parkinson, and B. Wynne. 1998. Uncertainty, complexity and concepts in climate change modeling: are GCMs the best tools? Climate Change 38(2): 159-205.

Shackley, S., and B. Wynne. 1996. Representing uncertainty in global climate change science and policy: boundary-ordering devices and authority. Science, Technology, and Human Values 21(3): 275-302.

Shindler, B., and K. Cheek. 1999. Integrating citizens in adaptive management: a propositional analysis. Conservation Ecology 3(1): 9. [online] URL: http://www.consecol.org/vol3/iss1/art9

Schlesinger, M., and Jiang, X. 1991. Revised projection of future greenhouse warming. Nature 350: 219.

Sims, J. H., and D. D. Baumann. 1974. Human behavior and the environment: interactions between man and his physical world. Maaroufa, Chicago, Illinois, USA.

Smith, J. B. 1996. Adapting to climate change: assessments and issues. Springer-Verlag, New York, New York, USA.

Steel, B., and N. Lovrich. 1997. An introduction to natural resource policy and public lands: changing paradigms and values. Pages 3-15 in B. Steel, editor. Public lands management in the west. Praeger, New York, New York, USA.

UN (United Nations). 1997. Protocols of the United Nations Framework Convention on Climate Change. UN, Tokyo, Japan.

U.S. House of Representatives. 1995. Scientific integrity and public trust: the science behind federal policies and mandates: case study 1, stratospheric ozone, myths and realities: hearing before the Subcommittee on Energy and Environment of the Committee on Science, U.S. House of Representatives, One Hundred Fourth Congress, first session, September 20, 1995. U.S. Government Printing Office, Washington, D.C., USA.

USDA (U.S. Department of Agriculture). 1999. National forest system land and resource management planning. Federal Register 64(192): 54074-54112.

USDA, Committee of Scientists. 1999. Sustaining the people's land: recommendation for stewardship of the national forests and grasslands into the next century. USDA, Washington, D.C., USA.

USDA, Office of Forestry and Economic Assistance. 1996. The Northwest Forest Plan: a report to the President and Congress. USDA, Portland, Oregon, USA.

USDA and USDI (U. S. Department of the Interior). 1994. Record of decision for amendments to Forest Service and Bureau of Land Management planning documents within the range of the Northern Spotted Owl; standards and guidelines for management of habitat for late-successional and old-growth forest related species within the range of the Northern Spotted Owl. U.S. Government Printing Office, Washington, D.C., USA.

Vitousek, P. M., H. A. Mooney, J. Lubchenco, and J. M. Melillo. 1997. Human domination of earth's ecosystems. Science 277: 494-499.

Walters, C. 1986. Adaptive management of renewable resources. McMillan, New York, New York, USA.

Walters, C. 1997. Changes in adaptive management of riparian and coastal ecosystems. Conservation Ecology 1(2): 1. [online] URL: http://www.consecol.org/vol1/iss2/art1

Wynne, B. 1995. Public understanding in science. Pages 361-388 in S. Jasanoff, E. Markle, J.C. Petersen, and T. Pinch, editors. Handbook of Science and Technology Studies. Sage, Thousand Oaks, California, USA.

Yaffee, S. L. 1994. The wisdom of the spotted owl: policy lessons for a new century. Island Press, Washington, D.C., USA.

Yearley, S. 1995. Environmental challenges to science studies. Pages 457-479 in S. Jasanoff, E. Markle, J.C. Petersen, and T. Pinch, editors. Handbook of Science and Technology Studies. Sage, Thousand Oaks, California, USA.