|
|
|
Copyright © 2000 by The Resilience Alliance
The following is the established format for referencing this article:
Vanclay, J. K. 2000. Scientific research or advocacy? Emotive labels and selection bias confound survey results. Conservation Ecology 4(1): r8. [online] URL: http://www.consecol.org/vol4/iss1/resp8/
A version of this article in which text, figures, tables, and appendices are separate files may be found by following this link.
Response to Robert Costanza 2000. "Visions of alternative (unpredictable) futures and their use in policy analysis" Scientific Research or Advocacy? Emotive Labels and Selection Bias Confound Survey Results Jerome K. Vanclay
Southern Cross University
Robert Costanza presents four compelling visions of the future, but the language he uses to describe them is emotive and value-laden and may bias the survey results. The descriptions and analogies used may evoke responses from the survey participants that reveal more about their reactions to the description than their attitudes toward a given scenario. It is hypothesized that the use of more neutral language may lead to more support for the scenario involving "self-limited consumption with ample resources" that Costanza calls "Big Government." If this hypothesis is correct, then the skeptic's policy that Costanza appears to prefer has the additional advantage of always leading to the favored outcome, regardless of the state of the world.
KEY WORDS: advocacy, selection bias, survey.
Published: June 20, 2000
Although Costanza's (2000) argument about envisioning is compelling, the empirical analysis of his case study raises some issues that detract from his discussion.
The labels and descriptions Costanza uses to characterize his visions of "Ecotopia," "Star Trek," "Big Government," and "Mad Max" are graphic but also emotive and value-laden. As a result, they may have prompted prejudiced responses before the participants had fully digested the scenarios. In particular, two of the scenarios are described as "nightmares," which may have contributed to the low scores on the top-right to bottom-left diagonal in the payoff matrix in Costanza's Fig. 3.
Participants who came to listen to Costanza (and who participate in the on-going survey in Conservation Ecology) are likely to be concerned about the environment and the depletion of limited resources. As a result, they are probably already inclined toward the "technological skeptic" view rather the "technological optimist" view of the world. This could have been a factor in the high scores assigned to the Ecotopia scenario.
The two scenarios that arise from the skeptic's view are reached via contrasting pathways: one of these is Ecotopia, where celebrities encourage the public to espouse ecological causes, and the other is Big Government, where the government enforces behavioral change. Some participants may have reacted to the process rather than the final scenario, which would also have resulted in higher scores for the Ecotopia scenario.
My preconception and null hypothesis are that informed citizens would tend to prefer the Big Government scenario, because it could, in principle, accommodate all the attractions of Ecotopia without the resource limitations. However, this is not borne out in the survey results.
While Costanza has not (yet) provided evidence to refute my hypothesis, there is a clear tendency to favor the Ecotopia scenario. Is my null hypothesis mistaken, or is there a bias in the survey methodology? Is it because Costanza writes more passionately about the Ecotopia alternative, or because participants are reacting to the way the scenarios unfold rather than to the final scenario itself?
I'll consider these possibilities in turn. I will not dwell on the issue of selection bias, because it is addressed elsewhere (e.g., Hug 1999), and because Costanza acknowledges the need for a scientific random sample.
I believe my null hypothesis to be sound. I base my assumption about Big Government on three observations:
There is no doubt that the labels used to characterize the scenarios are memorable icons. The question is to what extent emotive labels such as "Big Government: Reagan's worst nightmare" influence scoring by respondents. Haphazard trials suggest that the effect may be significant. I suspect that, if Costanza's Fig. 1 was presented a little differently (e.g., as in my Table 1), it would elicit a different response, in which the bottom-left cell (Costanza's "Big Government"; my "Life of leisure") would achieve higher scores and in which the "skeptic's view" would always lead to a preferable outcome. Although instructive, this alternative presentation is also flawed by many loaded terms. It is not easy to devise good labels that are value free!
Table 1. Another way to contrast the alternative scenarios.
| Resource use | Resource availability | |
|---|---|---|
| unlimited | limited | |
| extravagant | free-for-all | fight for the fittest |
| controlled | life of leisure | frugal existence |
The description of Big Government is a litany of death and disaster. Negative terms such as "scare," "disaster," "uninhabitable," "liability," "destruction," "devastating," and "doomed" appear several times. Although the description ends on a positive note, the unfolding scenario is gloomy. In contrast, Ecotopia is upbeat throughout. We frequently see words like "participatory," "popular," "positive," "sustainable," and "fair." In Big Government, we hear of eight billion people, but in Ecotopia we hear only of small villages. Dispassionate descriptions of the four scenarios are needed before a supplementary sample can offer a scientific basis for comparing participant preferences.
It is not only the language but also the unfolding of the scenario that detracts from the Big Government alternative. Costanza invokes a crisis in corporate America, a nuclear disaster in Europe, and unseasonable flooding before he admits an equitable society regulated by Government. But the scenario need not unfold this way; there are many other realistic ways that reform could be triggered without invoking a Wall Street crisis that might jeopardize people's pension funds. For instance, the scenario could begin …
"The turning point came in 2001, after a shooting tragedy at an American school was attributed to the influence of violent video games. More and more Americans rejected the view that corporations should be allowed to seek a profit at any cost, and demanded greater civic responsibility from corporate America and from all levels of government. This coincided with the spread of social democracy from Scandinavia throughout Europe …"
The unfolding scenario could be told in neutral language and driven by the will of the people rather than dictatorial government. Such a description may attract higher scores for this scenario.
Costanza's Fig. 1, which displays his visions, and his Fig. 3 payoff matrix are both 2 x 2 tables, but he appears to be sampling four cells selected from a larger three-dimensional matrix. He emphasizes two axes, "World View" and "Real State of the World" ("Resource use" and "Resource availability" in my table), but also seems to canvass a hidden axis relating to power structures. In Ecotopia, power is apparently resolved democratically at the village level, and national government is scarcely visible, whereas the Big Government scenario relies on a socialist central government. These alternatives are not on an even footing.
It appears that Costanza, believing that the skeptics are right, is urging society to take steps to ensure that we end up with Ecotopia rather than Mad Max, and I commend him for his efforts. However, his paper doesn't attempt to forecast the future Real State of the World, but rather the attitudes of respondents toward the alternative visions outlined. Science requires that we discriminate between advocacy and impartial scientific survey.
As it happens, the consequences of any bias appear small. As Costanza points out in his Worst Case Analysis, his survey leads to a payoff matrix in which the standard maximin decision rule leads us to adopt the skeptic's policy, a policy that keeps all options open. However, if my hypothesis is correct, the skeptic's policy has the added advantage of always leading to the favored outcome, irrespective of the state of the world.
Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.
Costanza, R. 2000. Visions of alternative (unpredictable) futures and their use in policy analysis. Conservation Ecology 4(1):5 [online] URL: http://www.consecol.org/vol4/iss1/art5
Hug, S. 1999. The perils of selection bias: incomplete datasets in studies of ethnic conflict, new social movements and new political parties. http://www.unige.ch/ses/spo/staff/simon/sel/selaj.pdf
Address of Correspondent:
Jerome K. Vanclay
School of Resource Science and Management
Southern Cross University
P.O. Box 157
Lismore NSW 2480 Australia
Phone: +61 2 6620 3147
Fax: +61 2 6621 2669
JVanclay@scu.edu.au
