INTRODUCTION

Citizens have been collecting data on environmental systems through participatory monitoring programs since the late 19th century. Weather observers have assessed rainfall and air temperatures since as early as 1890 (http://www.nws.noaa.gov/om/coop/). Citizens have collected data about bird populations through an organized program in the United States since 1900 (Lee 1994) and in the United Kingdom since 1962 (Greenwood 2003). They have monitored fish populations with the National Marine Fisheries Service since 1954 (Lee 1994).

With the advent of participatory decision-making initiatives across the globe (Rydin and Pennington 2000, Ellis and Waterton 2004, Van Rijsoort and Jinfeng 2005) and shrinking or limited monitoring budgets (U.S. Office of Management and Budget 2012), involving citizens in environmental monitoring is becoming more common. For example, in the United States, a 1998 survey by the U.S. Environmental Protection Agency identified nearly 800 volunteer environmental monitoring programs (Ely and Hamingson 1998). This number has more than doubled in the past 15 years. Results of a recent survey suggest there are at least 1675 volunteer water monitoring programs in the United States (Stepenuck 2013). The types of volunteer monitoring efforts are also diversifying. In addition to weather, bird, and water monitoring, citizens also monitor a multitude of types of environmental systems and organisms, including air (O’Rourke and Macey 2003), amphibians (Shirose et al. 1997), biodiversity (Cosquer et al. 2012), mammals (Moyer-Horner et al. 2012), invasive species (Gallo and Waitt 2011), forestry (Ballard et al. 2008), marine environments (Evans et al. 2000), and bees (Ashcroft et al. 2012). These efforts have produced invaluable data that have been used by scientists to track changes in populations and environmental trends over time (e.g., Ticheler et al. 1998, Boylen et al. 2004, Bhattacharjee 2005, Greenwood 2007, Dickinson et al. 2012) and by managers to modify practices to eliminate pollution (Da Silva Pinho 2000).

A variety of outcomes have also occurred for participants in volunteer environmental monitoring programs and for the communities in which these programs operate, but efforts to synthesize these have been limited. As a result, practitioners may lack knowledge to effectively model successful initiatives. This can lead to waste of valuable resources to develop effective programs, limit external funding support, and potentially stymie participation if personal benefits are not recognized and promoted (Measham and Barnett 2008). However, 4 notable studies provide a basis on which to build a current synthesis. The earliest explored participant empowerment and natural resource management outcomes that resulted from 15 locally based monitoring projects (Danielsen et al. 2005a). Four generalized benefits were found to have resulted across programs: improved communication between government and local stakeholders, increased knowledge and changed attitudes among participants, better adherence to natural resource regulations by community members, and empowerment of local stakeholders. The second study investigated 10 years of citizen science–related literature and relevant websites to identify benefits (Conrad and Hilchey 2011). These included science and the environment becoming more available to the public, local stakeholders becoming more engaged in ecosystem management and policy discussions, participants increasing scientific literacy, and social capital being built in communities in which either community-based monitoring or management were carried out. The third study considered the distinction between individual and community learning outcomes using examples from peer-reviewed literature to demonstrate outcomes (Jordan et al. 2012). Identified outcomes for individuals included increased knowledge, engagement in the scientific process, and environmental stewardship. Identified community outcomes included improved social capital, increased trust among stakeholders and land managers, and job creation. The fourth study categorized volunteer monitoring efforts by roles of the laypersons and scientists involved and then predicted outcomes based on these characterizations (Shirk et al. 2012). Science, social-ecological, and individual outcomes were predicted. Considering the latter two, reported social-ecological outcomes included improved environmental management and conditions. For individuals, outcomes included knowledge gain, stewardship actions, a sense of ownership, and broadened networks.

We build on the work of these researchers by providing a synthesis of peer-reviewed journal articles through 2012 that reported outcomes for individuals and communities resulting from volunteer environmental monitoring efforts. We set the situation by describing the realm of research focused on participatory environmental monitoring. We present the types of outcomes that have been observed and described in peer-reviewed journals. Finally, the discussion reflects on reported outcomes, identifies research gaps, and proposes suggestions for future investigations.

METHODS

Between October 2009 and October 2013, a series of online searches was conducted to identify peer-reviewed journal articles related to the field of participatory environmental monitoring. This included a search for articles that utilized layperson-generated data to build understanding of the populations and environments monitored, as well as articles that described, supported, questioned, and/or assessed outcomes of such participatory environmental monitoring efforts. The goal was to develop a broad understanding of the collective state of knowledge presented in peer-reviewed journals about participatory environmental monitoring and its impacts.

Initially, Proquest’s Ecology, Environmental Sciences and Pollution Management, and Water Resources Abstracts databases were searched using a variety of terms. These included: “volunteer monitoring,” “volunteer water monitoring,” “citizen science,” “community-based collaborative monitoring,” “participatory monitoring,” “public participation in scientific research,” “locally-based monitoring,” “community-based monitoring,” and “environmental collaborative monitoring.” The articles identified in these searches were reviewed, and additional volunteer environmental monitoring–related peer-reviewed research articles cited within them were located and examined. The newly found articles were then used to identify additional relevant research articles. In an effort to perform as comprehensive a search as possible, the Proquest databases were searched periodically over the 4-year time period to identify newly published or previously missed peer-reviewed articles related to participatory environmental monitoring. In 2012 and 2013, Google Scholar and EBSCO Host’s Social Sciences Full Text and Wildlife and Ecology Studies databases were also searched.

This iterative search process ultimately resulted in 436 peer-reviewed journal articles related to participatory environmental monitoring, the majority of which were published in the past decade (Table 1). Articles related to astronomy, geographic information systems, and public health, other than as related to water and air monitoring, were not included.

Review of the first 271 articles located by the end of 2010 allowed them to be categorized according to topic of focus. This was undertaken in an effort to broadly characterize peer-reviewed literature across the field. These articles fit into 8 general categories. As additional articles were located, they were assigned to these categories. Some articles were placed into more than 1 category. For the 436 articles located, the categories and count of articles included those that assessed participatory environmental monitoring programs (69), provided overviews of these programs (76), presented comparison studies of professional versus volunteer methods (61), discussed how to improve these programs (12), expressed support for these types of programs (62), discussed techniques for managing these programs over time (11), used layperson-generated data to describe a population or environmental condition (150), and discussed how participatory data might be used (18).

A collection of peer-reviewed articles that reported outcomes for individuals or communities was derived from the category of those that assessed participatory environmental monitoring programs. This collection is our focus. Importantly, it does not include research that reported on environmental monitoring conducted solely as part of a school curriculum or monitoring in which laypersons were paid to participate. Included articles focus specifically on volunteer environmental monitoring. These criteria differ somewhat from previous synthesis articles that discussed outcomes of citizen science programs for individuals and communities (Danielson et al. 2005a, Conrad and Hilchey 2011, Jordan et al. 2012, Shirk et al. 2012). However, defining the criteria in this manner allows the discussion to focus on individual and community outcomes of monitoring efforts in which layperson participants took part without monetary incentive or requirement, either of which may affect outcomes. Also, including only research that has been vetted through the peer-review process enhances the credibility of the synthesis and thus addresses this known challenge for participatory environmental monitoring (Crall et al. 2010, Ottinger 2010).

General information that characterized each volunteer monitoring effort was determined. This included determining the monitoring focus, location, number of programs assessed, and timescale to achieve outcomes. When authors did not report the length of time to reported outcomes, program age and article publication date were used to determine a maximum time period over which outcomes may have resulted.

To assess the quality of research conducted, journal impact factors and study design styles were determined for each article. In September 2013, the ISI Web of Knowledge Journal Citation Reports was used to identify impact factors for 2012. ResearchGate and journal websites were used to determine impact factors for specialty journals not listed in the Journal Citation Reports. Study design styles included the following: (1) research that employed surveys; (2) research in which interviews were conducted; (3) research in which data related to the effort were analyzed, including environmental monitoring data, content of electronic discussions and media reports about the efforts, and unsolicited participant feedback about the efforts; (4) research in which participants were observed by the researcher(s); and (5) research that provided evidence of outcomes as determined by anecdotal researcher reflection of the volunteer monitoring effort.

Outcomes reported in the articles were initially classified as affecting either individual participants or the community in which the volunteer monitoring program operated. Subsequently, categories of outcomes reported within these two groups were identified. Because researchers generally used similar language to describe outcomes, several common categories could be identified across the pool of articles. Previous syntheses were also used as a general guide for identifying outcome categories (Danielsen et al. 2005a, Jordan et al. 2012).

RESULTS

Just 35 of the larger pool of articles (436), plus the 4 reviews, reported outcomes for individuals who participated in volunteer environmental monitoring programs or for the communities in which these programs operated (Table 2). In 27 of the 35 nonreview articles, researchers assessed outcomes for a single volunteer environmental monitoring effort. Programs ranged in scope from monitoring in a single community to those focused on as broad a scale as North America. The majority (14) were carried out in the United States. About half as many were carried out in the United Kingdom and Canada, whereas other countries had lesser representation. Volunteer monitoring efforts often focused on living organisms, although they also monitored land, air, and water resources. Programs in which a variety of aspects of the environment were monitored were most often considered in regard to outcome assessment.

Although the majority of authors did not specify how long it took for outcomes to be achieved, most outcomes reported occurred within 8 years or less following initiation of monitoring (Table 2). Four articles reported on outcomes that were achieved within a year. Outcomes included building individual knowledge and community awareness (Evans et al. 2000, Overdevest and Mayer 2007, Kountoupes and Oberhauser 2008) and changing behaviors (Jordan et al. 2011). The minimum reported time to achieve policy-related outcomes was 2 months following data sharing (Becker et al. 2005). However, this outcome was following a year of monitoring.

Journal impact factors ranged from 0.707 to 5.236 across the 28 journals in which outcome-related articles were published. Four specialty journals did not have defined impact factors (Table 3). The types of outcomes most often reported in articles published in journals with the 10 highest impact factors included personal knowledge and community awareness building, personal benefits, and changes in attitudes and behaviors (Tables 4 and 5). Only 3 reported that volunteer monitoring influenced natural resource management practices or policies (Greenwood 2003, Overdevest and Mayer 2007, Ottinger 2010), and 2 discussed challenges or successes volunteers have had as participants in the decision-making process (Ballard et al. 2008, Fernandez-Gimenez et al. 2008).

Researchers employed between 1 and 4 methods to assess outcomes (Table 3). Interviews were the most commonly used method, reported in 16 of the articles. Anecdotal researcher reflection was next most prevalent, used by authors of 12 articles. Surveys were carried out to assess outcomes by authors of 11 articles, and outcomes reported in 9 articles were assessed, at least in part, based on review of data generated through the volunteer monitoring program. Finally, researchers of 4 articles determined outcomes based in part on participant observation. Data review and participant observation tended to be employed more often in articles published in journals with higher impact factors, whereas the other methods were used in articles published in journals representing the range of impact factors. However, no difference was observed in the types of outcomes reported when data analysis or participant observation techniques were used as compared with other research methods (Tables 4 and 5).

Five categories of outcomes were identified for individuals (Table 4). Assessing whether participants experienced a gain in knowledge was the most common; this was reported in 18 articles. The next most common, described in 11 articles, related to social and personal benefits of participation. Authors of 10 articles considered whether volunteer monitors had improved their ability to participate in decision-making processes. Nine authors also evaluated and reported on participants’ modified attitudes or behaviors. Finally, authors of 3 articles considered whether volunteer monitors had increased the amount and effectiveness of their civic participation since starting their involvement.

Community outcomes were divided into 4 categories (Table 5). Increased social capital, i.e., the economic and personal benefits gained by interactions among community members (Coleman 1988), and that volunteer environmental monitoring influenced natural resource management practices or policies were each reported in 12 articles. Increased community knowledge and awareness was reported in 11 articles. Changes in attitudes and behaviors were less often reported, with 5 articles describing such outcomes.

Outcomes for individuals

Gain in knowledge

Three main types of knowledge gain were reported for individuals involved with volunteer environmental monitoring programs. These included gaining scientific content knowledge, learning new skills, and experiencing social learning. Social learning can be described as learning as a result of interactions with others, which results in reassessment and potential change in an individual’s underlying assumptions about a group or an issue (Ballard et al. 2008, Fernandez-Gimenez et al. 2008). Learning was assessed primarily through interviews and surveys (Table 3).

Gain in content knowledge was, by far, the most reported of the 3 types of learning. It was considered in 18 articles, with positive outcomes reported in 16 and a negative or no change in knowledge reported in only 4. Content learned generally reflected the topic of focus in the monitoring effort, although some researchers identified knowledge gain that went beyond the immediate object of observation (Table 4). For instance, individuals in one effort gained understanding of the need for natural resource management regulations as they monitored fishing activities and observed negative impacts on fish populations (Obura et al. 2002). In 2 studies in which no difference in scientific knowledge was observed between experienced and inexperienced volunteers, researchers speculated that individuals with strong science backgrounds had elected to participate (Trumbull et al. 2000, Overdevest et al. 2004).

Learning new skills as an outcome of volunteer monitoring participation was reported in just five articles (Table 4). Very few explored this type of learning in depth. As expected, volunteers learned skills related to the process of monitoring (Obura et al. 2002, Bell et al. 2008, Cosquer et al. 2012). Moving a step further to explore skill building, some participants were reported to have learned skills related to community leadership, activism, media engagement, and advocacy (Measham and Barnett 2008).

Evidence of social learning, although not always called that, was described in five articles (Table 4). Two focused on understanding learning that occurred within monitoring groups carrying out community-forestry demonstration projects in the western United States (Ballard et al. 2008, Fernandez-Gimenez et al. 2008). Participants noted changes in personal feelings and assumptions based on interactions within the diverse group of players in the program. This was true for both scientists and laypersons involved in the efforts. When making management decisions, scientists learned the importance of local knowledge, and local residents better understood the importance of collecting data as evidence (Ballard et al. 2008). The more diverse the group of stakeholders that designed the monitoring, and the more rigorous the data collection, the more evidence of social learning that was observed (Fernandez-Gimenez et al. 2008). Social learning also included participants increasing their understanding of ecological processes, which resulted in altering their own underlying assumptions (Ballard et al. 2008).

Change in attitudes and/or behaviors

As evidenced by the examples discussed previously, changes in participant attitudes can be directly linked with social learning (Ballard et al. 2008, Fernandez-Gimenez et al. 2008). Aside from these social learning examples, only one article discussed individual attitudinal changes (Table 4). Specifically, no significant changes in volunteer bird monitors’ attitudes about science or the environment were observed in pre- and posttests (Brossard et al. 2005).

Authors of six articles reported that participants’ behaviors had changed (Table 4). Volunteers were reported to have increased their political participation (Overdevest et al. 2004) and altered their land-use management techniques (Becker et al. 2005, Evans et al. 2005, Van Rijsoort and Jinfeng 2005, Jordan et al. 2011, Cosquer et al. 2012). However, behavior changes were sometimes more superficial than desired. In one study, although the majority of invasive plant monitors reported altering their behavior in at least one way, most changes were passive (Jordan et al. 2011). For instance, participants noticed invasive plants more while hiking, but none changed their voting practices related to invasive plants. This was attributed to participants not feeling confident that active behavioral changes would positively impact the environment.

Attainment of social and personal benefits

A wide variety of social benefits to individuals were noted as important outcomes for volunteer environmental monitors (Table 4). Friendships and social networks were built (Gooch 2004, 2005, Kountoupes and Oberhauser 2008, Measham and Barnett 2008, Koss and Kingsley 2010). Participants enjoyed the work they did (Gooch 2005, Kountoupes and Oberhauser 2008, Lawrence 2009, Koss and Kingsley 2010) and were proud of their accomplishments (Kountoupes and Oberhauser 2008, Measham and Barnett 2008, Koss and Kingsley 2010). They had something tangible and meaningful in which to put their energy, giving them purpose (Measham and Barnett 2008, Lawrence 2009, Koss and Kingsley 2010) and feelings of empowerment and self-worth (Gooch 2005). They became attached to places (Gooch 2005). Some even based career choices on experience gained through participation in such programs (Newman et al. 2003, Gooch 2005).

Two articles reported negative social outcomes. In one study, volunteers felt overtaxed because of cuts in professional staff and increasing reliance on volunteers (Measham and Barnett 2008). In the other, engagement in environmental monitoring efforts was reported to lead participants to have concerns about the state of the environment (Lawrence 2009).

Attainment of voice in decision making

Volunteer monitors contributed to decision-making processes in a number of programs, but concerns about their effectiveness were also expressed. Eight articles reported positive outcomes, whereas five mentioned concerns (Table 4).

Positive outcomes related to citizens feeling empowered and more confident to express their ideas to natural resource managers and figures of authority (Obura et al. 2002, Gooch 2004, 2005, Ballard et al. 2008, Cornwell and Campbell 2012). In part, this was attributed to citizens recognizing the power in having scientific data to support their management requests (Sharpe et al. 2000, Cornwell and Campbell 2012). Ultimately, there was evidence of natural resource management rules and regulations being developed with local volunteer input, which represented a change from past practices (Danielsen et al. 2005b, Van Rijsoort and Jinfeng 2005).

Concerns expressed about the effectiveness of volunteer monitors in decision-making processes were related to both passive and active engagement of citizens. A plant monitor in the United Kingdom expressed frustration that policy makers failed to use citizen-generated data to inform decisions even though the volunteer had taken no action to share results with decision makers beyond submitting data to the monitoring program (Ellis and Waterton 2004). Citizens who took a more active approach to contribute to decision making also faced challenges. For instance, when participating in agency and government board meetings, volunteers in Australia often felt left out and manipulated (Gooch 2004, 2005). Ultimately, they were frustrated with one-sided outcomes. Additionally, “internal power differentials” were cited as a challenge to effective citizen participation in decision making in community-based forest monitoring projects in the western United States (Fernandez-Gimenez et al. 2008:14).

Increase in amount and effectiveness of civic participation

Research findings nonetheless demonstrate that civic environmental actions of volunteer monitors can increase over time with their participation in environmental monitoring programs. Three articles reported such outcomes (Table 4). These related to personal activities of volunteers (Overdevest et al. 2004, Jones et al. 2006) and their influence on administrative decisions and processes of industrial companies, thus helping to minimize environmental impacts (O’Rourke and Macey 2003).

Outcomes for communities

Increase in awareness

As with individuals, a motivation for implementing and an often-assumed outcome of volunteer environmental monitoring programs is that community awareness about an ecosystem will increase as a result of public involvement in monitoring and sharing results (Greenwood 2003, Gooch 2005, Fernandez-Gimenez et al. 2008). This expectation was supported across a variety of types of monitoring efforts (Table 5). Public awareness of an environmental issue was also built from media attention that resulted from citizen participation and subsequent broadscale dissemination of scientific information about the topic (Evans et al. 2000, Overdevest and Mayer 2007).

Change in attitudes and behaviors

Attitude and behavior changes of community members are tightly linked with community awareness building. Reported attitude and behavior changes primarily related to altered views of citizens and officials in regard to decision making about land-use management and policies (Table 5). These were assessed almost entirely through researcher reflection. One particularly intriguing example described how monitoring fog capture in highland forests of the Loma Alta Watershed in Ecuador helped community members understand its importance as a water resource to their livelihoods as farmers (Becker et al. 2005). Trees in the highlands were found to collect more fog than agricultural plants in the lowlands of the watershed (Becker 1999). The water in fog that dripped off trees contributed significantly to streams. Thus, forest clear-cutting directly affected community members’ ability to farm, yet no land-use regulations existed. Volunteer data demonstrated that the loss of water in the lowlands as a result of deforestation could cost the agricultural communities half of each family’s annual income. Only 15 months after volunteer monitoring was initiated, 95% of voters supported creation of a forest reserve, demonstrating a significant attitude change among community members. Attitude and behavior changes were also observed among community leaders who, within 10 years, enabled the reserve to grow from 100 to 1600 ha.

Increase in social capital

Another often-proclaimed outcome of volunteer environmental monitoring in the scientific literature is that it builds social capital in communities (Overdevest et al. 2004, Gooch 2005, Jones et al. 2006, Schwartz 2006, Greenwood 2007). Social capital comprises the economic and personal benefits gained for individuals and communities by interactions among community members (Coleman 1988). Communities with more social capital have a better chance of reaching achievable goals than those with less (Krishna 2002). Schwartz (2006) proposed that social capital can be built by engaging citizens in activities in which they can have a direct impact. As previously noted, this was the most reported community outcome among the collection of articles (Table 5). The example from the Loma Alta Watershed illustrates this tenet; once the forest reserve was established, citizens developed a forest conservation area that supported nature tourism (Becker et al. 2005). Further, citizens became naturalist guides, set up entrance fees and regulations for the conservation area, and developed a registration center for visitors. Still others secured funds to build a cabin for use by tourists and developed jewelry to sell to visitors. Other researchers reported increased social capital outcomes related to aspects of social capital such as increased size of personal networks (Overdevest et al. 2004, Gooch 2005, Pollock and Whitelaw 2005, Jones et al. 2006, Measham and Barnett 2008, Koss and Kingsley 2010), development of partnerships (Sharpe et al. 2000, Savan et al. 2003), and community influence (Pollock and Whitelaw 2005, Jones et al. 2006).

Influence on natural resource management practices or policies

Effecting change in natural resource management practices or policies is often the ultimate goal that volunteer monitoring groups seek to achieve (Ellis and Waterton 2004). In fact, examples demonstrating such success span the globe and environmental topics (Table 5). A variety of interim outcomes leading toward this goal have also been reported. Many of these interim outcomes parallel components of social capital. These include increased communication between citizens and managers or government (Obura et al. 2002, Danielsen et al. 2005b) and increased cooperation, collaboration, and trust among stakeholders (Becker et al. 2005, Van Rijsoort and Jinfeng 2005, Fernandez-Gimenez et al. 2008).

Several articles reported that natural resource management practices were improved after volunteer environmental monitoring data demonstrated negative environmental impacts (Obura et al. 2002, Becker et al. 2005, Danielsen et al. 2005b, Van Rijsoort and Jinfeng 2005). There is also evidence that such outcomes can occur in relatively short time periods. For instance, short implementation periods, averaging 97 days from recommendation to implementation, were observed in a program in the Philippines (Danielsen et al. 2005b). Such rapid implementation was attributed to having the same people carry out the monitoring, make the recommendations, and ultimately, make the management decisions.

DISCUSSION

Exploring findings and identifying knowledge gaps

Although limited, peer-reviewed journal articles describing volunteer environmental monitoring efforts contain evidence of a variety of benefits for both individuals and communities. Increased personal content knowledge and improved social capital were the most commonly reported outcomes for individuals and communities, respectively. In general, although potentially valuable to other practitioners, null or negative outcomes were reported less often. Those that were reported most often related to challenges for citizen participants in decision-making processes.

Although not an unexpected outcome, demonstrating knowledge gain among participants is particularly relevant for volunteer environmental monitoring programs, which often set out to educate particular audiences (Pattengill-Semmens and Semmens 2003, Brewer 2006). Thus, its prevalence as a reported outcome addresses a need of many volunteer monitoring efforts. For instance, 82% of volunteer water monitoring programs in the United States listed public education as a key objective in a recent survey (Stepenuck 2013). Demonstrating knowledge gain among participants may help such programs sustain financial and administrative support over time; thus, additional research about such outcomes is justified. Research on educational outcomes can also benefit agencies seeking to develop and incorporate volunteer monitoring efforts into their programming. A weakness of the existing collection of articles is that most focus on only assessing content knowledge. For agencies such as the U.S. Forest Service, which seeks to boost effectiveness in obtaining user feedback to identify problems by incorporating participatory monitoring into its research efforts (McKinley et al. 2012), understanding gains in content knowledge of participants is insufficient. For such agencies, understanding whether volunteer monitoring participants improve knowledge and skills related to diplomacy, public speaking, data synthesis, effective networking, or communications might provide greater insight into the value of volunteer environmental monitoring as a public participation tool. Such knowledge can help agencies plan and implement participatory monitoring efforts most effectively.

It is known that there is a bias against publishing negative or null results (Franco et al. 2014). In this synthesis, null or negative knowledge and attitude/behavior outcomes were reported in just four articles (Trumbull et al. 2000, Overdevest et al. 2004, Brossard et al. 2005, Jordan et al. 2011). Undeniably, there is an inherent risk in sharing null or negative results because doing so may mean loss of support for the program. However, when published, such knowledge affords others the opportunity to learn from reported outcomes and avoid pitfalls (Franco et al. 2014). For instance, the knowledge that volunteers felt overtaxed with the workload as professional staff were cut (Measham and Barnett 2008) is important for program managers to recognize. In volunteer environmental monitoring efforts, which often operate on limited budgets, having knowledge about program concerns may save practitioners considerable time and expense in program development by avoiding known pitfalls.

To address this conundrum, recall that the majority of articles reported on assessment of only a single program. The result is that a knowledge gap exists in understanding outcomes across volunteer environmental monitoring programs. Identifying whether there are commonalities in outcomes across programs that monitor specific organisms or environments, or that use a common design style, may help other practitioners develop initiatives to achieve certain desired goals more effectively. A recommendation for future research is for independent researchers to conduct cross-program outcome assessments, thus linking the need for researchers to publish null or negative outcomes with the idea that assessing outcomes across programs is both valuable and represents a gap in current knowledge. Information learned can be published without identifying program identities. This will enable null or negative outcomes to be shared to benefit other practitioners and will enhance understanding of connections between common program characteristics and outcomes. A recent example of such an assessment exists. Through a survey of 345 volunteer water monitoring programs across the United States, 7 characteristics of programs significantly related to success in natural resource policy and management were identified (Stepenuck 2013).

Reflecting on reported concerns related to citizen participation in decision-making processes, Arnstein (1969) defined a ladder of citizen participation that provides a framework by which to examine these findings. The ladder begins with nonparticipation and moves up to a stage in which citizens have both an opportunity to hear and have a voice in decision making, although decision makers may not heed citizen input. Partnerships between citizens and traditional decision makers come next on the ladder. The ladder culminates at a level in which decision-making power has been delegated to citizens. Concerns expressed about the limitations of the power of citizens’ voices in decision making following participation in volunteer environmental monitoring align with this framework (Obura et al. 2002, Gooch 2004, 2005, Fernandez-Gimenez et al. 2008). The design of most public decision-making processes is prescribed and thus limits how citizens can contribute (Obura et al. 2002). Although citizens might have unique knowledge to share, there is little ability for them to do so in a manner that can effectively inform decisions (Ellis and Waterton 2004). Nonetheless, power sharing between scientists and citizens was demonstrated in community-based forestry projects in the western United States when scientists incorporated the input of citizen participants into reports (Ballard et al. 2008). This suggests that volunteer monitors were able to operate at higher rungs of Arnstein’s ladder; they not only expressed their voice, but they entered into a discussion or were otherwise able to influence the decision-making process. Ultimately, however, it seems unlikely that the pinnacle of Arnstein’s ladder was reached in any of the volunteer monitoring efforts studied. Although a local monitoring program in Africa reported that other community members considered volunteer monitors experts in their field of study (Obura et al. 2002), total decision-making power was never reported to have been transferred to citizens. A research opportunity exists to attempt to better understand the range of outcomes that have been observed related to the effectiveness of citizens’ voices in the decision-making process and ultimately in influencing policy and management decisions. Are there specific characteristics about monitoring program design, participants, or the surrounding political environment that permit the most effective citizen participation? Research addressing these questions might be carried out across programs or may address a single monitoring initiative. One recommendation is for a rigorous data collection and statistical analysis to be carried out to assess such characteristics because this is often lacking in the current field of outcomes-focused research on volunteer environmental monitoring programs.

Interestingly, findings of this synthesis do not seem to align with a recent typology of participatory monitoring programs that distinguishes the degree to which citizens participate in the scientific research process and predicts the speed at which decision-making outcomes result (Shirk et al. 2012). These researchers predicted high potential for “prompt” decision making for projects in which citizens and scientists develop the scientific research together, i.e., cocreated projects. They predicted that decision making would be “slow to result” for projects in which citizens assisted scientists in developing a study and were involved in both data collection and analysis to address a shared research goal, i.e., collaborative projects, or for projects in which citizens’ main role in the scientific research process was to contribute data, i.e., contributory projects. In fact, the two monitoring efforts in this synthesis with the most rapid impacts on policies (Becker et al. 2005, Danielsen et al. 2005b) seem to align more as contributory or collaborative projects. However, decision making within those communities was, in fact, prompt. To explain this discrepancy, an alternative hypothesis to consider is that there is some evidence to suggest that those communities in which volunteer monitoring efforts resulted in the most immediate success in influencing policy changes (e.g., Becker et al. 2005, Danielsen et al. 2005b) faced environmental and/or economic crisis situations. Political science research suggests that a crisis situation can help influence policy change by helping to focus the attention of decision makers on a particular issue (Stern 1997). At such times, groups poised to bring their agenda to decision makers can help drive policy change (Kingdon 1993). As the typology describing cocreated, collaborative, and contributory projects is widely known among practitioners in this field (Cosquer et al. 2012, Dickinson et al. 2012), research to better understand the relationship between program design and speed of influence on decision making is warranted. Such research would also fill a knowledge gap left by the current outcomes-focused articles, which is to define how long until various types of outcomes are achieved. Available results suggest that learning outcomes and attitude and behavior changes can be accomplished within a year from project initiation, but the majority of authors did not explicitly state how long it took to observe various types of outcomes in their programs.

Another knowledge gap identified through this synthesis is that relatively few researchers have considered whether participants or communities have changed attitudes or behaviors as a result of the volunteer monitoring effort. As the world faces the challenges of changing climate and decreasing biodiversity, a focus of many volunteer environmental monitoring efforts, influencing change in human attitudes and behaviors is of utmost importance to protect biodiversity (Heller and Zavaleta 2009). For instance, McKay (2007) reflected that much has been done to document declines in bird populations but questioned if enough has been done to try to change attitudes and behaviors to help minimize these declines. This synthesis demonstrates that there have been encouraging results of attitude and behavior changes to protect natural resources, but only 12 authors have considered this across the ever-growing field of volunteer environmental monitoring. That is a paucity of information from a field that focuses entirely on engaging citizens in scientific research about the environment. Research that has been published about these outcomes tends to be more data rich, having been determined primarily through interviews, surveys, and program data review. A similar style of assessment is recommended for future research about these outcomes as well.

Overall, many opportunities exist for researchers in this domain. Such research will not only inform future program planning and development but also help build the credibility of volunteer environmental monitoring programs as effective contributors to natural resource management and policy decisions.

Limitations of this synthesis

There are limitations that should be acknowledged. First, other valuable information on this subject is available but was not included in this synthesis, which was limited to peer-reviewed journal articles. Excluded were books, book chapters, and non-peer-reviewed reports (e.g., Ely 1994, Fischer 1996, Craig et al. 2003, Ecological Monitoring and Assessment Network and the Canadian Nature Federation 2003, Viswanathan et al. 2004). Second, limitations of the searching techniques may have restricted the pool of peer-reviewed articles. For example, environmental monitoring in which volunteers assisted but were not identified in the article title or key words were not included. Similarly, the diversity of terms used to describe volunteer monitoring efforts varies across fields of science, and some relevant search terms may have been inadvertently excluded.

CONCLUSION

Peer-reviewed journal articles related to volunteer environmental monitoring efforts have reported a variety of outcomes for individuals and communities. These range from increasing participant knowledge and community awareness to promoting changes in attitudes and behaviors, building social capital, and ultimately, influencing change in natural resource management and policies. Mixed results have been reported in regard to citizen success as active participants in the political decision-making processes and in terms of knowledge gained through participation in volunteer monitoring efforts. However, research on such outcomes is limited, representing less than 10% of peer-reviewed journal articles related to volunteer environmental monitoring published through 2012. Thus, many opportunities exist to research and report outcomes of volunteer environmental monitoring efforts. Practitioners, sponsoring agencies and organizations, citizen participants, communities, and ultimately, the environment can benefit from what is learned.

The major recommendations of this research that address identified gaps in knowledge of outcomes of volunteer environmental monitoring efforts include the following:

1. Although important, research focused on identifying knowledge-related outcomes should consider not only content knowledge assessment but also assessment of knowledge and skills related to such things as diplomacy, public speaking, data synthesis, or communications, any of which will provide greater insight into the value of volunteer environmental monitoring as a public participation tool.
2. Considerably few articles reported null or negative outcomes, which can be of value to other similar initiatives. Further, the majority of researchers focused on assessment of just a single volunteer monitoring effort; thus, a gap exists in understanding commonalities in outcomes and program characteristics across volunteer environmental monitoring efforts. To address both of these knowledge gaps, independent research should be conducted across programs to assess outcomes. This will not only inform the field about commonalities in outcomes and characteristics but also enable null or negative results to be anonymously reported, thus protecting programs from potential negative repercussions of reporting such results openly.
3. A number of negative outcomes were reported related to the effectiveness of citizen participation in the decision-making process. An opportunity exists to consider this in more depth, including seeking to understand the role that such aspects as monitoring program design, characteristics of participants, and surrounding political environment may play in effective citizen participation in decision-making processes. For this research, a rigorous method of data collection and statistical analysis should be carried out.
4. The current field of research tends to overlook the time component of outcome achievement, which can be essential to certain volunteer monitoring efforts. Opportunities exist to assess a common monitoring program design typology, which defines programs as cocreated, collaborative, and contributory, among others, as related to time to achieving outcomes and to provide summarized information about a range of times observed to achieve decision-making outcomes in different situations.
5. As the world faces the challenges of changing climate and decreasing biodiversity, both impacted by human activities, attitude and behavior changes that can minimize impacts across the globe will be essential. As volunteer environmental monitoring is designed to engage citizens in scientific research directly related to these issues, understanding to what extent and how attitude and behavior changes have occurred is essential. Existing research fails to address these outcomes sufficiently. Thus, this is a superb opportunity for future research.

LITERATURE CITED

Arnstein, S. R. 1969. A ladder of citizen participation. Journal of the American Planning Association 35:216-224. http://dx.doi.org/10.1080/01944366908977225

Ashcroft, M. B., J. R. Gollan, and M. Batley. 2012. Combining citizen science, bioclimatic envelope models and observed habitat preferences to determine the distribution of an inconspicuous, recently detected introduced bee (Halictus smaragdulus Vachal Hymenoptera: Halictidae) in Australia. Biological Invasions 14(3):515-527. http://dx.doi.org/10.1007/s10530-011-0092-x

Ballard, H. L., M. E. Fernandez-Gimenez, and V. E. Sturtevant. 2008. Integration of local ecological knowledge and conventional science: a study of seven community-based forestry organizations in the USA. Ecology and Society 13(2): 37. [online] URL: http://www.ecologyandsociety.org/vol13/iss2/art37/

Becker, C. D. 1999. Protecting a garúa forest in Ecuador: the role of institutions and ecosystem valuation. Ambio 28(2):156-161.

Becker, C. D., A. Agreda, E. Astudillo, M. Costantino, and P. Torres. 2005. Community-based monitoring of fog capture and biodiversity at Loma Alta, Ecuador enhance social capital and institutional cooperation. Biodiversity and Conservation 14(11):2695-2707. http://dx.doi.org/10.1007/s10531-005-8402-1

Bell, S., M. Marzano, J. Cent, H. Kobierska, D. Podjed, D. Vandzinskaite, H. Reinert, A. Armaitiene, M. Grodzińska-Jurczak, and R. Muršič. 2008. What counts? Volunteers and their organisations in the recording and monitoring of biodiversity. Biodiversity and Conservation 17:3443-3454. http://dx.doi.org/10.1007/s10531-008-9357-9

Bhattacharjee, Y. 2005. Ornithology. Citizen scientists supplement work of Cornell researchers. Science 308(5727):1402-1403. http://dx.doi.org/10.1126/science.308.5727.1402

Boylen, C. W., E. A. Howe, J. S. Bartkowski, and L. W. Eichler. 2004. Augmentation of a long-term monitoring program for Lake George, NY by citizen volunteers. Lake and Reservoir Management 20(2):121-129. http://dx.doi.org/10.1080/07438140409354356

Brewer, C. 2006. Translating data into meaning: education in conservation biology. Conservation Biology 20(3):689-691. http://dx.doi.org/10.1111/j.1523-1739.2006.00467.x

Brossard, D., B. Lewenstein, and R. Bonney. 2005. Scientific knowledge and attitude change: the impact of a citizen science project. International Journal of Science Education 27(9):1099-1121. http://dx.doi.org/10.1080/09500690500069483

Brown, W. T., M. E. Kransy, and N. Schoch. 2001. Volunteer monitoring of nonindigenous invasive plant species in the Adirondack Park, New York, USA. Natural Areas Journal 21(2):189-196.

Coleman, J. S. 1988. Social capital in the creation of human capital. American Journal of Sociology 94(supplement):S95-S120. http://dx.doi.org/10.1086/228943

Conrad, C. C., and K. G. Hilchey. 2011. A review of citizen science and community-based environmental monitoring: issues and opportunities. Environmental Monitoring and Assessment 176:273-291. http://dx.doi.org/10.1007/s10661-010-1582-5

Cornwell, M. L., and L. M. Campbell. 2012. Co-producing conservation and knowledge: citizen-based sea turtle monitoring in North Carolina, USA. Social Studies of Science 42(1):101-120. http://dx.doi.org/10.1177/0306312711430440

Cosquer, A., R. Raymond, and A.-C. Prevot-Julliard. 2012. Observations of everyday biodiversity: a new perspective for conservation? Ecology and Society 17(4): 2. http://dx.doi.org/10.5751/ES-04955-170402

Craig, B., G. Whitelaw, J. Robinson, and P. Jongerden. 2003. Community-based ecosystem monitoring: a tool for developing and promoting ecosystem-based management and decision making in the Long Point World Biosphere Reserve. Pages 11-16 in Proceedings of the Fifth International Conference on Science and Management of Protected Areas Association Conference (Victoria, British Columbia, Canada, May 2003). Science and Management of Protected Areas Association, Wolfville, Nova Scotia, Canada.

Crall, A. W., G. J. Newman, C. S. Jarnevich, T. J. Stohlgren, D. M. Waller, and J. Graham. 2010. Improving and integrating data on invasive species collected by citizen scientists. Biological Invasions 12(10):3419-3428. http://dx.doi.org/10.1007/s10530-010-9740-9

Danielsen, F., N. D. Burgess, and A. Balmford. 2005a. Monitoring matters: examining the potential of locally-based approaches. Biodiversity and Conservation 14:2507-2542. http://dx.doi.org/10.1007/s10531-005-8375-0

Danielsen, F., A. E. Jensen, P. A. Alviola, D. S. Balete, M. Mendoza, A. Tagtag, C. Custodio, and M. Enghoff. 2005b. Does monitoring matter? A quantitative assessment of management decisions from locally-based monitoring of protected areas. Biodiversity and Conservation 14:2633-2652. http://dx.doi.org/10.1007/s10531-005-8392-z

Da Silva Pinho, O. 2000. Community involvement in projects to reduce nonpoint source pollution. Journal of Shellfish Research 19(1):445-447.

Dickinson, J. L., J. Shirk, D. Bonter, R. Bonney, R. L. Crain, J. Martin, T. Phillips, and K. Purcell. 2012. The current state of citizen science as a tool for ecological research and public engagement. Frontiers in Ecology and the Environment 10(6):291-297. http://dx.doi.org/10.1890/110236

Ecological Monitoring and Assessment Network and the Canadian Nature Federation. 2003. Improving local decision-making through community-based monitoring: toward a Canadian community monitoring network. Environment Canada Inquiry Centre, Ottawa, Ontario, Canada.

Ellis, R., and C. Waterton. 2004. Environmental citizenship in the making: the participation of volunteer naturalists in UK biological recording and biodiversity policy. Science and Public Policy 31(2):95-105.

Ely, E. 1994. Putting data to use. Volunteer Monitor 6(1):6-10.

Ely, E., and E. Hamingson. 1998. National directory of volunteer environmental monitoring programs. Fifth edition. U.S. Environmental Protection Agency, Washington, D.C., USA.

Evans, C., E. Abrams, R. Reitsma, K. Roux, L. Salmonsen, and P. P. Marra. 2005. The Neighborhood Nestwatch program: participant outcomes of a citizen-science ecological research project. Conservation Biology 19(3):589-594. http://dx.doi.org/10.1111/j.1523-1739.2005.00s01.x

Evans, S. M., A. C. Birchenough, and H. Fletcher. 2000. The value and validity of community-based research: TBT contamination of the North Sea. Marine Pollution Bulletin 40(3):220-225. http://dx.doi.org/10.1016/s0025-326x(99)00228-3

Fernandez-Gimenez, M. E., H. L. Ballard, and V. E. Sturtevant. 2008. Adaptive management and social learning in collaborative and community-based monitoring: a study of five community-based forestry organizations in the western USA. Ecology and Society 13(2): 4. [online] URL: http://www.ecologyandsociety.org/vol13/iss2/art4/

Fischer, C. 1996. Monitoring the monitors: bird banders track data quality. Volunteer Monitor 8(2):9-11.

Franco, A., N. Malhotra, and G. Simonovits. 2014. Publication bias in the social sciences: unlocking the file drawer. Science 345(6203):1502-1505. http://dx.doi.org/10.1126/science.1255484

Gallo, T., and D. Waitt. 2011. Creating a successful citizen science model to detect and report invasive species. BioScience 61(6):459-465. http://dx.doi.org/10.1525/bio.2011.61.6.8

García, C. E. R., and S. Brown. 2009. Assessing water use and quality through youth participatory research in a rural Andean watershed. Journal of Environmental Management 90(10):3040-3047. http://dx.doi.org/10.1016/j.jenvman.2009.04.014

Gooch, M. 2004. Volunteering in catchment management groups: empowering the volunteer. Australian Geographer 35(2):193-208. http://dx.doi.org/10.1080/0004918042000249502

Gooch, M. 2005. Voices of the volunteers: an exploration of the experiences of catchment volunteers in coastal Queensland, Australia. Local Environment 10(1):5-19. http://dx.doi.org/10.1080/1354983042000309289

Greenwood, J. J. D. 2003. The monitoring of British breeding birds: a success story for conservation science? Science of the Total Environment 310(1-3):221-230. http://dx.doi.org/10.1016/S0048-9697(02)00642-3

Greenwood, J. J. D. 2007. Citizens, science and bird conservation. Journal of Ornithology 148(1, supplement):77-124. http://dx.doi.org/10.1007/s10336-007-0239-9

Heller, N. E., and E. S. Zavaleta. 2009. Biodiversity management in the face of climate change: a review of 22 years of recommendations. Biological Conservation 142(1):14-32. http://dx.doi.org/10.1016/j.biocon.2008.10.006

Jones, F. C., D. Baird, M. Bowman, G. Cameron, B. Craig, B. Cutler, J. Diamond, N. Dmytrow, M. Nocil, J. Parker, T. Pascoe, H. Vaughan, and G. Whitelaw. 2006. Performance of Ontario’s Benthos Biomonitoring Network: impacts on participants’ social capital, environmental action, and problem-solving ability. Environments 34(1):37-53.

Jordan, R. C., H. L. Ballard, and T. B. Phillips. 2012. Key issues and new approaches for evaluating citizen-science learning outcomes. Frontiers in Ecology and the Environment 10(6):307-309. http://dx.doi.org/10.1890/110280

Jordan, R. C., S. A. Gray, D. V. Howe, W. R. Brooks, and J. G. Ehrenfeld. 2011. Knowledge gain and behavioral change in citizen-science programs. Conservation Biology 25(6):1148-1154. http://dx.doi.org/10.1111/j.1523-1739.2011.01745.x

Karney, R. C. 2000. Poor water quality? Not in my backyard! The effectiveness of neighborhood pond associations in the protection and improvement of shellfish growing water on Martha’s Vineyard. Journal of Shellfish Research 19(1):465-466.

Kingdon, J. W. 1993. How do issues get on public policy agendas? Pages 40-50 in W. J. Wilson, editor. Sociology and the Public Agenda. Sage, Newbury Park, California, USA. http://dx.doi.org/10.4135/9781483325484.n3

Koss, R. S., and J. Y. Kingsley. 2010. Volunteer health and emotional wellbeing in marine protected areas. Ocean & Coastal Management 53:447-453. http://dx.doi.org/10.1016/j.ocecoaman.2010.06.002

Kountoupes, D. L., and K. S. Oberhauser. 2008. Citizen science and youth audiences: educational outcomes of the Monarch Larva Monitoring Project. Journal of Community Engagement and Scholarship 1(1):10-20.

Krishna, A. 2002. Active social capital. Columbia University Press, New York, New York, USA.

Lawrence, A. 2009. The first cuckoo in winter: phenology, recording, credibility and meaning in Britain. Global Environmental Change 19(2):173-179. http://dx.doi.org/10.1016/j.gloenvcha.2009.01.006

Lee, V. 1994. Volunteer monitoring: a brief history. Volunteer Monitor 6(1):14-16.

McKay, L. 2007. Engaging the public in bird conservation: moving from science to behavior change. Waterbirds 30:177-182. http://dx.doi.org/10.1675/1524-4695(2007)030[0177:etpibc]2.0.co;2

McKinley, D. C., R. D. Briggs, and A. M. Bartuska. 2012. When peer-reviewed publications are not enough! Delivering science for natural resource management. Forest Policy and Economics 21:1-11. http://dx.doi.org/10.1016/j.forpol.2012.03.007

Measham, T. G., and G. B. Barnett. 2008. Environmental volunteering: motivations, modes and outcomes. Australian Geographer 39(4):537-552. http://dx.doi.org/10.1080/00049180802419237

Moyer-Horner, L., M. M. Smith, and J. Belt. 2012. Citizen science and observer variability during American pika surveys. Journal of Wildlife Management 76(7):1472-1479. http://dx.doi.org/10.1002/jwmg.373

Newman, C., C. D. Buesching, and D. W. Macdonald. 2003. Validating mammal monitoring methods and assessing the performance of volunteers in wildlife conservation—“Sed quis custodiet ipsos custodies?” Biological Conservation 113(2):189-197. http://dx.doi.org/10.1016/s0006-3207(02)00374-9

Obura, D. O., S. Wells, J. Church, and C. Horrill. 2002. Monitoring of fish and fish catches by local fishermen in Kenya and Tanzania. Marine and Freshwater Research 53:215-222. http://dx.doi.org/10.1071/MF01151

O’Rourke, D., and G. P. Macey. 2003. Community environmental policing: assessing new strategies of public participation in environmental regulation. Journal of Policy Analysis and Management 22(3):383-414. http://dx.doi.org/10.1002/pam.10138

Ottinger, G. 2010. Buckets of resistance: standards and the effectiveness of citizen science. Science, Technology & Human Values 35(2):244-270. http://dx.doi.org/10.1177/0162243909337121

Overdevest, C., C. Huyck Orr, and K. Stepenuck. 2004. Volunteer stream monitoring and local participation in natural resource issues. Human Ecology Review 11(2):177-185.

Overdevest, C., and B. Mayer. 2007. Harnessing the power of information through community monitoring: insights from social science. Texas Law Review 86:1493-1526.

Pattengill-Semmens, C. V., and B. X. Semmens. 2003. Conservation and management applications of the REEF volunteer fish monitoring program. Environmental Monitoring and Assessment 81(1-3):43-50.

Pollock, R. M., and G. S. Whitelaw. 2005. Community-based monitoring in support of local sustainability. Local Environment 10(3):211-228. http://dx.doi.org/10.1080/13549839.2005.9684248

Rydin, Y., and M. Pennington. 2000. Public participation and local environmental planning: the collective action problem and the potential of social capital. Local Environment 5(2):153-169. http://dx.doi.org/10.1080/13549830050009328

Savan, B., A. J. Morgan, and C. Gore. 2003. Volunteer environmental monitoring and the role of the universities: the case of Citizens’ Environment Watch. Environmental Management 31(5):561-568. http://dx.doi.org/10.1007/s00267-002-2897-y

Schwartz, M. W. 2006. How conservation scientists can help develop social capital for biodiversity. Conservation Biology 20(5):1550-1552. http://dx.doi.org/10.1111/j.1523-1739.2006.00421.x

Sharpe, T., B. Savan, and N. Amott. 2000. Testing the waters. Alternatives 26(4):30-33.

Shirk, J. L., H. L. Ballard, C. C. Wilderman, T. Phillips, A.Wiggins, R. Jordan, E. McCallie, M. Minarchek, B. V. Lewenstein, M. E. Krasney, and R. Bonney. 2012. Public participation in scientific research: a framework for deliberate design. Ecology and Society 17(2): 29. http://dx.doi.org/10.5751/es-04705-170229

Shirose, L. J., C. A. Bishop, D. M. Green, C. J. MacDonald, R. J. Brooks, and N. J. Helferty. 1997. Validation tests of an amphibian call count survey technique in Ontario, Canada. Herpetologica 53(3):312-320.

Stepenuck, K. F. 2013. Improving understanding of outcomes and credibility of volunteer environmental monitoring programs. Dissertation. University of Wisconsin–Madison, Madison, Wisconsin, USA.

Stern, E. 1997. Crisis and learning: a conceptual balance sheet. Journal of Contingencies and Crisis Management 5(2):69-86. http://dx.doi.org/10.1111/1468-5973.00039

Ticheler, H. J., J. Kolding, and B. Chanda. 1998. Participation of local fishermen in scientific fisheries data collection: a case study from the Bangweulu Swamps, Zambia. Fisheries Management and Ecology 5:81-92. http://dx.doi.org/10.1046/j.1365-2400.1998.00076.x

Trumbull, D. J., R. Bonney, D. Bascom, and A. Cabral. 2000. Thinking scientifically during participation in a citizen-science project. Science Education 84(2):265-275. http://dx.doi.org/10.1002/(SICI)1098-237X(200003)84:2%3C265::AID-SCE7%3E3.3.CO;2-X

U.S. Office of Management and Budget. 2012. Fiscal year 2013: cuts, consolidations, and savings. Budget of the U.S. government. U.S. Office of Management and Budget, Washington, D.C., USA.

Van Rijsoort, J., and Z. Jinfeng. 2005. Participatory resource monitoring as a means for promoting social change in Yunnan, China. Biodiversity and Conservation 14(11):2543-2573. http://dx.doi.org/10.1007/s10531-005-8377-y

Vaughan, H., G. Whitelaw, B. Craig, and C. Stewart. 2003. Linking ecological science to decision-making: delivering environmental monitoring information as societal feedback. Environmental Monitoring and Assessment 88:399-408. http://dx.doi.org/10.1023/A:1025593728986

Viswanathan, M., A. Ammerman, E. Eng, G. Gartlehner, K. N. Lohr, D. Griffith, S. Rhodes, C. Samuel-Hodge, S. Maty, L. Lux, L. Webb, S. F. Sutton, T. Swinson, A. Jackman, and L. Whitener. 2004. Community-based participatory research: assessing the evidence. Agency for Healthcare Research and Quality, Rockville, Maryland, USA.