INTRODUCTION

In various Western countries, large multiactor programs have been or are being carried out in which joint knowledge production (JKP) for climate adaptation takes place between scientists, policy makers, and other societal actors (Boon and Horlings 2013). JKP implies that actors cooperate directly in the exchange, production, and application of knowledge (Edelenbos et al. 2011, Hegger et al. 2012). An inventory identified 14 such temporary programs in various Organisation for Economic Co-operation and Development countries, each of them emphasizing the importance of systemic innovation, involvement of societal actors in knowledge production, and the realization of both scientific and nonscientific output (Boon and Horlings 2013). Large climate research programs such as the Dutch programs Climate Changes Spatial Planning, Knowledge for Climate and Living with Water, or the German Klimzug Program are cases in point (Veraart et al. 2014). JKP is also one of the spearheads of Future Earth, a recently started 10-year international initiative on global sustainability research (Mauser et al. 2013, Groot et al. 2014) and is advocated in several position papers (e.g., Driessen et al. 2013).

JKP (Edelenbos et al. 2011, Hegger et al. 2012) can be positioned in broader debates on how to arrive at socially more relevant knowledge (Funtowicz and Ravetz 1993, Gibbons et al. 1994, Scholz et al. 2000). Recent empirical analyses of JKP have started to zoom in on the quality of the knowledge production processes in projects in which scientists, public policy makers, and sometimes other societal actors collaborate in “climate proofing” specific regions (Hegger et al. 2012, Edelenbos et al. 2011). Other recent studies have dealt with project structures and project management in transdisciplinary research teams (Boon et al. 2014), knowledge management in the context of ecosystem-based management (Giebels et al. 2015), interactive knowledge development in coastal projects (Seijger et al. 2014), JKP in sustainability partnerships (Offermans and Glasbergen 2015), and knowledge gaps regarding the desirable link between science-policy interfaces and problem types (Van Enst et al. 2014).

There is some empirical evidence that JKP may contribute to the societal relevance of research (Walter et al. 2007, Edelenbos et al. 2011, Lang et al. 2012, Hegger et al. 2012, 2014, Hegger and Dieperink 2014). Knowledge on the scientific merit of JKP has, however, a much weaker empirical basis. Literature suggests that close cooperation between scientists and policy makers can be beneficial for the scientific enterprise (Scholz et al. 2000, Wickson et al. 2006, Regeer and Bunders 2009, WIMEK 2014, Veraart et al. 2014) but also warns against a too one-sided focus on science for rather than science of adaptation (Lövbrand 2011, Swart et al. 2014). However, more detailed and systematic empirical insights into the scientific merit of JKP projects are still lacking. This is a problem, first, because so far it remains unclear what the observed trend toward an emphasis on participatory forms of knowledge production means for the credibility, salience, and legitimacy of the science related to climate adaptation (Cash et al. 2003). Second, we lack knowledge of the extent to which scientists’ interests are being satisfied in JKP projects, which forms an important condition for their sustained engagement in such projects (Armitage et al. 2011).

We aim to address the observed knowledge gap by providing an empirical exploration of what the scientific merit of JKP projects may be. We do so by, first, developing a set of hypotheses regarding this scientific merit in terms of the process, output, and impact of JKP projects. The set of hypotheses has been based on a review of literature from the fields of science studies, the sociology of knowledge, and environmental governance, as well as discussions with six environmental science research leaders. We subsequently compiled a short survey that was distributed among 594 researchers whose contact data were included in a database made available by the Dutch multiactor research program Knowledge for Climate. We used the 144 responses (response rate of 25%) we received to validate our hypotheses.

METHODS

To develop a set of hypotheses regarding the scientific merit of JKP, we carried out a review of literature from the fields of science studies, the sociology of knowledge, and environmental governance. We furthermore interviewed six environmental science research leaders to validate and complement the findings from the literature. We inductively clustered the aggregated findings from the literature review into “process, output, and impact.” This has led to a list of 21 hypotheses.

Next, we compiled a survey that included questions that were used to operationalize the hypotheses. The questions mainly contained categorical variables, where respondents could check a box if they deemed something to be the case, e.g., that their project has led to more follow-up projects. We also included some background questions, e.g., age, years of experience as researcher, and disciplinary background, to be able to determine the external validity of the survey as well as factual questions on the process of doing JKP, e.g., frequency of meetings or issues upon which collaboration has taken place, to be able to characterize the process of JKP. The survey questions were tested in advance by discussing them with five Dutch speaking researchers from the Dutch climate research community. Based on their oral and written feedback, the survey was finalized and put online using Surveymonkey.

Upon request, the Knowledge for Climate program supplied us with a data file with contact details of more than 1000 different persons involved in the Knowledge for Climate Network, including researchers, policy makers, members of the steering group, societal stakeholders, and administrative assistants among others. From this file we selected 720 persons whom we thought were researchers because their affiliation was a university or knowledge institute. In January 2015, a link to the survey was e-mailed to them three times. We asked the recipients to reply if they did not consider themselves part of the target group (people who did so were not sent reminders). In total, 89 persons were not reached, while another 39 persons replied that they were not part of the target group. Another 20 persons e-mailed us that their actual experience with JKP would be too modest. Out of the remaining 570 persons, 144 completed the survey, implying a response rate of 25%, which we consider to be relatively high. Table 1 lists the key characteristics of the survey population. The table shows that our respondents report different frequencies of collaboration with nonscientists, and differ in experience as researchers as well in disciplinary backgrounds. Because our key informants suggested that these variables may be relevant we have checked their relevance using X-square tests for independence or Fischer’s exact test. We compared respondents who collaborated on a daily or weekly basis with the group that collaborated less than once a month. We also compared more experienced with less experienced respondents as well as natural scientists (n = 74) with social scientists (n = 30). Fischer’s exact test was used instead of X-square test for independence. The latter requires that the sample is representative of the researched population and that the used variables are categorical variables and the expected count in at least 80% of the cells in a crosstab is at least 5. For those cases in which the last assumption was not met, we used Fischer’s exact test as prescribed in Pallant (2007).

THE SCIENTIFIC MERIT OF JKP: A LITERATURE REVIEW

Both within the philosophy of science and within the scientific enterprise itself, scientific rigor is generally assumed to form the core of what distinguishes science from nonscience. This holds both for more objectivistic and more hermeneutic forms of science (e.g., Giddens 1976). Science is generally distinguished from nonscience by referring to it as being explicit, controllable, and controlled, e.g., via peer review procedures, and based on evidence that is systematically collected and analyzed (Raymond et al. 2010, Enengel et al. 2012). It is therefore to be seen as “often explicit knowledge that has been derived from applying more formal methods that aim to increase rigor in relation to different positions on validity and reliability” (Raymond et al. 2010:1769). A commonly held understanding of “good science” is that it should be original as well as scientifically and socially relevant. It is argued that rigor is achieved by being self-critical in a Popperian (1959) sense (good scientific theories are ones that are falsifiable) as well as by applying Merton’s (1973) CUDO norms, i.e., communalism, disinterestedness, and organized skepticism, that imply, among other things, that scientific work should follow a clear line of argumentation, depart from a clear and well-supported problem definition, that the methods chosen to do empirical research fit this problem definition, and that the results and conclusions of the research are independent from researchers’ personal preferences and interests. Nevertheless, although Popper’s principle of falsification as well as Merton’s CUDO norms are endorsed by many scientists, we do not want to suggest that there would be one single kind of science or a single type of universal scientific epistemology (a positivist dominant one in natural sciences).

We admit that there are large differences between scientists and scientific disciplines, including, inter alia, the following: (i) the extent to which a natural science(-based) positivist epistemology should be taken as a starting point (e.g., Giddens 1976); (ii) the extent to which there would exist such a thing as a uniform and clearly identifiable knowledge base to which scientists could add new knowledge, as opposed to what some authors would call “more open knowledge systems” (Tàbara 2013, Tàbara and Chabay 2013); (iii) the research methods and approaches that are seen as adequate for connecting theory and empirics, e.g., deduction, induction, abduction. Moreover, science can be seen as a result of a series of institutional processes and commitments, social interactions, as well as cultural contexts and worldviews (Gieryn 1983).

In our opinion, the position that science can be distinguished from non-science by it being explicit, controllable, and controlled, e.g., via peer review procedures, and based on evidence that is systematically collected and analyzed holds, even if one considers two prominent strands of literature that nuance it. First, literature from the field of sustainability science argues that a science that focuses on sustainability problems differs from other forms of science because of its inherent focus on issues that are seen as problematic by societal actors, and about which different perspectives in society exist (Kemp and Martens 2007). However, this is not an a priori reason to make such forms of science, including the science within JKP projects, subject to different or lower quality criteria. Second, a critical branch of science studies argues that boundaries between science and nonscience are not fixed but continuously negotiated, a process that can be labeled boundary work (Gieryn 1983, Hoppe 2009). Negotiating boundaries between domains, including those of science and nonscience is said to take place on a day-to-day basis (Gieryn 1983, Hoppe 2009). To some degree, boundary work can be seen as a power play in which actors have certain interests in portraying specific contributions as science or not, and in which they portray the relevance of certain forms of science as self-evident or on the other hand question it. We argue that in boundary work generally it is not the Mertonian norms as such that are questioned, but rather the domains to which they are deemed applicable.

As explained, based on our literature review we inductively identified three approaches to assess the quality of science in JKP projects: by looking at its process, product, and impact. To assess the quality of the process, it will be necessary to try to determine to what extent scientific rigor as set out above has actually been achieved. This process is rarely addressed in scientific literature. Regarding its product, publications are currently seen as an important performance indicator (Hessels and Van Lente 2008). In most disciplines, publishing in peer-reviewed journals or books indicates that research is seen as good science by peers. A dominant focus on publications and citations is, however, also criticized (Mauser et al. 2013). Reward structures for researchers are being debated and the research landscape is by some said to be changing and in need of further change to make the scientific enterprise more relevant to society (Bridle et al. 2013, Mauser et al. 2013). The impact of JKP projects on science can be determined in various ways, including an assessment of citations, the extent to which researchers manage to acquire new research projects, as well as their reputation, both within research and practice, and the scientific and societal networks in which they participate. We assume that the merits and pitfalls of JKP will be more manifest in cases in which there is more frequent interaction of scientists with policy makers or practitioners. Table 2 provides an overview of the potential scientific merit and pitfalls in JKP projects derived from literature.

Changes in producing knowledge

Literature mentions that JKP projects may lead to more scientific debate on problem analysis and methods, enriching the scientific research agenda ([H1]; Knowledge for Climate, interview #1,5,6). This is said to be due to the fact that the research agenda is produced in interaction with societal actors, leading to refinement and a combination of both fundamental research that is translated in applicable knowledge and the actual application of this knowledge in experiments and pilot projects. Such an approach was followed in the process of determining the research themes of the Dutch "Knowledge for Climate" research program (Bridle et al. 2013 and Mauser et al. 2013).

JKP is also said to lead to a broader empirical knowledge base because researchers have more access to the empirics including network contacts ([H2]; Enengel et al. 2012, Hegger et al. 2014). In so doing, JKP can help to unearth contextual, phenomenological, and/or tacit knowledge and include it in the scientific discourse (Wickson et al. 2006, Regeer and Bunders 2009, Enengel et al. 2012, Hegger et al. 2014) and hence lead to more grounded scholarship. JKP is also said to apply a dynamic, responsive methodology and to be more action-oriented, contributing to new research practices complementing mode 1 research ([H3]; Wickson et al. 2006, Mattor et al. 2014).

According to Wickson et al. (2006:1055) the confrontation of different types of knowledge (of different scientific disciplines, but also of local, tacit, or lay knowledge) leads to paradoxes: “the problem of the paradox and the conceptual creativity it requires might encourage TD [Transdisciplinary] researchers to employ both logic and intuition in their research approaches” [H4]. Working at the science-policy interface in the case of wicked global change and sustainability problems is also said to oblige researchers to reflect on how their own frames of reference, values, beliefs, assumptions, etc. have shaped the conceptualization of the problem as well as the development of the method of investigation and the solution (Wickson et al. 2006, Lövbrand 2011, Cornell et al. 2013, Wesselink et al. 2013; interview #5). One can logically assume that a transdisciplinary context as can be found in JKP projects would be conducive to such reflexivity [H5].

Literature has also documented some potential pitfalls of JKP projects. One of them is the irreconcilability of JKP projects with current reward structures for researchers, especially doctoral students ([H6]; Regeer and Bunders 2007, Hessels and van Lente 2008, Hegger et al. 2012, Cornell et al. 2013, Groot et al. 2014). Enengel et al. (2012) found that combining transdisciplinary projects with a PhD trajectory can be challenging. The real-world focus of transdisciplinary projects creates new dependencies for the researcher, e.g., dependencies on the course of affairs in policy processes, which can be difficult to reconcile with current practice in which PhD research should often be finalized within a fixed term. It has been shown that researchers’ experiences with combining PhD research and JKP projects have varied (Fry et al. 2006, Hegger and Dieperink 2014). There are also indications that some researchers are better equipped to translate socially relevant work to scientific output than others (interview #1). Another potential pitfall of JKP projects is that some scientists may not be genuinely interested in the societal goals of a JKP project and employ strategic behavior (Edelenbos et al. 2011). In some cases researchers pay lip service to the wishes of their funding institutions but at the same time devote most energy to reaching their publication goals and putting forward their own research agenda (Hessels and Van Lente 2008; [H7]). In the Dutch context, researchers have been shown to be driven to a large extent by the things they are rewarded for: the number of scientific publications they can produce and the ranking of the journals in which they publish (De Goede and Hessels 2014). This may invite researchers to engage in JKP projects strategically.

Scientific products

The impact of JKP on the quantity of the products of science produced in JKP projects is still unclear. JKP may facilitate the writing of scientific publications, including journal articles, reports, and peer-reviewed books because of enhanced access to empirical data (interview #1; [H8]). Alternatively, JKP may complicate the writing of especially journal articles and hence lead to fewer scientific publications because researchers have to devote their time to other activities, e.g., discussions with policy makers, the production of nonscientific commissioned reports, etc. [H12].

JKP projects may also lead to publications with a different substantive content from publications related to non-JKP projects. This difference may be due to the fact that the research on which it is based has taken place closer to the context of application. It is as yet unclear if the latter (different substantive content) is to be seen as a merit or a pitfall [H9].

A third potential merit regarding the products produced through science in the context of JKP projects may be more publications and invited presentations for practitioners or the broader public. JKP projects are carried out in close cooperation with policy makers and practitioners, leading to increased interaction and hence more possibilities of seeing opportunities to write something for these target groups and more chance of being invited to do so by them (interview #1; [H10]).

Finally, JKP projects may lead to the establishment of communities of research and practice (CORP). This can be both a product and an impact of JKP projects (Iyalomhe et al. 2013). The distinction lies in the fact that a JKP project can be carried out with the explicit aim of establishing or contributing to the continuation of such a CORP, so it is explicitly denominated as a product of a JKP project, while it can also be a medium and (intended or unintended) outcome of JKP projects [H11].

Scientific impacts

It is still unclear whether JKP projects can be expected to lead to publications that are more or that are less frequently cited, although some arguments for the former are raised (WIMEK 2014; [H13, 19]). Another unclear issue is whether JKP would enhance researchers’ potential to be critical of the social order [H14] or if it would decrease this potential [H20]. One could assume that JKP projects could enhance the potential of scientists to be critical of the social order because scientists who do their work closer to society may have more detailed knowledge of this social order, the flaws in it, and the reasons for them being there (interview #3). On the other hand, according to Lövbrand (2011:231): “... there may be a trade-off between research co-produced to be accountable to the knowledge needs of societal decision-makers, and co-produced research that seeks to challenge and transform existing ways of thinking.”

JKP may lead to the identification of research questions and opportunities for follow-up research as well as to an enhanced reputation of researchers both in circles of research and policy/practice (interview #1 and #5; WIMEK 2014; [H15]). One can logically expect this to lead to more newly commissioned research. The substantive content of follow-up projects of JKP projects may be different. It is as yet unclear to what extent this would be a merit or a pitfall [H16].

JKP may lead both to increased or decreased career opportunities for researchers, inside and outside academia [H17, 21]. It is difficult to predict what young researchers’ opportunities will be in terms of conventional academic careers because (i) we lack knowledge on the extent to which researchers in JKP projects manage to fit into conventional reward structures and (ii) we cannot predict to what extent academic reward structures will change in the future to better serve the interests of researchers with a JKP background. At the same time, JKP researchers’ closer engagement with policy makers and practitioners can be expected to increase their career chances outside academia (interviewee #5).

Finally, JKP may lead to motivation and gratification derived from interaction with society (Enengel et al. 2012, WIMEK 2014; [H18]). According to Enengel et al. (2012:11), “Particularly rewarding are the motivation and gratification provided by early feedback from potential users, the social competences gained, and knowledge accumulated by interacting with different groups inside and outside academia and the feeling of producing results that are also relevant outside tight disciplinary or academic boundaries.” One can logically assume that feelings of reward lead to better motivated researchers and that this, in turn, leads to higher quality research.

CHARACTERISTICS OF JKP

We provide a general characterization of JKP as reported by our survey respondents. These findings do not relate directly to specific hypotheses but must be seen as a specification of the practice of JKP. Table 3 shows which research steps were discussed in meetings in which the respondents collaborated.

As Table 3 shows, the research steps upon which most collaboration took place are the results and conclusions (63.2%); translation of results and conclusions to policy and practice (52.8%); and the research questions (50%). Fewer researchers collaborated on other issues. We consistently found that frequent collaborators collaborated more on each particular research step as compared with the infrequent collaborators. Natural scientists collaborated significantly more on results and conclusions, translation of results, and conclusions to policy and practice, research questions, progress reports, and juridical/financial issues, while social scientists collaborated more on research methods and policy recommendations.

As Table 4 shows, making available reports and papers on the research results was the most important activity for collaboration (57.6%). Other important activities were the participation in (53.5%) and the organization of workshops (47.2%). Unsurprisingly, the frequent collaborators engaged significantly more in all activities compared with the infrequent collaborators. For most activities we found that natural scientists engaged more in them than social scientists, the only exceptions being the organization of workshops and participation in working visits.

SCIENTIFIC MERITS AND PITFALLS OF JKP: EMPIRICAL CONFRONTATION OF THE HYPOTHESES

Changes in producing knowledge

We found confirmation for the first part of H1 (more societal debate on problem analysis and methods) but only a weak indication that this has enriched the scientific research agenda. Table 5 shows that 38.9% of the respondents indicated that debate with societal actors on the problem analysis had taken place because of the collaboration with policy and practice. A significantly higher proportion of the frequent collaborators check-boxed the item compared with the infrequent collaborators. Similar results were found for the extent to which collaboration on research methods has taken place (Table 3). The researchers were also asked the open question of how they experienced the collaboration (predominantly positive or negative) and why (not in the table). One respondent remarked that the collaboration led to sharper research questions.

Our data confirmed the hypothesis that JKP may lead to a broader empirical knowledge base (H2). Of all respondents, 47.2% indicated that collaboration with policy and practice has led to access to relevant contacts; 28.5% reported access to reports and data; and 25.7% reported advice regarding the selection of empirical cases. We found higher proportions for the frequent collaborators than for the infrequent collaborators.

We found only limited confirmation for the hypothesis that JKP may lead to new research practices complementing “Mode 1” research (H3). Of the respondents, 16.7% indicated that because of the collaboration with policy and practice they had used different research methods from those they would otherwise have used, which is low compared with other items in the table. On the other hand, similar to the previous hypotheses we found higher proportions for the frequent collaborators as opposed to the infrequent collaborators.

As a fourth hypothesis we suggested that JKP may lead to the promotion of creativity and intuition on the part of researchers through the confrontation of different types of knowledge and associated paradoxes. We only found confirmation for the first part of this hypothesis (the promotion of creativity) because we were unable to assess to what extent the responses received were attributable to “confrontation of different types of knowledge” or to “paradoxes.” Of our respondents, 33.3% indicated that their creativity was stimulated through cooperation with policy and practice; 22.2% indicated that the collaboration enabled them to do the things they were passionate about. These scores are not very high compared with other items, but again we found significantly higher scores for the frequent collaborators than for the infrequent collaborators. Also, in response to the open question of how they experienced the collaboration (predominantly positive or negative) several of the positive respondents wrote down that they saw the collaboration as stimulating and inspiring, especially because it enabled them to see what their results were used for.

Our data confirmed the hypothesis that JKP may lead to more reflexivity on the part of researchers, stakeholders, and knowledge users (H5). Of all respondents, 41.7% indicated that the collaboration with policy and practice has helped them to place their own research in a broader perspective. This item ranked the second highest of all the process characteristics assessed. Again, the frequent collaborators scored higher than the infrequent collaborators.

We only found limited confirmation for the hypothesis that JKP may lead to irreconcilability with current reward structures for researchers, especially doctoral students (H6). Of all respondents, 15.3% indicated that the collaboration with policy and practice had reduced their time available for scientific work, the lowest score in Table 5. Moreover, it was the only item without a significant difference between the frequent and the infrequent collaborators. On the other hand, in response to the open question of how they experienced the collaboration (predominantly positive or negative) and why, one respondent remarked that the collaboration led to insufficient time for the more fundamental or theoretical part of the work and to too much overlap with the work of consultancies. We did not find indications that irreconcilability, if any, with current reward structures would be more serious for doctoral students.

That JKP may lead to strategic behavior of scientists who are not genuinely interested in the societal goals of a JKP project (H7) is only slightly confirmed by our data, while its relevance across the surveyed sample seems to be minor. Compared with other items, relatively few respondents indicated that collaboration with policy and practice enabled them to do the things they themselves found important (23.6%), with only three items ranked lower. A skeptical interpretation could be that these results are a sign of a lack of genuine interest. On the other hand, this score is again significantly higher for the frequent collaborators as opposed to the infrequent collaborators. Moreover, in response to the open question of how they experienced the collaboration (predominantly positive or negative) and why, only a small minority was negative (6.3%) or ambiguous (1.4%) about the collaboration. It should be mentioned, though, that one of the negative respondents explicitly mentioned the “forced” character of the collaboration.

Scientific products

JKP may lead either to more (H8) or fewer scientific publications (H12). We found that both hypotheses may hold in specific contexts. Table 6 shows that on average, the respondents reported that the projects they participated in resulted in 4.1 journal articles, 0.6 books, three scientific reports, and one book chapter (if we assume that for everyone who did not respond to the question the answer was 0, these numbers drop to 2.2, 0.2., 1.7 and 0.3, respectively). Table 6 shows the researchers’ own opinion on the influence of the collaboration on research output. Regarding all types of output, including scientific publications, there were more researchers who perceived a positive influence than those who perceived a negative influence. On the other hand, we did not find any significant differences in the quantity of scientific publications between the frequent and infrequent collaborators (not in the table). Also, some researchers (11.8%) endorsed the statement that one would have published more if one had not collaborated with policy and practice (Table 7), and significantly more of the frequent collaborators (as opposed to the infrequent collaborators) did so.

Our data confirm H9. Of all respondents, 22.2% indicated that their scientific publications would have dealt with other content-wise issues than if they had not collaborated with policy and practice (see Table 7). Compared with the other items in the table, this is relatively high (only two items scored higher). We did not find a significant difference between the frequent and the infrequent collaborators.

JKP has also resulted in more publications for policy makers/practitioners/the broader public (H10). On average, the respondents reported that the projects they participated in resulted in 2.4 professional publications and 2.1 publications for the broader public (Table 6). Regarding these types of output, there were more researchers who perceived a positive influence than those who perceived a negative influence. Also, a very low number of respondents indicated that they would have produced more nonscientific publications if they had not collaborated with policy makers and practitioners (Table 7).

H11 suggests that JKP may lead to the establishment of a Community of Research and Practice. This hypothesis has been confirmed. Of all respondents, 45.1% endorsed the statement that it would have been more difficult to expand their personal network if they had not collaborated with policy and practice; 34.0% endorsed the statement that the establishment of a new community of researchers and practitioners would have been more difficult; 14.6% endorsed the statement that it would have been more difficult to maintain an existing community of researchers and practitioners. Only for the first of these three statements did we find a significant difference between frequent and infrequent collaborators (the former endorsed the statement more). Compared with the other items in Table 7, the first two scored relatively high.

Scientific impacts

In Table 1 we specified that JKP may lead either to more (H13) or fewer cited publications (H19). We found that our survey results are inconclusive with regard to the question which of the two competing hypotheses is dominant. Of all respondents, 12.5% indicated that the project has led to “more citations.” Significantly more of the frequent collaborators check-boxed this option (Table 8). Compared with the other options in the table, this is a relatively low score. On the other hand, we can logically expect that researchers will be reluctant to relate the fact that they participated in a project to their number of citations.

We found only moderate confirmation of H14 (JKP may lead to enhanced potential to be critical of the social order [because of better knowledge about this social order]). However, our data offer insufficient reason to drop the hypothesis that JKP may lead to trade-offs between usefulness for policy and academic reflection, i.e., decreased potential to be critical of the social order (H20). Of all respondents, 35.4% reported that collaboration with policy and practice has led to a broader view in general; 28.5% reported that it has led to a broader view on the role of their own research. There was no significant difference between the frequent and infrequent collaborators (Table 8). Both items score relatively low compared with others. However, a broader view cannot be equated one to one with the potential to be critical.

H15, that JKP may lead to more follow-up research projects, was confirmed. Of all respondents, 29.2% indicated that they thought that collaboration with policy and practice has led to new projects (significantly more of the frequent collaborators did so). Even higher scores were given for items that we expect to increase the chance of starting new projects: extending the network of the researcher’s organization (45.8%); contributing to the reputation of the researcher’s organization (39.6%); and generating new research questions (38.9%), with in all cases significantly more frequent collaborators than infrequent collaborators endorsing the statements.

We found only weak support for H16 that JKP may lead to different types of follow-up research projects. We did not directly assess whether any follow-up projects had a different substantive content or whether the respondents believed this to be the case. One could logically assume this to be true in cases in which follow-up projects are a direct result of collaboration with policy and practice, i.e., in cases in which the collaboration was the reason for the follow-up projects being there.

Our results confirm neither H17 nor H21 on enhanced or decreased career opportunities for early career researchers within and/or outside academia. Our data suggest that respondents do not see enhanced career opportunities as a salient outcome of the collaboration. Of all respondents, (i) 12.5% indicated that collaboration with policy and practice has led to the successful completion of a PhD thesis of someone supervised by them, which we expect to increase career opportunities within academia; (ii) 11.1% indicated that it has led to increased career opportunities outside academia; (iii) 9.0% that it has led to increased career opportunities inside academia; and (iv) 8.3% that it has led to the completion of their own PhD. In the case of item (ii) and (iii) the frequent collaborators scored significantly higher than the infrequent collaborators. All scores are among the lowest of the items listed in Table 8. However, each of the above-mentioned items will be relevant only to part of the respondents, (i) mainly to more senior researchers; (ii), (iii), and (iv) mainly to more junior researchers.

Finally, we found moderate confirmation for H18, that JKP may lead to motivation and gratification derived from interaction with society. Of all researchers, 22.2% indicated that the collaboration with policy and practice has led to more motivation for doing research. However, we did not find any significant differences between the frequent and the infrequent collaborators. Also, 22.2% indicated that it enabled them to do things they were passionate about, with a significant difference between the frequent and the infrequent collaborators (Table 5).

We also asked the interviewees about the occurrence of other potential impacts of JKP projects. Of the respondents, 54.2% endorsed the statement that the project had led to the production of policy relevant knowledge; 38.2% indicated that the project had led to expertise that can be marketed at a later stage; 31.9% indicated that the project led to direct financial revenues; 16.7% indicated that the project enabled developing material that can be used for educational purposes; and 13.2% indicated that the project stimulated entrepreneurship. These impacts, however, are not directly related to our hypotheses.

DISCUSSION

We researched the scientific merit of JKP projects through the empirical confrontation of a set of hypotheses derived from literature. As the previous sections have shown, most of the questions used to operationalize the hypotheses were discriminatory in the sense that the scores varied between questions and, most importantly, the scores of frequent and infrequent collaborators for the same question. We see the latter as a strong sign that at least the hypotheses that were strongly confirmed are a substantial characteristic of scientific knowledge production of the projects and programs in which the surveyed researchers participated. On the other hand, the survey measured to a large extent the self-reported merits of JKP. These may be less reliable than findings that have been derived from more objective statistics, but given the goal of our research, to sketch a first picture of the scientific merit of JKP, we find it legitimate to draw largely on self-reported findings.

The internal validity of the survey, i.e., the extent to which the respondents are representative of the approached population as a whole, cannot be determined with certainty. However, it seems logical that among the respondents there is an overrepresentation of persons with (strong) opinions on the topics covered in the survey and hence an overrepresentation of persons with actual experience, at project level, with JKP. In terms of external validity, we deem it safe to say that the approached population forms the core of what may be termed “the Dutch climate research community.” We deem this to be the case, because (i) the Knowledge for Climate program has had an important networking function, bringing many researchers, policy makers, and stakeholders together, (ii) according to the KfC program office, some people were on their list not because of involvement in KfC projects but because of their past involvement in related programs (Climate Changes Spatial Planning, Living With Water).

The generally positive attitude toward JKP may partly be due to self-selection. It seems plausible that projects and programs that claim to attach much importance to JKP show an overrepresentation of researchers with a positive stance toward JKP. On the other hand, we have also seen variations and gradations in the frequency with which respondents collaborated. Some projects are JKP projects to a greater extent than others.

CONCLUSIONS AND RECOMMENDATIONS

The notion of what science is, to which uses it can be put, and how it relates or should relate to other societal domains is continuously subject to debate. Despite this we have argued that some core values of science are shared by many (maybe all) scientific disciplines. Science asks for the pursuit of a systematic approach to the development of knowledge and is collective in the sense that scientists build upon each other’s work, correct each other, and are (ideally) willing to revise any knowledge claims on the basis of new incoming information. An open question, to which this paper has intended to contribute, is what it precisely is that JKP adds to the scientific enterprise

We started with the observation that there is a theoretical and empirical lack of systematic insights into the scientific merit of JKP projects in the adaptation domain. To address this knowledge gap, we have developed a set of concepts and hypotheses, addressing the scientific process, output, and impact in JKP projects. We have confronted the framework with the empirics through a survey of 144 researchers from the Dutch Knowledge for Climate Research community, which can be considered the core of the Dutch climate research community. As we have shown, although all respondents participated in projects that are in name JKP projects, only a minority collaborated frequently (on a daily or weekly basis) with policy and practice. But if they did so, they collaborated more on different aspects of the research process than the infrequent collaborators.

Table 9 summarizes the merits and pitfalls of JKP for which we found strong empirical confirmation, showing that these pertain mostly to the scientific process and products in JKP projects. However, there is no reason to dismiss any of the other hypotheses as irrelevant, neither in the specific context of Dutch climate research nor more generally in the context of assessing the scientific merit of JKP. On the contrary, we think that more in-depth and comparative research on the precise conditions under which these hypotheses may or may not hold is in order.

In conclusion, this paper has contributed to the broader literature on participatory forms of knowledge production by providing an integrated framework for assessing the scientific merit of JKP projects. We also confirmed the relevance of including each of the developed hypotheses for follow-up research. Our findings point toward a need for a continued and nuanced assessment of this scientific merit, for instance through more comparative assessments or, on the contrary, through more in-depth case study research that helps to acquire insights into the underlying mechanisms contributing to certain merits and pitfalls of JKP. As the literature currently stands, from a scientific point of view there is still the question of what would be an appropriate balance between more and less participatory forms of scientific research. However, based on our findings, we argue that there should indeed be a balance; there is no reason to assume that one form of research would be inherently better than the other.

LITERATURE CITED

Armitage, D., F. Berkes, A. Dale, E. Kocho-Schellenberg, and E. Patton. 2011. Co-management and the co-production of knowledge: learning to adapt in Canada’s Arctic. Global Environmental Change 21:995-1004. http://dx.doi.org/10.1016/j.gloenvcha.2011.04.006

Boon, W. P. C., M. M. H. Chappin, and J. Perenboom. 2014. Balancing divergence and convergence in transdisciplinary research teams. Environmental Science & Policy 40:57-68. http://dx.doi.org/10.1016/j.envsci.2014.04.005

Boon, W. P. C., and E. Horlings. 2013. Kenniscoproductie voor de grote maatschappelijke vraagstukken. Rathenau Institute, The Hague, The Netherlands. [online] URL: http://www.rathenau.nl/uploads/tx_tferathenau/Kenniscoproductie_Rathenau.pdf

Bridle, H., A. Vrieling, M. Cardillo, Y. Araya, and L. Hinojosa. 2013. Preparing for an interdisciplinary future: a perspective from early-career researchers. Futures 53:22-32. http://dx.doi.org/10.1016/j.futures.2013.09.003

Cash, D. W., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, D. H. Guston, J. Jäger, R. M. Mitchell. 2003. Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences of the United States of America 100(14):8086-8091. http://dx.doi.org/10.1073/pnas.1231332100

Cornell, S., F. Berkhout, W. Tuinstra, J. D. Tàbara, J. Jäger, I. Chabbay, B. De Wit, R. Langlais, D. Mills, P. Moll, I. M. Otto, A. Petersen, C. Pohl, and L. Van Kerkhof. 2013. Opening up knowledge systems for better responses to global environmental change. Environmental Science & Policy 28:60-70. http://dx.doi.org/10.1016/j.envsci.2012.11.008

De Goede, M., and L. K. Hessels. 2014. Drijfveren van onderzoekers. Rathenau Institute, The Hague, The Netherlands. [online] URL: http://www.rijksoverheid.nl/bestanden/documenten-en-publicaties/rapporten/2014/11/25/drijfveren-van-onderzoekers/drijfveren-van-onderzoekers.pdf

Driessen, P. P. J., J. Behagel, D. L. T. Hegger, H. L. P. Mees, L. Almesjö, S. Andresen, F. Eboli, S. Helgenberger, K. Hollaender, L. Jacobsen, M. Järvelä, J. Laessoe, S. Oberthür, D. Avelar, U. Brand, A. Brunnengräber, H. Bulkeley, D. Compagnon, S. Davoudi, H. Hackmann, J. Knieling, C. Larrue, B. O. Linnér, O. Martin, K. O’Brien, S. O’Neill, H. F. M. W. Van Rijswick, B. Siebenhuener, C. A. J. M. Termeer, and A. Verbruggen. 2013. Societal transformations in the face of climate change: research priorities for the next decade. JPI Climate, Brussels, Belgium. [online] URL: http://www.jpi-climate.eu/publications/articles/10829915/Paper-Societal-transformations-in-the-face-of-climate-change-Research-priorities-for-the-next-decade

Edelenbos, J., A. Van Buuren, and N. Van Schie. 2011. Co-producing knowledge: joint knowledge production between experts, bureaucrats and stakeholders in Dutch water management projects. Environmental Science and Policy 14:675-684. http://dx.doi.org/10.1016/j.envsci.2011.04.004

Enengel, B., A. Muhar, M. Penker, B. Freyer, S. Drlik, and F. Ritter. 2012. Co-production of knowledge in transdisciplinary doctoral theses on landscape development—an analysis of actor roles and knowledge types in different research phases. Landscape and Urban Planning 105(1-2):106-117. http://dx.doi.org/10.1016/j.landurbplan.2011.12.004

Fry, G. L. A., B. Tress, and G. Tress. 2006. PhD students and integrative research. Pages 193-205 in B. Tress, G. Tress, G. L. A. Fry, and P. Opdam, editors. From landscape research to landscape planning - aspects of integration, education and application. Springer, Dordrecht, The Netherlands.

Funtowicz, S. O., and R. Ravetz. 1993. Science for the post-normal age. Futures 25(7):739-755. http://dx.doi.org/10.1016/0016-3287(93)90022-L

Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzmann, P. Scott, and T. Martin. 1994. The new production of knowledge: the dynamics of science and research in contemporary societies. Sage, London, UK.

Giddens, A. 1976. New rules of sociological method. Basic books, London, UK.

Giebels, D., A. Van Buuren, and J. Edelenbos. 2015. Using knowledge in a complex decision-making process—evidence and principles from the Danish Houting project’s ecosystem-based management approach. Environmental Science & Policy 47:53-67. http://dx.doi.org/10.1016/j.envsci.2014.10.015

Gieryn, T. F. 1983. Boundary-work and the demarcation of science from non-science: strains and interests in professional ideologies of scientists. American Sociological Review 48(6):781-795. http://dx.doi.org/10.2307/2095325

Groot, A., K. Hollaender, and R. Swart. 2014. Productive science-practice interactions in climate change adaptation. Lessons from practice. A Circle II research policy brief. Foundation of the Faculty of Sciences, Lisbon, Portugal.

Hegger, D., and C. Dieperink. 2014. Toward successful joint knowledge production for climate change adaptation: lessons from six regional projects in the Netherlands. Ecology and Society 19(2):34. http://dx.doi.org/10.5751/ES-06453-190234

Hegger, D., M. Lamers, A. Van Zeijl-Rozema, and C. Dieperink. 2012. Conceptualising joint knowledge production in regional climate change adaptation projects: success conditions and levers for action. Environmental Science and Policy 15:52-65. http://dx.doi.org/10.1016/j.envsci.2012.01.002

Hegger, D. L. T., A. Van Zeijl-Rozema, and C. Dieperink. 2014. Toward design principles for joint knowledge production projects: lessons from the deepest polder of The Netherlands. Regional Environmental Change 14(3):1049-1062. http://dx.doi.org/10.1007/s10113-012-0382-6

Hessels, L. K., and H. Van Lente. 2008. Re-thinking new knowledge production: a literature review and a research agenda. Research Policy 37:740-760. http://dx.doi.org/10.1016/j.respol.2008.01.008

Hoppe, R. 2009. Scientific advice and public policy: expert advisers’ and policymakers’ discourses on boundary work. Poiesis & Praxix 6(3):235-263. http://dx.doi.org/10.1007/s10202-008-0053-3

Iyalomhe, F., A. Jensen, A. Critto, and A. Marcomini. 2013. The science-policy interface for climate change adaptation: the contribution of communities of practice theory. Environmental Policy and Governance 23(6):368-380. http://dx.doi.org/10.1002/eet.1619

Kemp, R., and P. Martens. 2007. Sustainable development: how to manage something that is subjective and never can be achieved? Sustainability: Science, Practice, & Policy 3(2):5-14. [online] URL: http://sspp.proquest.com/archives/vol3iss2/0703-007.kemp.html

Lang, D. J., A. Wiek, M. Bergmann, M. Stauffacher, P. Martens, P. Moll, M. Swilling, and C. J. Thomas. 2012. Transdisciplinary research in sustainability science: practice, principles, and challenges. Sustainability Science 7(1):25-43. http://dx.doi.org/10.1007/s11625-011-0149-x

Lövbrand, E. 2011. Co-producing European climate science and policy: a cautionary note on the making of useful knowledge. Science and Public Policy 38(3):225-236.

Mattor, K., M. Betsill, C. Huayhuaca, H. Huber-Strearns, T. Jedd, F. Sternlieb, P. Bixler, M. Luizza, and A. S. Cheng. 2014. Transdisciplinary research on environmental governance: a view from the inside. Environmental Science & Policy 42:90-100. http://dx.doi.org/10.1016/j.envsci.2014.06.002

Mauser, W., G. Klepper, M. Rice, B. S. Schmalzbauer, H. Hackmann, R. Leemans, and H. Moore. 2013. Transdisciplinary global change research: the co-creation of knowledge for sustainability. Current Opinion in Environmental Sustainability 5:420-431. http://dx.doi.org/10.1016/j.cosust.2013.07.001

Merton, R. K. 1973. The normative structure of science. Pages 267-278 in The sociology of science: theoretical and empirical investigations. University of Chicago Press, Chicago, Illinois, USA.

Offermans, A., and P. Glasbergen. 2015. Boundary work in sustainability partnerships: an exploration of the Round Table on Sustainable Palm Oil. Environmental Science & Policy 50:34-45. http://dx.doi.org/10.1016/j.envsci.2015.01.016

Pallant, J. 2007. An SPSS survival manual: s step by step guide to data analysis using SPSS for Windows. Open University Press, Maidenhead, UK.

Popper, K. R. 1934(1959). The logic of scientific discovery. Routledge, London, UK.

Raymond, C. M., I. Fazey, M. S. Reed, L. C. Stringer, G. M. Robinson, and A. C. Evely. 2010. Integrating local and scientific knowledge for environmental management. Journal of Environmental Management 91:1766-1777. http://dx.doi.org/10.1016/j.jenvman.2010.03.023

Regeer, B. J., and J. G. F. Bunders. 2007. Kenniscocreatie: samenspel tussen wetenschap en praktijk. RMNO, The Hague, The Netherlands.

Regeer, B. J., and J. G. F. Bunders. 2009. Knowledge co-creation: interaction between science and society - a transdisciplinary approach to complex societal issues. VU University Amsterdam, Athena Institute, Amsterdam, The Netherlands.

Scholz, R. W., H. A. Mieg, and J. E. Oswald. 2000. Transdisciplinarity in groundwater management - towards mutual learning of science and society. Pages 477-487 in S. Belkin, editor. Environmental challenges Springer, Dordrecht, The Netherlands. http://dx.doi.org/10.1007/978-94-011-4369-1_37

Seijger, C., J. Van Tatenhove, G. Dewulf, and H. S. Otter. 2014. Responding to coastal problems: interactive knowledge development in a US nature restoration project. Oceans & Coastal Management 89:29-38. http://dx.doi.org/10.1016/j.ocecoaman.2013.12.011

Swart, R., R. Biesbroek, and T. Capela Lourenço. 2014. Science of adaptation to climate change and science for adaptation. Frontiers in Environmental Science 2:29. http://dx.doi.org/10.3389/fenvs.2014.00029

Tàbara, J. D. 2013. A new vision of open knowledge systems for sustainability: opportunities for social scientists. Pages 112-118 in World Social Science Report 2013: changing global environments. OECD and UNESCO, Paris, France. http://dx.doi.org/10.1787/9789264203419-en

Tàbara, J. D., and I. Chabay. 2013. Coupling human information and knowledge systems with social-ecological systems change: reframing research, education, and policy for sustainability. Environmental Science & Policy 28:71-81. http://dx.doi.org/10.1016/j.envsci.2012.11.005

Van Enst, W. I., P. P. J. Driessen, and H. A. C. Runhaar. 2014. Towards productive science-policy interfaces: a research agenda. Journal of Environmental Assessment Policy and Management 16(1). http://dx.doi.org/10.1142/S1464333214500070

Veraart, J. A., K. Van Nieuwaal, P. P. J. Driessen, and P. Kabat. 2014. From climate change to climate compatible development: experiences and progress in The Netherlands. Regional Environmental Change 14:851-863. http://dx.doi.org/10.1007/s10113-013-0567-7

Walter, A. I., S. Helgenberger, A. Wiek, and R. W. Scholz. 2007. Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method. Evaluation and Program Planning 30:325-338. http://dx.doi.org/10.1016/j.evalprogplan.2007.08.002

Wesselink, A., K. S. Buchanan, Y. Georgiadou, and E. Turnhout. 2013. Technical knowledge, discursive spaces and politics at the science-policy interface. Environmental Science and Policy 30:1-9. http://dx.doi.org/10.1016/j.envsci.2012.12.008

Wickson, F., A. L. Carew, and A. W. Russell. 2006. Transdisciplinary research: characteristics, quandaries and quality. Futures 38:1046-1059. http://dx.doi.org/10.1016/j.futures.2006.02.011

WIMEK (Wageningen Institute for Environment and Climate Research). 2014. Challenges and successes in interdisciplinary and transdisciplinary research and education - WIMEK experiences. WIMEK, Wageningen, The Netherlands.