It’s all uncertainty. It’s all risk. (Water utility representative, on the challenges of managing a hydropower project)
Uncertainty, the “perceived lack of knowledge, by an individual or group, that is relevant to the purpose or action being undertaken” (Abbott 2005:238, see also Sigel et al. 2010), is a fundamental feature and challenge of environmental decision making. Socio-environmental systems are dynamic, entailing nonlinear and complex interactions within and between social and environmental domains (Liu et al. 2007a, Dudney et al. 2018, Pulver et al. 2018). Managers face numerous unknown environmental and social parameters, including incomplete information about the causes or consequences of an environmental problem and the ways public or institutional values will change over time (Newig et al. 2005, Liu et al. 2007b). However, many environmental scholars tend to define these uncertainties solely as a risk management problem: “It is highly unlikely that a decision-maker (or analyst) will know all of the possible future states for the global system, or the probabilities of those states. Without this knowledge, it is not possible to define the conditional probability of outcomes” (Polasky et al. 2011:400). In this framing, uncertainty shapes the outcomes of a policy, and resolving the uncertainty entails collecting more information to reduce the uncertainty and/or using scenarios to envision a range of possible outcomes (Brugnach et al. 2008, Raadgever et al. 2011, Stelzenmüller et al. 2015).
However, this so-called scientific or substantive uncertainty represents only one dimension of uncertainty relevant to decision making. For instance, environmental planning and management processes often entail engaging with a number of different stakeholders; predicting what their values are and how individuals will relate to one another can complicate managing the process (Koppenjan and Klijn 2004, Abbott 2005, Bijlsma et al. 2011, Zandvoort et al. 2019). In addition, stochastic, truly unpredictable uncertainties, such as unexpected changes of government, large-scale environmental disturbances, or war, can drastically affect the context in which decision making occurs (Abbott 2005, Balint et al. 2011) and reveal a suite of new unanswered questions (Gross and Bleicher 2013). These uncertainties can affect the quality and outcomes of decision making. Uncertainties lead people to make irrational decisions (Kahneman and Tversky 2000) and encourage selfish behavior (Barrett and Dannenberg 2014). Moreover, individuals perceive risk and uncertainty in different ways, which can lead to miscommunication, inadequate ranking of values, and ineffective handling of environmental problems (Douglas and Wildavsky 1982, Lukacs et al. 2017). However, uncertainties are not uniformly detrimental to decision making, because they provide an opportunity for managers to learn and innovate (Gross and Bleicher 2013, Roberts 2013).
Scholars often point to collaborative governance, a governing arrangement in which decision-making power is shared among multiple organizations, as a tool for managing uncertainty (Koppenjan and Klijn 2004, Balint et al. 2011, Bijlsma et al. 2011, Raadgever et al. 2011, Duncan 2013, Hutter 2016, Kirschke and Newig 2017). Cross-organizational collaboration enables managers to incorporate diverse types of knowledge into a process and to understand how stakeholders might respond to a policy decision (Newig et al. 2005). For instance, through a participatory or collaborative modeling approach, stakeholders can work through competing value structures and openly evaluate the types of environmental and social unknowns that exist in a system (Cravens 2016, Ulibarri 2018a). In addition, inclusive, deliberative processes help build the problem-solving capacity and reduce the transaction costs necessary to deal with ambiguous, complex settings and relationships (Feldman and Khademian 2008, Gerlak et al. 2012). For these reasons, public organizations often shift toward more collaborative approaches when their problem setting becomes more uncertain and “wicked” (Lach et al. 2005, Hutter 2016). Indeed, uncertainty is recognized as a necessary driver of collaborative governance, because individuals seek to work together to share risk (Emerson and Nabatchi 2015).
However, collaboration may not always be an effective tool for managing uncertainty. It might increase uncertainty over the short term, because an issue “that is crystal clear to the competent authority might be reframed by the public participants” to make it less straightforward (Newig et al. 2005:341). Stakeholders may have competing views about what information is certain or uncertain (Zandvoort et al. 2019). Furthermore, the process of generating ideas with a diverse group of stakeholders serves to raise uncertainty about possible outcomes (Abbott 2005). These complications suggest that the interaction between collaboration and uncertainty may be more nuanced than commonly assumed.
I seek to clarify the interaction between uncertainty and collaboration in the management of environmental challenges, specifically the case of licensing a hydropower dam in California. I first investigate the types of uncertainty that arose during the collaborative process and describe the challenges they posed to decision making. I then explore how uncertainty and collaboration interact, asking (1) how collaboration worked to mitigate and/or amplify uncertainty and (2) how uncertainty supported or hindered the collaborative process. I conclude with implications for the design and practice of collaborative environmental governance.
Uncertainty is a challenging term; it is the topic of study across numerous disciplines and, as such, lacks common unifying characteristics (Smithson 2008). The Oxford English Dictionary defines uncertainty, somewhat tautologically, as “the quality of being uncertain in respect of duration, continuance, occurrence, etc.; liability to chance or accident … the quality of being indeterminate as to magnitude or value … The state of not being definitely known or perfectly clear; doubtfulness or vagueness.” At its most basic, uncertainty is thus an absence of knowledge about a topic (Abbott 2005).
However, the literature notes that this absence of knowledge may have different qualities and sources. First, the content of what is unknown can vary. In environmental governance work, a frequently considered type of uncertainty is scientific, also known as substantive or causal uncertainty, “a lack of knowledge about the substance … of the policy problem, e.g., the relation between soil properties and vegetation, the volatility of market prices, or the effect of land use on groundwater” (Bijlsma et al. 2011, see also Tickner 1999, Koppenjan and Klijn 2004, Abbott 2005, Balint et al. 2011). Much attention goes toward reducing scientific uncertainty, into trying to better model or predict the relationship between relevant components in a system: If we increase river flows, will this increase salmon habitat? A lack of knowledge may also arise about the rules and regulations that shape what decision makers can do, known as administrative or institutional uncertainty (Koppenjan and Klijn 2004, Balint et al. 2011). Scholars in management and planning also note that the people involved may be unpredictable, leading to “strategic” or “human and organizational uncertainty” about anticipating what other individuals know, will demand, or how they will react (Koppenjan and Klijn 2004, Abbott 2005). Finally, there are truly unknowable and unpredictable uncertainties, the stochastic or chance events that can drastically change the context in which decision making occurs (Abbott 2005, Balint et al. 2011).
Each type of uncertainty can be further assessed based on whether gathering more information will help resolve the uncertainty. In an “epistemological” uncertainty, which includes many instances of scientific, strategic, and institutional uncertainty, there are key knowledge gaps about the system. For instance, it is hard to estimate the exact distribution and amount of groundwater or the causes of asthma, but these knowledge gaps can be narrowed by gathering more information. In an “ontological” uncertainty, the system is inherently unpredictable, and gathering more information will not help (Walker et al. 2003, Brugnach et al. 2008). For instance, even with the best models, predicting the magnitude of sea level rise in an estuary has an inherent set of unknowns.
Finally, for each type of uncertainty, an individual may or may not be aware that he or she lacks knowledge about a topic (Böschen et al. 2010). If an individual is aware of the uncertainty, it becomes “nonknowledge,” and the individual has the choice to either account for that nonknowledge in decision making or ignore it. In contrast, a complete absence of knowledge, i.e., the “unknown unknowns,” which by definition cannot be perceived, is called “nescience.” Importantly, an unknown unknown can become known; it will be a surprise to the affected individuals, but then it enables them to recognize that there is a new set of information that they do not know (Gross 2010, Gross and Bleicher 2013).
It is important to note that the absence of knowledge is distinct from ignorance about existing knowledge (Roberts 2013). People may be unaware that they have knowledge, e.g., the norms embedded in the behavior or a group, or people may actively suppress certain types of knowledge, e.g., through maintaining secrets or taboos (Smithson 2008, Roberts 2013). I focus on ways individuals address the absence of knowledge, i.e., the known or unknown unknowns, rather than the deliberate or unconscious burying of existing knowledge.
Collaborative governance is “a governing arrangement where one or more public agencies directly engage nonstate stakeholders in a collective decision-making process that is formal, consensus-oriented, and deliberative and that aims to make or implement public policy or manage public programs or assets” (Ansell and Gash 2008:544). According to the collaborative governance regime framework (Emerson and Nabatchi 2015), which is based on one of the most comprehensive reviews of the collaboration literature, collaborative governance is composed of three interacting elements, or “dynamics,” that together yield changes in the world, both proximate outputs like learning and managerial capacity and longer term effects on the environment. The first, “principled engagement,” is the process by which participants engage with one another, specifically the use of deliberation and interest-based negotiation to make joint decisions. The second, “shared motivation,” is the affective stance of individual participants to one another and to the process as a whole, i.e., whether they trust the other participants, feel like their interests are respected by the group, and view the collaborative process as legitimate. The third, “capacity for joint action,” involves the structure and resources necessary to support engagement, including facilitation, leadership, and scientific information (Emerson and Nabatchi 2015). These three dynamics form the crux of collaboration.
I explore the interrelationship between uncertainty and collaboration through the ethnographic study of a process to license a hydropower dam in California. Managing hydropower is a quintessential wicked problem (Rittel and Webber 1973): It entails balancing generating electricity with other societal goods like water supply, recreational activities, water quality, and aquatic habitat, and the science surrounding hydropower’s effect on those resources is uncertain. Deciding whether to generate hydroelectricity, when, and how much thus engages competing and equally legitimate views about how the resource should be managed.
All hydropower dams in the United States are regulated by the Federal Energy Regulatory Commission (FERC), which issues 30- to 50-year licenses to project operators. The majority of dams were built in the mid-20th century, so as the original licenses near expiration, the project owners must apply for recertification from FERC and other environmental regulatory agencies. This “relicensing” process takes about 5 to 10 years and is designed to bring the project into alignment with changed public expectations about the value of environmental protection and with environmental regulations like the Clean Water Act and Endangered Species Act. Relicensing entails (1) a scoping process to identify potential resources impacted by the dam, (2) conducting studies to quantify those impacts, and (3) developing a license application with proposed operating requirements and/or management actions that will mitigate negative impacts. After the utility submits an application, it undergoes a variety of reviews to ensure compliance with federal protections (Ulibarri 2018b). Under FERC regulations, the hydropower utilities are required to hold several public meetings: to initiate the scoping process, share their proposed study plans, and share study results. However, some utilities, like the one I studied, opt for a highly collaborative approach, working closely with government agencies, nongovernmental organizations (NGOs), and other interested parties during each stage of license development (Ulibarri 2015a, b). FERC does not participate directly in the proceedings, nor is it present at meetings; instead, it acts as an external mediator and judge for decisions that are made by, or questions that arise from, the utility or stakeholder group.
The case studied is an ongoing hydropower relicensing process in California’s Central Valley. The utility, a local public water agency, has held more than 400 stakeholder meetings since initiating the relicensing in 2009. The meetings are convened by an engineering consulting firm that runs most hydropower licensing processes in California; the meetings themselves are run by a third-party facilitator. A fairly diverse set of stakeholders attends these meetings, including federal resource agencies, e.g., U.S. Forest Service (USFS); state resource agencies, e.g., California Department of Fish and Wildlife (CDFW); and environmental and recreational nonprofit organizations, e.g., Sierra Club, a sportfishing NGO. The process is oriented toward collaborative decision making, with a deliberative, consensus-based approach as the goal for negotiations.
I use an ethnographic, single-case approach drawing on meeting observations and stakeholder interviews, a useful approach for generating insights to build theory (Yin 2009). Between 2012 and 2016, I observed 71 stakeholder meetings, totaling more than 300 hours. When I started observations, the relicensing group had already finalized its list of technical studies, so this time frame captures how it carried out the studies, interpreted the results, and negotiated the proposed operating requirements. Observing decision making in situ provided a real-time record of who was present, what they discussed, how they engaged in the material and with one another, and how the group worked through conflicts, disagreements, and uncertainties; field notes captured these and many other topics.
In spring 2016, I conducted 26 interviews with representatives of all key organizations involved in the relicensing, including the hydropower utility; the consulting firm; federal agencies, e.g., USFS and National Marine Fisheries Service (NMFS); state agencies, e.g., CDFW and State Water Resources Control Board; NGOs; and the process facilitator. These 26 individuals represented the most frequent attendees of meetings, as well as several individuals whose organizations played a significant role in the relicensing but chose not to attend meetings. The interviews were semistructured, with prompts about how the interviewee had been involved in the process, with a particular focus on the development and implementation of the technical studies; on who they had worked with and why; on any challenges they faced, with regard to interacting with other participants and with making the decisions they were required to make, a question that elicited many answers regarding both collaboration and uncertainty; and on anything they would change if they could go through the process again. The interview protocol did not explicitly cover uncertainty. As many interviewees had participated in numerous other relicensings, I also asked each individual to compare his or her experience in this relicensing process to any others the individual had been engaged in, to contrast collaboration and decision making in this case with the broader norms of joint decision making and water management.
The field notes and interview transcripts were analyzed using a modified grounded theory approach (Corbin and Strauss 2008). The first round of codes identified instances of uncertainty, which I conceptualized broadly as any time when stakeholders did not know something they needed to make a decision. I then iteratively refined these codes to develop a typology of uncertainty, which mirrors, but was not influenced by, the literature reviewed previously. To explore whether and how collaborative approaches were used to manage each uncertainty, I revisited the field notes and interviews to evaluate the context under which each uncertainty arose and what actions stakeholders took, individually or as a group, to resolve or move past each uncertainty.
About 3 years into the relicensing, the utility conducted some routine monitoring work at a diversion dam and discovered that the reservoir was filling with sediment. This was a surprise, causing concern for the dam’s integrity and the utility’s ability to divert water to the hydropower facility. At the following stakeholder meeting, the utility raised the idea of testing whether it could flush sediment through a low-level outlet during rainstorms. Environmental organizations, especially the USFS, liked this idea because it would introduce more sediment into the river, which would be beneficial for fish habitat and for riparian vegetation. From the water supply, electricity generation, and environmental perspectives, this was a win-win solution.
The first step was figuring out whether the idea was feasible. The utility had never used the outlet and so did not know how much water it would pass: “The engineering design says the sluice gate can pass 800 cfs [cubic feet per second], but there’s a lot of sediment there … Probably flows are more realistically 100 cfs” (engineering consultant speaking on behalf of the utility). To find out how much water it would pass, and therefore estimate how much sediment it would mobilize, the utility needed to (1) excavate the sediment held behind the dam and (2) test the sluice gate to see how much water would flow under it. However, this seemingly simple task took almost 2 years to complete because of numerous interacting uncertainties.
The utility first needed to identify and obtain permits to excavate sediment. Exactly which permits it needed and when those permits would be available was unclear. For instance, the dam was partially located on USFS land, but it was unclear whether the utility needed federal permission to undertake construction. There was also disagreement as to whether a permit was needed from the state (CDFW), because the activity might have been allowed under an umbrella exemption to operate the project, and if so, whether the dredging and sluice gate testing required separate permits. After it was determined that each of these permits was needed, the group then had the uncertainty of not knowing when the permits would be issued. Permits were the topic of a brief conversation at almost every meeting during the multiple months they were outstanding. Second, once permits were in hand, the utility needed a period without rainfall, and several days’ notice that flows would be low to mobilize people and equipment, to excavate behind the dam. Third, once the sediment was excavated, the utility needed an additional set of permits and a multiday rainstorm to conduct the test flows. As this occurred during a severe, multiyear drought, the timing of the rainstorm was highly unpredictable, i.e., days or years away. Finally, throughout this process, the group needed to estimate when it would have the information needed from my study to proceed with making management recommendations, which were interrelated with other areas of the relicensing, and to hope it was in time for the regulatory deadline set by FERC.
This whole process was made even more complex by an unpredictable external change. Three months after the sediment problem was discovered, the federal government shut down for the first time in 17 years. The USFS and U.S. Fish and Wildlife Service could no longer attend meetings, the U.S. Army Corps of Engineers could not process permit applications, and FERC was not available for guidance on how to manage the outstanding sediment questions or when a final decision would be due. Although nonfederal stakeholders continued to meet, the group had no way of knowing when the shutdown would end and full-group decision making could resume.
This example illustrates that a fairly straightforward scientific uncertainty, i.e., how much water would pass through the outlet, became a far more complex problem to solve because each decision highlighted additional unknowns that needed to be answered. Indeed, it took more than 3 years for the group to “answer” its initial question, i.e., how much sediment would pass through the sluice gate, when from a purely technical perspective, and with cooperative weather, it could have been resolved within a month. Although this is a single example, it encapsulates the sorts of cascading, complex uncertainties that were a core feature of most meetings I attended.
Throughout the relicensing, numerous different types and qualities of uncertainty arose. The whole sediment incident started with a surprise, an unknown unknown being made known (Gross and Bleicher 2013), and with it awareness of numerous new questions that needed to be addressed. Some of these open questions, such as how much sediment the outlet would pass, were epistemological uncertainties: They had a knowable range of possible outcomes, and by gathering additional information, the group could narrow that range. Others, like when it would rain, were ontological: The group knew it had a knowledge gap but could not resolve that gap by gathering new information. Moreover, these uncertainties interacted and built on one another, e.g., having rain forecast meant nothing if permits had not been obtained to do the construction work.
Faced with the far-reaching and interactive nature of uncertainties in the relicensing process and a statutory deadline from FERC, the stakeholders in the relicensing had to find ways to keep the decision-making process moving forward in the face of uncertainty. The convening utility was committed to a collaborative approach, so it held stakeholder meetings multiple times per month and made most decisions, including decisions on how to address uncertainties, with input from a diverse set of stakeholders. However, collaboration varied in how effectively it could address the different types of uncertainty stakeholders faced.
From a substantive perspective, collaboration enables exchange of information among actors who would not otherwise communicate or who would not communicate as efficiently (Newig et al. 2010, Berardo 2014, Steelman et al. 2014). Under this framing, aggregating information that exists somewhere within the network of stakeholders should help to resolve or at least manage uncertainties. For addressing many of the epistemological uncertainties in the relicensing, which required gathering more information, collaboration was an apt solution. For instance, when trying to identify what permits were required for a particular activity, stakeholders would talk through the requirements of the activity (e.g., Which access roads will you use? Will you have a representative from CDFW present to help handle fish?) to determine what regulations would be triggered and therefore what permits were needed. Sometimes stakeholders in the room had the expertise to finalize exactly which permits were needed; if not, stakeholders would ask their networks, e.g., legal experts within their organization or people they knew from other relicensings, to specify what protections were needed. Because of the collaborative process, deciding which permits were needed was thus fairly straightforward when the lead agency was participating in the process. In contrast, it was much more difficult to get guidance on permits required by agencies that were not present, e.g., U.S. Army Corps of Engineers and NMFS.
However, resolving uncertainties sometimes required more than simply aggregating information that existed within the network, because not all members of the group were initially willing to either share everything they knew or to accept what other people were saying at face value. One interviewee described the slow dialogic process it had taken for the group to stop withholding information and believe one another’s statements:
I’m just learning to just recognize that most of [when someone says they can’t do something] is just smoke, and that if we stay at the table, we keep talking, eventually these understandings will come … Staying at the table where misinformation, disinformation, sequestered information, all that stuff that goes on in this process becomes moot when information becomes available through the collaboration.
For several other interviewees, the importance of trusting the other participants’ words emerged when they compared this relicensing with other relicensings in which they had participated. For instance, one interviewee said that in many other relicensings, the utilities liked to play “hide the ball” with stakeholders, being deliberately unforthcoming with stakeholders.
The collaborative governance literature has shown that collaboration can build trust between stakeholders such that people are more willing to exchange and/or receive information (Koppenjan and Klijn 2004, Feiock et al. 2010, Emerson and Nabatchi 2015, Ulibarri 2018a). As Emerson and Nabatchi (2015:66) report, the development of trust “happens over time as parties work together, get to know each other, and prove to each other that they can be reasonable, predictable, and dependable.” In this case, collaboration appears to have been serving this role. By staying at the table and talking to one another, participants were able to move past the assumption that other participants were deliberately skewing or withholding information and adopt information shared at face value. At the same time, it helped individuals be more willing to share their interests and knowledge fully. This meant that the epistemological uncertainties arose because of an honest lack of knowledge, rather than deliberate construction of uncertainty through secrecy (Smithson 2008).
A number of other uncertainties that arose during the relicensing involved unknown external parameters, such as when it would rain, when FERC would make a decision on an unresolved question, or whether a nearby relicensing would go into a study dispute, parameters about which no one in the room, and in some cases no one save the rain gods, had information. For these ontological uncertainties, gathering more information would not resolve them. From a collective action perspective, these uncertainties should be harder to resolve through collaboration because they involve unknowns that are external to the core group of actors (York and Schoon 2011). However, stakeholders still turned to collaborative approaches to address these more unresolvable uncertainties.
First, with guidance from the group facilitator, a “process team” met regularly (every 2 to 3 months) to set meeting schedules and agendas. This process team had a representative from the utility, the consulting team, all major participating government agencies, and a coordinator from the consortium of participating nonprofits. At its meetings, the team would assess the status of current negotiations and reprioritize what it thought would be the best direction to move forward. The process team meetings served as a critical tool for stakeholders to triage around uncertainties and ensure that nothing was forgotten, even if the right information or personnel were not available to make an immediate decision. Process team meetings were also the first point of entry for triaging with other relicensing processes.
Second, in full stakeholder meetings, the group members discussed uncertainties and talked explicitly about their process. When there was an outstanding permit or an unpredictable weather pattern, the group members talked about it and made a plan so that they would be ready to act as soon as the conditions were right. For instance, to handle a study delayed by rain, a participant recommended, “I think try to keep it moving along so [the study] can go whenever it starts raining. Can we put in on an agenda? Especially the methodology to track sediment movement?” The group members wanted to have the study design, methodology, and who would be responsible decided in advance, so that they were ready as soon as the physical conditions were ripe.
In both of these approaches, collaboration helped by getting diverse sets of people in the room talking to one another. Talking openly about uncertainty with the full stakeholder group created a setting that enabled more thoughtful and efficient handling of uncertainty. With a diverse group in the room, stakeholders could both identify uncertainties quickly and consider many different ways to work around them. Moreover, the group held expertise about administration, biology, and hydrology, enabling more nuanced understanding of each uncertainty’s broader implications. When a series of fish surveys failed to find any fish because an ongoing drought made river temperatures too high, the biologists in the room could quickly think through what the low counts meant for making inferences about the health of the river and what other methods they could use.
Although collaboration helped stakeholders move forward and work with uncertainty, it is important to note that collaboration sometimes increased uncertainty over the short term. When the group identified a new knowledge gap, such as the sediment buildup or the fish survey’s failure to catch fish, this made the perceived uncertainty increase. As stakeholders would discuss each new uncertainty, they would raise questions, e.g., how do we think FERC will react to this new information, or what options do we have at our disposal to address the question? Each question thus begat one or more new uncertainties, which stakeholders had to either answer or manage around. Without diverse stakeholders in the rooms, i.e., without collaboration, it is conceivable that some of these questions would never have been raised. In other words, collaboration made unknown unknowns visible to the full group.
Many interviewees felt that uncertainty helped to facilitate collaboration because it brought stakeholders together to solve a problem. As one consultant described,
the uncertainty that [a regulatory unknown] introduces to all of the parties, to every party involved, is so significant if you think about it a little bit. That’s a benefit to me, especially to the licensee, but it’s a benefit to everybody. It creates an atmosphere where you’d want to collaborate, almost collaborate, because … Nobody knows what power they really have … While that’s not certain on the [utility] side, it creates enough uncertainty that it really moves people to go to discussions in a serious way. You don’t know how it’s going to come out. It will take forever, because there’ll be so many lawsuits that would be filed and appeals to be had under that. It could go on 20 years. The uncertainty over that I think creates a collaborative environment … I think this uncertainty weighs on everybody. People have thought about it.
In this case, the interviewee highlights the uncertainty caused specifically by the threat of lawsuits, which increase costs, create delays, and can undermine previously “certain” decisions. The consultant felt that by collaborating, the stakeholders could make lawsuits less likely. However, lawsuits occur frequently in FERC relicensing processes, even highly collaborative ones (Ulibarri 2015b).
Likewise, a big concern for many stakeholders was not knowing how FERC would rule on the management recommendations the group put forward. As a representative of the utility, the group with the most to gain or lose from the relicensing, said,
We’re really handcuffed. If we had to do the study, report the results as a data dump, and then wait until the FERC does their environmental analysis based on the data and preliminary terms and conditions, that’s just way too uncertain for us just to sit around and wait.
Instead, he preferred trying “to reach a local decision, rather than throw it up to the FERC and let them make the decision, because no one’s happy with FERC’s decision.” These statements echo a common understanding in the literature, wherein uncertainty is a necessary driver of collaborative governance (Ansell and Gash 2008, Emerson et al. 2012, Emerson and Nabatchi 2015).
In practice, uncertainty appeared to initiate collaboration for uncertainties arising because a decision was out of an individual’s or the group’s control. When FERC, regulatory agencies, or the legislature was making a decision that would directly affect the group, stakeholders turned deliberately to working toward unanimous consent-based decisions. This approach was most prominent for managing the “elephant in the room,” which was the absent regulatory agencies. Stakeholders felt that FERC was more likely to approve requested management decisions if there was broad support for them: “I think there is benefit to saying that everyone but [the missing agency] has agreed to X—yes they have regulatory authority, but we worked collaboratively [to develop our proposal]” (state agency representative). Uncertainty was motivating people to work toward a single shared decision, a fairly undeniable example of uncertainty driving collaboration.
However, very few other uncertainties resulted immediately in collaborative dialogue among the full stakeholder group. Instead, when confronted with a new uncertainty, stakeholders would often request a short caucus without the utility and its consulting team present. In these breakout sessions, representatives of the government agencies and NGOs would discuss their individual interests as they related to the uncertainty and develop a unified narrative or response to share with the utility. Only after they had worked through their individual concerns and hesitations would they discuss the uncertainty with the utility. In these instances, uncertainty did lead individuals to collaborate, but only a constrained group of stakeholders, i.e., the agencies and NGOs. Although these stakeholders had diverse interests and mandates, they were less diverse than the full group, including the hydropower utility.
For example, if asked when they would be ready to discuss in-stream flow requirements in the downstream section of river, the agencies called a caucus with the NGOs. The caucus consisted of updates about an internal GIS analysis that the U.S. Fish and Wildlife Service was conducting, debates about whether and how the agency staff needed to bring their bosses into the discussion, and what to do about the fact that a key regulatory agency was missing. After discussing these outstanding questions, the group returned to announce that it was not ready to discuss the downstream section until the GIS analysis was complete. In this case, the agencies and NGOs were essentially prioritizing which uncertainties they felt mattered most for triaging the discussion. To do this prioritization, they raised all of the open questions about the topic, which increased the diversity of options, and therefore the uncertainty of outcomes, for a short time; however, the agencies and NGOs chose to do this without the utility, the organization with the most distinct interests, in the room.
Caucuses were also called after questions about how to change the modeled hydrology to address changing upstream flows and whether certain flow recession rates would improve riparian habitat. In these cases, there was a disagreement among the utility, agencies, and/or NGOs about the uncertainty’s implications for decision making, i.e., about whether it was an uncertainty and, if so, whether it was solvable. Stakeholders appeared unwilling to speak up about these disagreements with the full group but would break to raise it with more like-minded people. Once they had an internal narrative about why it was an uncertainty and how it mattered for decision making, they would return to the full group. In these instances, the caucuses increased ambiguity (Brugnach et al. 2008) over the short term.
Finally, it is important to note that all caucuses did not follow an uncertainty, because some were called when the utility asked the group for a firm decision. They followed questions such as: How should we stock fish? What requirements do we want for a monitoring group? Should we remove the temperature loggers yet? In these cases, as with uncertainty-related caucuses, the agency/NGO group would come to a decision internally and then report back to the utility.
I explored the interrelationship between uncertainty and collaborative governance in the context of a multistakeholder hydropower licensing process. Numerous types of uncertainties arose during the 4 years of negotiation, and these uncertainties often interacted in messy and hard-to-predict ways. Collaboration was an important tool for stakeholders to handle the uncertainties that arose. In some cases, collaboration helped move information through the network, helping resolve uncertainties more quickly; in other cases, collaboration helped stakeholders triage around uncertainties that could not be resolved by gathering more information. In addition, many stakeholders talked about uncertainty as a motivating factor in why they chose to collaborate. At the same time, the link between uncertainty and collaboration was sometimes blurred, as stakeholders opted to address some uncertainties in a two-stage process, with internal collaboration to handle the initial messiness and then full-group collaboration to work through bigger picture implications.
The first implication of this work is that many types of uncertainty arise in environmental governance, not just the scientific or substantive uncertainty that environmental scholars often consider (e.g., Sigel et al. 2010, Polasky et al. 2011, Raadgever et al. 2011, Head 2014, Hurlbert and Gupta 2016, Hutter 2016, Poff et al. 2016, Lukacs et al. 2017). This included strategic uncertainties about the goals and actions of individuals and organizations (Koppenjan and Klijn 2004, Abbott 2005) and surprise events that fundamentally reshaped the decision-making context (Balint et al. 2011). Some of the uncertainties arose from within the stakeholder group, e.g., trying to predict how long it would take to negotiate a flow regime, whereas others were external to the collaboration and outside of any stakeholder’s control, e.g., when it might rain during an extreme drought. The group developed tools to plan around these uncertainties in a collaborative manner, yet the uncertainties themselves made environmental problem solving and negotiation more challenging. Importantly, my study is not the first to acknowledge the diversity or complexity of uncertainty for solving environmental problems (Abbott 2005, Bijlsma et al. 2011, Kirschke and Newig 2017). It is necessary to say again, however, given the field’s overwhelming emphasis on uncertainty as relating predominately to unknown causes of a problem or outcomes of a decision.
This research also reveals several new observations about the nature of uncertainty in environmental management. First, existing typologies of uncertainty often consider different types of uncertainty in isolation (Koppenjan and Klijn 2004, Abbott 2005, Brugnach et al. 2008, Balint et al. 2011). In this research, uncertainties rarely arose individually but instead interacted and compounded. In the sediment example, the resolution of just a physical or administrative uncertainty alone, e.g., a rainfall event without having permits in hand, would have been useless for helping the group make progress. Second, other than accounting for large-scale stochastic events, existing typologies tend to neglect the way that external factors affect what happens in a given decision-making forum. The exception is Abbott (2005:242), whose external uncertainty, i.e., “uncertainty about the wider social environment and how it relates to and influences the situation,” describes the need to account for external variables that change the boundary conditions under which a decision is made. However, Abbott’s example of external uncertainty (migration) is quite distinct from the planning process he studies. Although large-scale external changes certainly created, and sometimes resolved, uncertainties, the FERC relicensing also had a number of external uncertainties that were tightly linked to the planning process. In particular, there was a closely interlinked set of policy or planning venues, including other relicensings and watershed forums, with the same set of actors engaged in numerous venues across the region (Lubell et al. 2010, Lubell 2013) and similar environmental problems being addressed. Decisions in one venue, from setting a meeting date to negotiating an interagency memorandum of agreement or a new set of flow requirements, had cascading effects on uncertainty throughout the network. This type of external uncertainty has not previously been described.
As for the interaction between collaboration and uncertainty, this work supports the common supposition in the literature that collaboration is well suited to address uncertainty (Koppenjan and Klijn 2004, Balint et al. 2011, Bijlsma et al. 2011, Raadgever et al. 2011, Duncan 2013, Hutter 2016, Kirschke and Newig 2017). For epistemological uncertainties, having diverse stakeholders in the room expanded the suite of knowledge and expertise they had available, so they could efficiently find answers to outstanding questions. For ontological uncertainties that could not be “solved” but were instead inherent to the system, interorganizational collaboration set up the process, structure, and relationships for stakeholders to manage uncertainty. Process mattered in that talking openly about uncertainty with the full stakeholder group created a setting that enabled more thoughtful and efficient handling of uncertainty, particularly through triaging so the decisions could keep moving forward. Both of these approaches benefited from the structure of regular meetings and time to build trust-based working relationships (Emerson and Nabatchi 2015), which highlights the need for full collaboration, i.e., joint decision making by diverse stakeholders, rather than just sharing information via consultation or coordination (Arnstein 1969). Although some scholars have argued that collaboration can make the handling of uncertainty less effective (Zandvoort et al. 2019), this tendency was not apparent in the FERC case.
Uncertainty, however, was both a driver of and barrier to collaboration. Interviewees felt that uncertainty helped bring stakeholders to the table (Ansell and Gash 2008, Emerson et al. 2012, Emerson and Nabatchi 2015), but the dynamic of two-stage negotiations around many newly arisen uncertainties showed that uncertainty sometimes undermined collaborative dialogue and deliberation or at least encouraged stakeholders to deliberate first with more kindred individuals.
Unfortunately, the data do not reveal why the resource agencies and NGOs opted to use so many caucuses, but there are several plausible explanations. First, the participants could be strategically capitalizing on the uncertainty as an opportunity to frame the debate in a way that is most favorable to them. In negotiation theory, how an offer is framed and when it is presented affects the types of substantive outcomes that result (Fisher et al. 2011). By ironing out internal inconsistencies, the agency/NGO caucus could put forward a proposal that was most mutually beneficial. If this explanation is correct, the likely mechanism underpinning two-stage negotiation is level of trust in the other participants: If all the participants trusted one another to act in the group’s mutual best interest, they would be more likely to work jointly with the full group throughout the process.
Conversely, stakeholders could be fearful of the uncertainty and feel that they maintain a greater sense of control if they are able to handle it in a more constrained group. Ideas of psychological safety suggest that when people feel insecure they seek their comfort zone (Edmondson and Lei 2014, Frazier et al. 2017), and stakeholders could be responding to uncertainty by turning to the more like-minded individuals in the group. Dialogue within that constrained group would give people a sense of control over the uncertainty, helping them feel secure enough to return to the full group. If this second explanation is correct, the likely mechanism driving collaboration is trust in the process: Despite their experience, stakeholders do not believe that principled engagement will serve to manage the uncertainties effectively. However, if the natural tendency when faced with an uncertainty is to bury one’s head in the sand, the fact that people are coming to the table eventually is critical.
What is particularly interesting about this pattern of two-stage negotiations is that the risk of being wrong about the uncertainty most often fell to the utility, not to the stakeholders calling the caucus. The utility, which bankrolls the entire relicensing, risks paying more money in consulting fees and study costs in the case of delays; in the case of a miscalculated uncertainty that leads to infrastructure failure, it might lose reputation, be sued, face bankruptcy, or lose its license. Although a wildlife protection agency does face some risk of endorsing a decision that leads to species decline, the overall risks they bear from the relicensing are minimal. One would expect that the organization bearing risk more directly would be most likely to have a need for psychological safety, but this did not appear to be the case. In some instances, the utility and its consulting team did know about the uncertainty before the full group, so they would have had some time to do internal planning, akin to the agency/NGO caucuses, but this was not always the case.
As a single case study, a key outstanding question is how similar uncertainties would have played out in other settings. Although careful attention to uncertainty was necessary for moving the dialogue forward in this case, in a less collaborative setting, not discussing uncertainties as a group could lead stakeholders to feel that they were not being addressed adequately and therefore drive participants away from the table. In a context where organizations are already choosing to allocate their scarce time, staff, and resources to one collaborative group at the expense of others (Lubell et al. 2010), knowing how to attract a core of participants and sustain their engagement over time is critical for developing effective collaboration (Bonnell and Koontz 2007, Hui et al. 2018, Scott et al. 2018).
Additionally, although stakeholders grappled with diverse and interacting uncertainties, the uncertainties in this case were perhaps more constrained than might exist in other collaborative settings. In contrast to many grassroots collaborations that arise in the environmental space, FERC relicensing has a relatively narrow set of issues that is up for negotiation and a single, fixed goal: develop a new operating license. Thus, this case might represent a lower bound on the types of uncertainty stakeholders face; increasing the number and types of uncertainties faced may make collaboration less effective as it creates more individual components to triage.
In conclusion, I argue for a more nuanced understanding of the relationship between collaboration and uncertainty in the management of environmental problems. By expanding the range of uncertainties that arise in collaborative processes and how those uncertainties interplay with stakeholder interaction, this work seeks to enable project managers and others who convene collaborative groups to better manage uncertainty’s role in decision making. Careful consideration of how collaboration and uncertainty interact can help conveners and participants in collaborative governance processes optimize their interactions, which ultimately results in better outcomes for the environment and the people who rely on it (Scott 2015, Ulibarri 2015b).
 In this manuscript, I use the term uncertainty because it is the most common term used by my audience, i.e., environmental scientists, when discussing the unknown. Scholars in the sociology of science adopt the term ignorance (Böschen et al. 2010, Gross 2010, Roberts 2013), which captures both nonknowledge, i.e., things that are unknown, what I am calling uncertainty, as well as knowledge that is consciously or unconsciously suppressed.
This paper benefited from numerous sets of feedback from colleagues near and far. The author is grateful to Amanda Cravens, Megan Mullin, Rebecca Nelson, Anne Siders, and the Center for Organization Research at UC Irvine, particularly Martha Feldman and Maritza Salazar, for helping frame my ideas, as well as helpful suggestions from two anonymous reviewers. The work was supported by grants from the Emmett Interdisciplinary Program for Environment and Resources, the Vice Provost for Graduate Education, the Stanford Center for International Conflict and Negotiation, the School of Earth, Energy and Environmental Sciences at Stanford University, and a David and Lucille Packard Foundation Stanford Graduate Fellowship.
Abbott, J. 2005. Understanding and managing the unknown: the nature of uncertainty in planning. Journal of Planning Education and Research 24(3):237-251. https://doi.org/10.1177/0739456X04267710
Ansell, C., and A. Gash. 2008. Collaborative governance in theory and practice. Journal of Public Administration Research and Theory 18(4):543-571. https://doi.org/10.1093/jopart/mum032
Arnstein, S. R. 1969. A ladder of citizen participation. Journal of the American Institute of Planners 35(4):216-224. https://doi.org/10.1080/01944366908977225
Balint, P. J., R. E. Stewart, A. Desai, and L. C. Walters. 2011. Wicked environmental problems: managing uncertainty and conflict. Island, Washington, D.C., USA. https://doi.org/10.5822/978-1-61091-047-7
Barrett, S., and A. Dannenberg. 2014. Sensitivity of collective action to uncertainty about climate tipping points. Nature Climate Change 4:36-39. https://doi.org/10.1038/nclimate2059
Berardo, R. 2014. Bridging and bonding capital in two-mode collaboration networks. Policy Studies Journal 42(2):197-225. https://doi.org/10.1111/psj.12056
Bijlsma, R. M., P. W. G. Bots, H. A. Wolters, and A. Y. Hoekstra. 2011. An empirical analysis of stakeholders’ influence on policy development: the role of uncertainty handling. Ecology and Society 16(1):51. https://doi.org/10.5751/ES-03865-160151
Bonnell, J. E., and T. M. Koontz. 2007. Stumbling forward: the organizational challenges of building and sustaining collaborative watershed management. Society & Natural Resources 20(2):153-167. https://doi.org/10.1080/08941920601052412
Böschen, S., K. Kastenhofer, I. Rust, J. Soentgen, and P. Wehling. 2010. Scientific nonknowledge and its political dynamics: the cases of agri-biotechnology and mobile phoning. Science, Technology, & Human Values 35(6):783-811. https://doi.org/10.1177/0162243909357911
Brugnach, M., A. Dewulf, C. Pahl-Wostl, and T. Taillieu. 2008. Toward a relational concept of uncertainty: about knowing too little, knowing too differently, and accepting not to know. Ecology and Society 13(2):30. https://doi.org/10.5751/ES-02616-130230
Corbin, J., and A. Strauss. 2008. Basics of qualitative research: techniques and procedures for developing grounded theory. Third edition. Sage, Los Angeles, California, USA. https://dx.doi.org/10.4135/9781452230153
Cravens, A. E. 2016. Negotiation and decision making with collaborative software: how MarineMap ‘changed the game’ in California’s Marine Life Protected Act Initiative. Environmental Management 57(2):474-497. https://doi.org/10.1007/s00267-015-0615-9
Douglas, M., and A. Wildavsky. 1982. Risk and culture: an essay on the selection of technological and environmental dangers. University of California Press, Berkeley, California, USA.
Dudney, J., R. J. Hobbs, R. Heilmayr, J. J. Battles, and K. N. Suding. 2018. Navigating novelty and risk in resilience management. Trends in Ecology & Evolution 33(11):863-873. https://doi.org/10.1016/j.tree.2018.08.012
Duncan, R. 2013. Opening new institutional spaces for grappling with uncertainty: a constructivist perspective. Environmental Impact Assessment Review 38:151-154. https://doi.org/10.1016/j.eiar.2012.07.004
Edmondson, A. C., and Z. Lei. 2014. Psychological safety: the history, renaissance, and future of an interpersonal construct. Annual Review of Organizational Psychology and Organizational Behavior 1:23-43. https://doi.org/10.1146/annurev-orgpsych-031413-091305
Emerson, K., and T. Nabatchi. 2015. Collaborative governance regimes. Georgetown University Press, Washington, D.C., USA.
Emerson, K., T. Nabatchi, and S. Balogh. 2012. An integrative framework for collaborative governance. Journal of Public Administration Research and Theory 22(1):1-29. https://doi.org/10.1093/jopart/mur011
Feiock, R. C., I. W. Lee, H. J. Park, and K.-H. Lee. 2010. Collaboration networks among local elected officials: information, commitment, and risk aversion. Urban Affairs Review 46(2):241-262. https://doi.org/10.1177/1078087409360509
Feldman, M. S., and A. M. Khademian. 2008. The continuous process of policy formation. Pages 37-59 in K. Ahmed and E. Sánchez-Triana, editors. Strategic environmental assessment for policies: an instrument for good governance. The World Bank, Washington, D.C., USA.
Fisher, R., W. L. Ury, and B. Patton. 2011. Getting to yes: negotiating agreement without giving in. Revised edition. Penguin Books, New York, New York, USA.
Frazier, M. L., S. Fainshmidt, R. L. Klinger, A. Pezeshkan, and V. Vracheva. 2017. Psychological safety: a meta-analytic review and extension. Personnel Psychology 70(1):113-165. https://doi.org/10.1111/peps.12183
Gerlak, A. K., T. Heikkila, and M. Lubell. 2012. The promise and performance of collaborative governance. Pages 413-434 in M. E. Kraft and S. Kamieniecki, editors. The Oxford handbook of U.S. environmental policy. Oxford University Press, Oxford, UK. https://doi.org/10.1093/oxfordhb/9780199744671.013.0019
Gross, M. 2010. Ignorance and surprise: science, society, and ecological design. MIT Press, Cambridge, Massachusetts, USA. https://doi.org/10.7551/mitpress/9780262013482.001.0001
Gross, M., and A. Bleicher. 2013. ‘It’s always dark in front of the pickaxe’: organizing ignorance in the long-term remediation of contaminated land. Time & Society 22(3):316-334. https://doi.org/10.1177/0961463X12444059
Head, B. W. 2014. Evidence, uncertainty, and wicked problems in climate change decision making in Australia. Environment and Planning C: Government and Policy 32(4):663-679. https://doi.org/10.1068/c1240
Hui, I., N. Ulibarri, and B. Cain. 2018. Patterns of participation and representation in a regional water collaboration. Policy Studies Journal, in press. https://doi.org/10.1111/psj.12266
Hurlbert, M., and J. Gupta. 2016. Adaptive governance, uncertainty, and risk: policy framing and responses to climate change, drought, and flood. Risk Analysis 36(2):339-356. https://doi.org/10.1111/risa.12510
Hutter, G. 2016. Collaborative governance and rare floods in urban regions – dealing with uncertainty and surprise. Environmental Science & Policy 55(2):302-308. https://doi.org/10.1016/j.envsci.2015.07.028
Kahneman, D., and A. Tversky. 2000. Choices, values, and frames. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511803475
Kirschke, S., and J. Newig. 2017. Addressing complexity in environmental management and governance. Sustainability 9(6):983. https://doi.org/10.3390/su9060983
Koppenjan, J. F. M., and E.-H. Klijn. 2004. Managing uncertainties in networks: a network approach to problem solving and decision making. Routledge, London, UK.
Lach, D., S. Rayner, and H. Ingram. 2005. Taming the waters: strategies to domesticate the wicked problems of water resource management. International Journal of Water 3(1):1-17. https://doi.org/10.1504/IJW.2005.007156
Liu, J., T. Dietz, S. R. Carpenter, M. Alberti, C. Folke, E. Moran, A. N. Pell, P. Deadman, T. Kratz, J. Lubchenco, E. Ostrom, Z. Ouyang, W. Provencher, C. L. Redman, S. H. Schneider, and W. W. Taylor. 2007a. Complexity of coupled human and natural systems. Science 317(5844):1513-1516. https://doi.org/10.1126/science.1144004
Liu, J., T. Dietz, S. R. Carpenter, C. Folke, M. Alberti, C. L. Redman, S. H. Schneider, E. Ostrom, A. N. Pell, J. Lubchenco, W. W. Taylor, Z. Ouyang, P. Deadman, T. Kratz, and W. Provencher. 2007b. Coupled human and natural systems. AMBIO: A Journal of the Human Environment 36(8):639-649. https://doi.org/10.1579/0044-7447(2007)36[639:CHANS]2.0.CO;2
Lubell, M. 2013. Governing institutional complexity: the ecology of games framework. Policy Studies Journal 41(3):537-559. https://doi.org/10.1111/psj.12028
Lubell, M., A. D. Henry, and M. McCoy. 2010. Collaborative institutions in an ecology of games. American Journal of Political Science 54(2):287-300. https://doi.org/10.1111/j.1540-5907.2010.00431.x
Lukacs, H. A., N. Sawe, and N. Ulibarri. 2017. Risk, uncertainty, and institutional failure in the 2014 West Virginia chemical spill. Case Studies in the Environment 1(1):1-7. https://doi.org/10.1525/cse.2017.000604
Newig, J., D. Günther, and C. Pahl-Wostl. 2010. Synapses in the network: learning in governance networks in the context of environmental management. Ecology and Society 15(4):24. https://doi.org/10.5751/ES-03713-150424
Newig, J., C. Pahl-Wostl, and K. Sigel. 2005. The role of public participation in managing uncertainty in the implementation of the Water Framework Directive. European Environment 15(6):333-343. https://doi.org/10.1002/eet.398
Poff, N. L., C. M. Brown, T. E. Grantham, J. H. Matthews, M. A. Palmer, C. M. Spence, R. L. Wilby, M. Haasnoot, G. F. Mendoza, K. C. Dominique, and A. Baeza. 2016. Sustainable water management under future uncertainty with eco-engineering decision scaling. Nature Climate Change 6:25-34. https://doi.org/10.1038/nclimate2765
Polasky, S., S. R. Carpenter, C. Folke, and B. Keeler. 2011. Decision-making under great uncertainty: environmental management in an era of global change. Trends in Ecology & Evolution 26(8):398-404. https://doi.org/10.1016/j.tree.2011.04.007
Pulver, S., N. Ulibarri, K. L. Sobocinski, S. M. Alexander, M. L. Johnson, P. F. McCord, and J. Dell’Angelo. 2018. Frontiers in socio-environmental research: components, connections, scale, and context. Ecology and Society 23(3):23. https://doi.org/10.5751/ES-10280-230323
Raadgever, G. T., C. Dieperink, P. P. J. Driessen, A. A. H. Smit, and H. F. M. W. van Rijswick. 2011. Uncertainty management strategies: lessons from the regional implementation of the Water Framework Directive in the Netherlands. Environmental Science & Policy 14(1):64-75. https://doi.org/10.1016/j.envsci.2010.11.001
Rittel, H. W. J., and M. M. Webber. 1973. Dilemmas in a general theory of planning. Policy Sciences 4(2):155-169. https://doi.org/10.1007/BF01405730
Roberts, J. 2013. Organizational ignorance: towards a managerial perspective on the unknown. Management Learning 44(3):215-236. https://doi.org/10.1177/1350507612443208
Scott, T. 2015. Does collaboration make any difference? Linking collaborative governance to environmental outcomes. Journal of Policy Analysis and Management 34(3):537-566. https://doi.org/10.1002/pam.21836
Scott, T. A., N. Ulibarri, and R. P. Scott. 2018. Stakeholder involvement in collaborative regulatory processes: using automated coding to track attendance and actions. Regulation & Governance, in press. https://doi.org/10.1111/rego.12199
Sigel, K., B. Klauer, and C. Pahl-Wostl. 2010. Conceptualising uncertainty in environmental decision-making: the example of the EU water framework directive. Ecological Economics 69(3):502-510. https://doi.org/10.1016/j.ecolecon.2009.11.012
Smithson, M. 2008. The many faces and masks of uncertainty. Page 13-26 in G. Bammer and M. Smithson, editors. Uncertainty and risk: multidisciplinary perspectives. Routledge, London, UK.
Steelman, T. A., B. Nowell, D. Bayoumi, and S. McCaffrey. 2014. Understanding information exchange during disaster response: methodological insights from infocentric analysis. Administration & Society 46(6):707-743. https://doi.org/10.1177/0095399712469198
Stelzenmüller, V., T. Vega Fernández, K. Cronin, C. Röckmann, M. Pantazi, J. Vanaverbeke, T. Stamford, K. Hostens, E. Pecceu, S. Degraer, L. Buhl-Mortensen, J. Carlström, I. Galparsoro, K. Johnson, J. Piwowarczyk, V. Vassilopoulou, R. Jak, M. L. Pace, and L. van Hoof. 2015. Assessing uncertainty associated with the monitoring and evaluation of spatially managed areas. Marine Policy 51:151-162. https://doi.org/10.1016/j.marpol.2014.08.001
Tickner, J. A. 1999. A map toward precautionary decision making. Pages 162-186 in C. Reffensperger and J. A. Tickner, editors. Protecting public health and the environment: implementing the precautionary principle. Island, Washington, D.C., USA.
Ulibarri, N. 2015a. Collaboration in federal hydropower licensing: impacts on process, outputs, and outcomes. Public Performance & Management Review 38(4):578-606. https://doi.org/10.1080/15309576.2015.1031004
Ulibarri, N. 2015b. Tracing process to performance of collaborative governance: a comparative case study of federal hydropower licensing. Policy Studies Journal 43(2):283-308. https://doi.org/10.1111/psj.12096
Ulibarri, N. 2018a. Collaborative model development increases trust in and use of scientific information in environmental decision-making. Environmental Science & Policy 82:136-142. https://doi.org/10.1016/j.envsci.2018.01.022
Ulibarri, N. 2018b. Does collaboration affect the duration of environmental permitting processes? Journal of Environmental Planning and Management 61(4):617-634. https://doi.org/10.1080/09640568.2017.1327845
Walker, W. E., P. Harremoës, J. Rotmans, J. P. van der Sluijs, M. B. A. van Asselt, P. Janssen, and M. P. Kraver von Krauss. 2003. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integrated Assessment 4(1):5-17. https://doi.org/10.1076/iaij.188.8.131.5266
Yin, R. K. 2009. Case study research: design and methods. Fifth edition. Sage, Thousand Oaks, California, USA.
York, A. M., and M. L. Schoon. 2011. Collective action on the western range: coping with external and internal threats. International Journal of the Commons 5(2):388-409. https://doi.org/10.18352/ijc.286
Zandvoort, M., R. van der Brugge, M. J. van der Vlist, and A. van den Brink. 2019. Dealing with uncertainty in collaborative planning: developing adaptive strategies for the IJsselmeer. Journal of Environmental Planning and Management 62(2):248-265. https://doi.org/10.1080/09640568.2017.1409196