|Home | Archives | About | Login | Submissions | Notify | Contact | Search|
Copyright © 2004 by the author(s). Published here under license by The Resilience Alliance.
Go to the pdf version of this article.
The following is the established format for referencing this article:
Failing, L., G. Horn, and P. Higgins. 2004. Using expert judgment and stakeholder values to evaluate adaptive management options. Ecology and Society 9(1): 13. [online] URL: http://www.ecologyandsociety.org/vol9/iss1/art13/
A version of this article in which text, figures, tables, and appendices are separate files may be found by following this link.
Synthesis Using Expert Judgment and Stakeholder Values to Evaluate Adaptive Management Options Lee Failing1, Graham Horn, and Paul Higgins
1Compass Resource Management
This paper provides an example of a practical integration of probabilistic policy analysis and multi-stakeholder decision methods at a hydroelectric facility in British Columbia, Canada. A structured decision-making framework utilizing the probabilistic judgments of experts, a decision tree, and a Monte Carlo simulation provided insight to a decision to implement an experimental flow release program. The technical evaluation of the expected costs and benefits of the program were integrated into the multi-stakeholder decision process. The framework assessed the magnitude of the uncertainty, its potential to affect water management decisions, the predictive ability of the experiment, the value of the expected costs and benefits, and the preferences of stakeholders for alternative outcomes. As a result of the analysis, the initial experimental design was revised, and a multi-stakeholder group reached consensus on a program of experimental flow releases to test the response of salmonids to flow. The approach treats adaptive management as a policy alternative within a broader decision problem, and it demonstrates the utility of combining expert judgment processes and stakeholder values with adaptive management to improve the likelihood that proposed experimental approaches will deliver net value to society.
KEY WORDS: adaptive management, decision analysis, expert judgment, hydroelectricity, multi-attribute evaluation, multi-stakeholder consultation, risk management, value of information.
Published: May 19, 2004
Much has been written about risk-based stakeholder decision processes. In its 1996 report, Understanding Risk, the National Academy of Sciences highlighted the roles of both scientific and value-based input for effective risk management, and identified the need to find the right balance between analysis and deliberation in risk-based decision making (NAS 1996). Research on participatory methods for risk management decision making has flourished; however, tension between science and values and uncertainty about their roles in decision making persist. Scientists, as well as a variety of resource management agencies and industrial interests, remain concerned that the requirement for deliberative stakeholder processes will compromise the integrity and importance of science as a tool for risk management; stakeholders fear that scientific knowledge gives technical experts unequal power and the ability to manipulate facts to support a particular point of view (Charnley 2000). There is also concern, backed by evidence from behavioral research (Slovic et al. 1977, Keeney 1992, Gregory et al. 1993), that an emphasis on science and technical analysis, to the exclusion of value-focused thinking and structured decision aiding, will compromise the quality of decisions.
Of particular interest in the effort to address complexity and uncertainty in the management of natural resources is a trend toward adaptive management (AM) (Walters 1986). Rather than using existing knowledge and predictive models to select a single “best” plan, an AM approach explicitly recognizes the existence of uncertainty, documents hypotheses about the response of ecological systems to management intervention, monitors actual responses, and adjusts management actions over time. Typically, an adaptive approach emphasizes the use of simpler rather than elaborate models, followed by an update of the model based on observation and learning. “Active” AM involves planned experimental manipulation of a system, using either concurrent or sequential trials, accompanied by comprehensive monitoring and hypothesis testing. Adoption of a management strategy occurs only after monitoring results have confirmed which policy alternative is better.
Although the concept of AM is highly appealing, and the approach is at the forefront of modern ecological science, the case list of successful applications remains small. A review article by Walters (1997), for example, cites only seven examples of AM that have resulted in relatively large-scale management experiments. Walters and others (for example, Halbert 1993) note a variety of technical issues that complicate implementation. Furthermore, the social and institutional challenges are formidable. Although there are some excellent examples of thoughtful approaches to assessing the potential for AM and the value of ecological research (Sainsbury 1991, Walters and Green 1997, Kim et al. 2003, Walters et al. 2000), there appears to be relatively little experience with methods for characterizing the costs and benefits of AM in terms that are meaningful for decision makers. Experimentation is seldom costless. Before investing in an experimental approach, prudent decision makers will ask many questions of scientists: What are the nature and bounds of possible outcomes? What is the probability associated with each outcome? Are some policy options more risky than others? What are the non-fish impacts? But scientists propose experimental approaches precisely because they cannot predict ecological responses to policy changes. As a result, there is tension between characterizing the likely impacts of policy options based on the best available information, and acknowledging the uncertainty inherent in any attempt to predict or bound responses.
This paper describes a value-of-information analysis conducted to assess the merits of a proposed AM program within the context of a multi-stakeholder, multi-attribute decision process. Our goals were a) to integrate the AM decision into the broader multi-stakeholder decision process, while still basing the experimental design and decisions on a defensible technical analysis; and b) to combine expert judgment and stakeholder values to characterize the probability, magnitude and desirability of gains under alternative flow regimes and experimental designs. It is our hope that the paper will contribute both to the literature and case experience on the integration of science, stakeholder processes, and decision making, as well as to methodological debates about approaches to evaluating investments in experimental AM.Adaptive Management and Structured Decision Processes
A growing body of literature outlines the benefits of structured decision processes, founded on the principles of decision analysis, as the basis for stakeholder deliberations on risk-based decision making (Gregory et al. 1993, Slovic and Gregory 1999, Maguire and Boiney 1994, Renn 1999, Arvai et al. 2001). In general, research and applications suggest that a structured value-focused process for aiding decisions increases the capacity of participants to make informed and broadly accepted decisions in the context of complex risk management problems.
A well-structured decision process can typically be summarized in three key steps (Keeney 1992, Clemen 1996):
Within this structured decision framework, detailed technical analysis on defined sub-problems can be conducted; we view AM as one of these detailed sub-problems. In contrast to most of the literature and practice on the subject, we treat AM not as an all-encompassing planning framework, but as as a policy alternative. In this context, an AM program, like any other management alternative, must be justified on the basis of its costs and benefits relative to other alternatives, including non-experimental alternatives, alternative experimental designs, and other strategies for gaining information. It must be subject to an open debate in a multi-stakeholder process, in which trade-offs and risks—biological risk, financial risk and scientific risk (e.g., of an inconclusive experiment)—are exposed and discussed.The Lower Bridge River
The Lower Bridge River is a significant salmon and hydroelectric power-producing tributary of the Fraser River in southern British Columbia. It is part of BC Hydro’s Bridge-Seton facilities, which consist of three impoundment dams, three reservoirs, and four generating stations, with the capacity to produce about 3000 gigawatt hours annually. When the Terzaghi Dam was constructed, no continuous releases from Carpenter Reservoir were required for the protection of fisheries resources, thus a 4-km section of the river channel immediately below the dam was left essentially dry, with inflows to sections downstream derived exclusively from groundwater and inflow from tributaries. The remaining 15 km of the river has experienced a more than hundred-fold reduction in flow.
Beginning in the late 1980s, concerns about the lack of instream flow release from the Terzaghi Dam were expressed by fisheries regulatory agencies. In 1993, instream flow assessment studies were initiated by BC Hydro to help predict the response of salmonid populations and resolve the instream flow issues. Fish habitat use and physical habitat modeling studies were conducted from 1993 to 1995 to examine the effect of flow release on physical habitat conditions. However, these studies demonstrated that standard methods for instream flow assessment could only explain about 50% of the variation in fish density at baseflow levels, inspiring little confidence that habitat simulation modeling could be used to predict population response at different flow levels. It was concluded that physical habitat modeling and ecological monitoring are useful tools for understanding some of the factors regulating fish population response to flow. However, not surprisingly (see Hilborn and Walters 1981), they do not provide sufficient information to make reliable decisions about the long-term flow release policy from the reservoir.
After 8 years of litigation and research, an out-of-court settlement resulted in provision of a 3 m3 per second (cms) water budget per year for instream flow releases in perpetuity or until such time as a Water Use Plan was authorized by the Comptroller of Water Rights for the province of British Columbia and agreed to by the Canadian Department of Fisheries and Oceans. The water budget was negotiated, representing a compromise between the instream flow recommendations forwarded by BC Hydro and the fisheries regulatory agencies. It was recognized that there was considerable uncertainty about expected ecological response and accordingly it was specified that, at a minimum, follow-up monitoring was required to determine the biological benefits of the 3 cms release and, also, that the benefits of an AM flow experiment be explored through the Water Use Planning process.Competing Hypotheses
Discussions among biologists and fisheries managers have produced a common conceptual model for describing how instream flow may affect the Bridge River salmonid population. Annual recruitment to the adult population is the number of smolts (S) leaving the Bridge River and is a function of the quantity of wetted area of channel available for rearing and the quality of the habitat:
S = f(habitat area, habitat quality)
There is little uncertainty or disagreement about the effects of increased flows on the quantity of wetted channel area. Topographic channel surveys and hydraulic modeling have been used to make predictions about the change in wetted habitat area. Wetted area changes result from increased river length caused by the rewatering of the first 4 km of river channel below the Terzaghi Dam and from incremental changes in wetted width over the channel length. However, there is considerable uncertainty about how different flow release strategies will affect the productive capacity of the habitat. Reservoir releases are expected to reduce mean water temperature and clarity, affecting primary and secondary productivity, as well as fish behavior, foraging, and growth. Increased river flows may also affect the hydraulic suitability of the habitat for juvenile fish. The Bridge River channel is relatively steep and confined over its length, so it is expected that increased flows will result in increased velocity; however, the effects of this on fish habitat use are unknown.
The functional relationship between reservoir release and salmonid recruitment can be represented by the relation:
where S is relative productivity of salmonids under different reservoir release (Q) conditions to be described by different parameter combinations (a,b) with log normal distributed random variation w. This model provides a flexible functional relationship that can accommodate a wide range of possible forms (linear, power, and quadratic) and specification of prior assumptions about the “optimum flow” level for the river and the total number of smolts that would be produced under that flow condition, depending on specific parameter combinations. Habitat assessment and ecological monitoring studies conducted for Bridge River instream flow research have been insufficient to provide quantitative assessment of the likelihood of a given parameter set, however, available data can be used to develop an envelope of hypotheses about how reservoir releases would influence salmonid recruitment. In generalized terms, these hypotheses are: “High Good”—high flows are better for fish, and “Low Good”—low flows are better for fish.
The “High Good” hypothesis is that juvenile salmonid productivity is directly proportional to the wetted area of the channel. The underlying assumption of this model is that, within the range of flow releases under consideration, habitat quality is independent of the magnitude of water releases from the reservoir and that there is a direct linear relationship between wetted habitat area and smolt production (i.e., the conventional “more flow = more fish” hypothesis). However, as the relationship between wetted area and total river discharge is controlled by the hydraulic geometry of the channel, the form of the functional relationship between reservoir release and smolt production is expected to be curvilinear. To estimate the functional relationship between reservoir release and smolt production under this extreme hypothesis, hydraulic simulation results can be used to quantify the relationship between wetted area and total river discharge. Fish sampling data from Bridge River can be used to quantify smolts produced per square meter of habitat under the baseline conditions, and these data can be extrapolated to estimate smolts produced under different flow releases. These predictions can then be used to estimate the parameters for the functional relationship between reservoir release and smolt production.
The “Low Good” hypothesis is that reservoir releases adversely affect the quality of habitat. Physical habitat simulation studies conducted as part of instream flow assessment research suggest that reservoir releases greater than approximately 1 cms will considerably reduce the hydraulic suitability of habitat for fry and parr. Further reductions in habitat quality may result from possible trophic impacts associated with alterations to the river temperature regime. Steady-state plug flow mixing models have been used to estimate how temperature regime will be altered from hypolimnetic reservoir releases and to predict a “thermal inversion” of the downstream temperature regime, where growing season temperatures (i.e., May through August) will be reduced on average by 2°C and fall/early winter temperatures will be increased by 2°C. Under this hypothesis, additional flows increase the wetted area of river channel, but additional flow is detrimental to fish habitat quality and smolt production (i.e., the controversial “more flow ≠ more fish” hypothesis). In the most pessimistic scenario, wetted habitat area gains are outweighed by reductions in habitat quality so that any increase in flow results in a reduction in smolt production.The Bridge-Seton Water Use Plan
British Columbia’s Water Use Plan (WUP) process is designed to review and, as required, develop new water use plans for approximately 20 major hydroelectric facilities in the province. It recognizes that these facilities, historically operated almost exclusively for the purposes of generating hydroelectric power, also affect other interests, principally flood protection, fish, wildlife, recreation, and First Nations culture and heritage. Basic to the WUP process are site-specific consultations with involved parties that link the values and concerns of participating public, aboriginal (First Nation), and agency stakeholders with information from technical experts about the anticipated consequences of different management plans. The WUP Guidelines (http://lwbc.bc.ca/water/wup/) identify AM as one of the key principles of the program, used for providing flexible and responsive water management approaches over time.
Following the core steps of a structured decision process, the consultative committee for the Bridge-Seton WUP developed objectives and performance measures for all parts of the system, identified information gaps and conducted studies to address them, and then identified and evaluated alternative operating strategies against the objectives and performance measures (BC Hydro 2003). Reservoir management alternatives and instream flow alternatives were developed in all parts of the system. In the remainder of this paper, we focus only on the instream flow alternatives for the Lower Bridge River (LBR).
An experimental flow release program on the LBR was identified as one management alternative (or set of alternatives, given different experimental design possibilities). Consideration of an experimental flow release program is based on the existence of the two generalized hypotheses described above. Although the parties involved in regulatory negotiations had agreed in principle to an AM approach, many participants in the WUP remained unconvinced of the merits of experimentation, especially when they realized that committing to experimentation meant foregoing other types of habitat enhancements because of concern about confounding effects. Furthermore, an agreement in principle to develop an AM program does not answer the question of which flow treatments to implement, over what time period, and at what cost. Therefore, the multi-party WUP consultative committee directed one of its working groups, the Fisheries Technical Committee (comprising fisheries scientists, managers and local and First Nations experts), to explore the merits of an experimental flow release program designed to provide better information on the actual impact of flows on fish, and compare it with non-experimental flows. Flow treatments evaluated by the WUP process included:
Assessing the cost of the experimental program is complicated by the fact that the response of decision makers to new information is unknown and, as a result, the flow regime that will ultimately be selected is unknown. Assessing the benefits is complicated by the fact that the fish response to flow is unknown. Therefore, the framework used to assess costs and benefits is a probabilistic one, based on expert judgments and “expected” values.
In setting up the analytical framework, we identified several elements that we considered critical for ensuring both a defensible technical analysis and effective integration into the stakeholder decision process.
Impacts must be presented in a simple framework that exposes key bottom-line trade-offs. An important distinguishing feature of the approach was the decision to report fisheries benefits in natural units, and to report the trade-offs between financial costs and fisheries benefits in the simplest possible way. We rejected the use of a utility function as unwieldy and insufficiently transparent in a multi-stakeholder environment. To make trade-off analysis manageable, it was necessary to reach agreement on a performance measure that could serve as a useful proxy for multiple aquatic ecosystem benefits.
Impacts must be expressed in probabilistic terms. The costs and benefits of alternative policies, whether experimental or not, and regardless of the amount of uncertainty, must be characterized. As noted in Walters and Green (1997), the expectation of surprise and the fact that the actual fish response is likely to be “none of the above”, should not prevent a structured review of the hypotheses by qualified scientists to ensure that it is at least plausible that the test policies will lead to better results. The presence of significant uncertainty requires that the characterization be probabilistic (Morgan and Henrion 1990). Methods for eliciting probabilistic judgments must follow accepted methods to minimize judgmental biases.
The value of information must be reported as improvements in expected future performance. From a societal perspective, knowledge is a means to more fundamental ends, not an end in itself. Decision makers need to know whether the probability of realizing ecological or other gains has increased or decreased as a result of improved knowledge, and whether the magnitude of those gains exceeds the costs of acquiring the knowledge. It follows that the value of AM should be stated, not in terms of greater knowledge or reduced uncertainty, but in changes in the “expected” value of more fundamental indicators of performance—usually ecological gains, financial costs and, in some cases, others such as recreational use, wildlife habitat, or aesthetics. Statistical measures (e.g., statistical power analyses), although informative, do not constitute a sound rationale for evaluating or selecting experimental designs (Walters and Green 1997, MacGregor et al. 2002).
Decision makers must provide value-based input about whether benefits outweigh costs. Unless costs and other trade-offs are negligible, the existence of large uncertainty alone is not sufficient justification for conducting experimental trials. The evaluation process must directly ask the question: Is it worth it? Given the best possible characterization of the probability and magnitude of different outcomes, is the investment in the search for ecological or other benefits worthwhile? This will involve asking decision makers to provide value-based judgments about the magnitude of benefits required to offset the costs, as well as about trade-offs across time, as there is the potential for incurring significant financial costs early, with ecological benefits accruing only much later in time.
On the LBR, stakeholders began by clarifying the objectives and defining suitable attributes for summarizing the benefits. The costs of alternative water management policies in this case are the financial costs associated with a) the release of water (which, if released, is not available for generation) and b) the monitoring. The benefits are the expected change in juvenile salmonid biomass and related instream/ecosystem benefits for which this measure is a proxy. In setting these attributes, decision makers distinguished between the evaluation criteria that would be used to select flow regimes worth testing, and the evaluation criteria that would be needed to evaluate and select a preferred flow regime post-implementation. The latter included additional concerns about recreational and aesthetic quality and wildlife habitat. These concerns influenced the inclusion of monitoring studies to provide information about the effect of alternative flow regimes with respect to aesthetics and wildlife, but were not seen as essential components of an a priori analysis, as the probability of irreversible effects arising from any of the proposed experimental flows was negligible. This distinction helped simplify the analysis.
Once the objectives were defined, the analysis focused on characterizing the impacts of alternative policies on the objectives. Decision makers had the option of either selecting a single non-experimental flow release (where non-experimental refers to selecting a single fixed flow based on current information and accepting the uncertainty in possible outcomes) or selecting an experimental flow program (for which several possible designs involving different levels of investment and information quality were possible). To determine the value of an experimental approach, four questions were considered: How great is the uncertainty about the benefits? Does the experiment have sufficient predictive ability to reduce the uncertainty? Across a plausible range of values, does the uncertainty have the potential to affect a management decision? Do the expected benefits outweigh the costs of the experiment? The first two required technical judgments from appropriate experts, the latter two required value judgments from stakeholders. Our method for addressing these questions was as follows:
Two experts (one a senior fisheries scientist with the Canadian Department of Fisheries and Oceans, the other a senior fisheries biologist for BC Hydro) were asked to provide a conditional probabilistic estimate of biomass production at four different flow levels (e.g., 1, 3, 6, and 9 cms) under each competing hypothesis. To begin, an initial scoping meeting was held with both experts. At this meeting, we clarified the reasons for the analysis and how the results would be used. We agreed on an appropriate metric to represent the benefits of alternative flow releases (juvenile salmonid biomass measured as fall standing stock and integrated over reaches one through four), and how it should be interpreted (as a proxy for multiple instream benefits). The metric was reviewed with stakeholders to ensure that it addressed their concerns before it was incorporated in the analysis.
The experts also agreed on a set of data and background information that would be relevant in making the judgments. These included: existing data on the relationship between flow and wetted area, results from physical habitat simulation modeling that predict a relationship between flow and various habitat types, baseline data on juvenile salmonid biomass at zero discharge, and a meta-analysis that reports salmonid biomass density across a large sample of British Columbia streams. At this meeting, the two competing hypotheses were also defined (“High Good” and “Low Good”, as above).
In this case, judgments were made directly about the endpoint of interest—juvenile salmonid biomass. (In contrast, see Kim et al. (2003) in which experts were asked to provide subjective probabilities for five parameter values that were the inputs of an integrated lower trophic level productivity model; the model then produced estimates of higher level endpoints of interest). Both experts used the conceptual model described above. They used hydraulic simulation results to quantify the relationship between total river discharge and wetted area. Smolt enumeration data from Bridge River were used to provide a quantitative estimate of smolts produced per square meter of habitat under the baseline conditions. From this common starting point, individual judgments about likely biomass under different flow releases were made by each expert.
Both experts were familiar with probabilistic analysis and did not require training, but basic methods for bias avoidance (such as motivation bias, availability bias, anchoring and adjustment, and over-confidence) were addressed (Morgan and Henrion 1990). Initially, they made their judgments independently. Then they reviewed each other’s approach and were given the opportunity to modify their judgments. Interestingly, both experts were initially reluctant to provide a probability distribution, as they felt such a specification would give a false impression of precision. As a first step, we asked that they provide their “best guess” for juvenile biomass—a value for which they thought there was an equal probability that the value would be above and below. As we expected, they had low confidence in the results, and felt (appropriately) uncomfortable about how these results might be used in decision making. They were subsequently asked to provide two additional values for each parameter: 5th and 95th percentiles. This resulted in the designation of a confidence or belief interval that the experts were comfortable with: each expert was 90% confident that the true value would fall within the stated bounds.
Results of the elicitations for Expert 1 are shown in Fig. 1. Under the “High Good” hypothesis, Expert 1 estimates biomass could rise continuously from about 640 kg at 0 cms (measured baseline conditions at zero discharge). Expert 1 believes there is less than a 5% probability that the biomass value at 9 cms will exceed about 1700 kg or that it will be less than about 900 kg. This upper bound is the result of the existence of limiting factors other than flow, such as the cold, turbid water and canyon-like characteristics of the river. Under the “Low Good” hypothesis, Expert 1 suggests a peak in biomass at 1 cms, before dropping steadily at higher flows. The lower bound on this estimate represents the extreme case, where the introduction of any flow at all results in a net loss of biomass. A peak at 1 cms is consistent with physical habitat modeling results in the river. Expert 2’s judgments (not shown) follow a similar pattern, the only notable exception being a higher peak on the “Low Good” estimate at 1 cms, and a higher peak on the “High Good” estimate at 9 cms.
In our case, the uncertainty is relatively small across experts, but reasonably large across the hypotheses, particularly for some policy options. Therefore, we use a simple aggregation rule (Morgan and Henrion 1990, Clemen and Winkler 1999). Judgments were aggregated using equal weights, but the extreme outer bounds of the individual judgments continued to be represented on the results presented to the stakeholder group.Judgments about the Hypotheses and the Experimental Design
The experts were then asked to attach a probability to each hypothesis or state of Nature (P[N]), and indicate the likelihood that the eXperiment would predict the correct state of nature (P[X|N]) (see Tables 1 and 2). In assigning probabilities, the experts considered the estimated natural variability and the estimated detectable effect size for biomass changes, and the extent to which secondary monitoring indicators (e.g., trophic responses, etc.) would support inferences about changes in salmonid biomass under each flow treatment.
Using these inputs, two additional probabilities were calculated: the probability of a given eXperimental outcome (P[X]), and the probability of a given hypothesis (or state of Nature) being true given a certain eXperimental outcome (P[N|X]). (Methods for deriving these probabilities can be found in standard texts on Bayesian statistics or decision analysis; for example, see Clemen (1996).) These two probabilities have a significant impact on the value of an experiment. For example, if the state of nature is fairly certain (i.e., P[N] is high) and the experiment is known to be fairly unreliable (i.e., P[N|X] is low), then the experiment will have little value. However, if the state of nature is very uncertain and the experiment is very reliable, then the experiment will have high value. But even then, whether the experiment offers sufficient value to justify proceeding depends on the value at stake and the cost of the experiment. These issues are discussed further below.Calculation of Benefits and Costs
Using the above probabilities and biomass estimates, the high, low, and expected benefits of each alternative for each expert were calculated. This involved setting up a simple decision tree that could be solved for biomass for a given decision maker and a given set of inputs from that decision maker (Fig. 2). The non-experimental alternatives (i.e., 1, 3, 6 or 9 cms), as well as the experimental alternative, were represented in the tree. The figures in the uncertainty nodes (ovals) are expected biomass, and those in the decision nodes are the expected biomass associated with the preferred alternative (see methods for solving decision trees in standard decision analysis texts such as Clemen (1996). The water and monitoring costs were combined and represented as a string of annual costs for each alternative. This string was then levelized and entered into the tree in order to compute the range of annual costs associated with each option.
Up to this point, the six separate judgments from each expert have been treated individually in the analysis (these being, for each hypothesis: the probability of it being true, the biomass response if is true, and the probability that the experiment will correctly identify it as the true state of nature). In order to calculate a single expected benefit for each alternative, a monte carlo simulation was used to combine logical combinations of inputs (i.e., probabilities and biomass levels) as specified by the different experts.
As expected, the simulation resulted in the same extreme range of benefits as when the judgments were treated individually, but with a narrower 90% confidence band. For example, in the case of the 1 cms option (the histogram is shown in Fig. 3 for illustration purposes), biomass could range from as low as about 400 kg to as high as 1500 kg, while 90% of the values (for both experts and both hypotheses combined) fell between about 600 and 1200 kg of biomass. The 6 and 9 cms fixed flows show a similar proportional narrowing, although the effect is less for the 3 cms flow as it had a narrower distribution to start with.
The expected values and confidence bands for all alternatives were presented in a simple two-way chart that showed both the extreme range of costs and benefits and the 90% confidence band (Fig. 4).
From Fig. 4, four key observations are:
The results and the value judgments of decision makers were used in three ways.
To modify the experimental design. In the original experimental design, the results of the first treatment(s) determined the magnitude of subsequent test flows. However, this design could fail to test 1cms under some conditions. For example, a biomass increase at 6 cms would trigger the next treatment at 9 cms, omitting 1 cms. However, scientists considered it plausible that the existence of a biomass peak or increase at 6 cms did not preclude the existence of another peak at 1 cms. The analysis exposed the potential win-win at 1 cms (high biomass at low cost), and influenced stakeholders, including scientific experts, to modify the trial design so that testing of 1 cms was guaranteed.
Stakeholders also evaluated a “stopping rule” proposal that would give a post-implementation management committee the authority to halt the trials after the 1 cms treatment under certain biomass outcomes. The rationale was that if a large increase in biomass were observed at 1 cms, then a) the probability of realizing an even higher increase in biomass at higher flows would be reduced, and b) stakeholders would be satisfied with the ecological performance of the LBR relative to other British Columbia streams. The stopping rule proposal was viewed favorably by those concerned about the proposed duration and associated costs and operational uncertainties of the experimental program. It was viewed unfavorably by those focused on the learning objective of documenting the functional relationship between fish and flow. Although not adopted in this case, this kind of proactive values elicitation might be useful in reaching agreement when the cost or institutional barriers to experimentation are particularly high.
To defer costly test flows with a low probability of changing water management decisions. On the basis of the results in Fig. 4, stakeholders rejected the 9 cms option as too costly, given the uncertainty of the benefits at this time. Although they did not reject the possibility of testing this flow in the future, they considered it premature to commit to test it now. In addition to financial considerations, stakeholders felt that a 9 cms flow could have unacceptable consequences for riparian vegetation and wildlife. In addition, given the duration of the trials, the first three test flows would take 11 years to complete. By then, stakeholders concluded that there might be new information about biological impacts, significant changes in the value of power, and changes in the trade-offs that people are willing to make between power and ecological benefits. In sum, based on current information, the probability that a 9 cms flow could be preferred seemed low. Conversely, stakeholders confirmed that, across the test range of 1–6 cms, given the range of costs and benefits shown in Fig. 4, it is plausible that the preferred alternative lies in this range, and thus that the information from the test flows has a high probability of affecting future water management decisions. Thus, incorporation of stakeholder values in addition to scientific input resulted in a phased decision process, involving stakeholder review of the results of the 1, 3 and 6 cms treatments prior to a decision to test a 9 cms release. The combined effect of deferring the 9 cms flow and changing the experimental design decreased the expected cost of the program from $4.9 million to $3.5 million.
To compare experimental alternatives with non-experimental alternatives. The analysis allowed comparison of the incremental costs and benefits of the experiment with the non-experimental options (selecting a single fixed flow on the basis of current information), and explicitly asked “is the investment in reducing uncertainty worth it?” In the absence of an experimental alternative, the analysis demonstrated the dominance of the 1 cms flow. This flow regime has a higher expected value, a narrower risk profile, and could be achieved at least cost. In the end, most stakeholders indicated that they could support adoption of either a 1 or 3 cms release (without experimentation). There was less support for 6 cms, and none for 9 cms. However, all stakeholders strongly supported the experimental program as their preferred flow regime, because the incremental benefits were seen to far outweigh the incremental costs, whether compared with the 1 or 3 cms flow. Stakeholders also considered the benefits of institutional learning and cooperation that would result from an unprecedented level of collaboration among agencies, the power utility and the Stl’atl’imx Nation, to be significant.
The core of the approach involves asking experts to provide four explicit judgments:
Furthermore, it requires that decision makers explicitly address the question “Is it plausible that water management decisions could change, given the estimated costs and the potential range of benefits across the test flows?”
There are many approaches to answering these questions. Whether they are answered qualitatively or quantitatively, or exactly which methods of quantification are used are less important than whether the questions are answered explicitly. Finding a useful approach involves finding an optimal balance between technical accuracy, understandability to decision makers and stakeholders, and the ability to deliver timely and cost-effective results. Our method, like any other, has both strengths and limitations.
One limitation in our methodology was the use of only two experts, particularly experts who had worked closely together and shared a similar conceptual model of the system. This may have resulted in an understatement of uncertainty. We believe that the approach was adequate to expose key insights and improve the decision. Nonetheless, it would have been preferable methodologically to use three to five experts, and to explore different models of ecosystem processes.
Other methodological refinements that could be warranted for problems with higher stakes and more information include eliciting full distributions for uncertain variables/outcomes, specifying correlations between variables, and explicitly modeling the decision rule (rather than assuming flow decisions will be based on highest biomass alone). However, in our opinion, none of these refinements were warranted for this problem. The specification of a 90% confidence interval was valuable for bounding the expectations of stakeholders (some of whom were imagining tenfold increases in biomass under high test flows). It was also a cognitively reasonable task to demand of the experts, who had access to both a meta-analysis summarizing biomass density in comparable British Columbia streams and detailed data and modeling of the specific conditions of the LBR to guide their judgments. Little would be gained by eliciting a full distribution, both because it would have overstepped expert knowledge, and it would have added little insight to the relative benefits of the options. We also consider it futile to model a future decision process in detail, given that we cannot even identify all the factors that will affect it, let alone estimate them.
In total, this analysis required about a week of analytical time, a day of “expert” time, and two meetings with the fisheries technical working group prior to its presentation to the broader stakeholder group. The models used are relatively simple spreadsheet models, and we used simplifying assumptions wherever possible. The goal was not quantitative precision, but insight. Deliberations focused on the results, as presented in Fig. 4. To explain the experimental results, we reminded people of the individual judgments that led to the Fig. 4 results (e.g., biomass estimates, probability of each hypothesis, probability the experiment is right) and showed how those judgments influenced the summary results. We did not spend time describing the decision tree or monte carlo calculations in detail. Participants accepted these methods as generally applicable, and focused their attention on understanding and challenging the initial judgments and the Fig. 4 results.
We believe that the explicit and quantitative estimates of biomass response that were elicited from experts provided critical bounds to the benefits of the test flows. This information clearly influenced the outcome of the decision process by exposing the low probability that both experts assigned to the large gains that were being suggested by some advocates of higher flows. Qualitative estimates of biomass response (high, medium, low) are vastly inferior, as they place no bounds on the range of benefits; research has shown that such terms are interpreted very differently by different people (Lichtenstein and Newman 1967). Unless experimentation is costless, bounds matter. Thus, we believe the quantification of benefits, in the sense of bounding them within a defined confidence interval, is important.
In this application, we also quantified the expected value of the experiment. In essence, the approach integrates three judgments: the biomass response under each hypothesis, the probability of each hypothesis being true, and the probability that the experiment correctly discriminates among the hypotheses. It is appealing from a technical perspective, as it accurately integrates relevant judgments into a single performance metric. From a decision making perspective, it is appealing because it is sensitive to changes in the level of investment in experimental methods (e.g., it is possible to quantify the benefits associated with greater degrees of reliability in the experiment). It was also reasonably practical; given our simplifications, it was not hard to develop or use the model. However, there is a fine line between a useful level of analysis that is timely and informative, and an overly technical analysis that is time-consuming and confusing. It is arguable that expected value calculations are unfamiliar and cognitively difficult and in some cases could prove distracting. However, the alternative—multiple sets of biomass estimates under competing hypotheses as estimated by different experts, along with additional indicators for “learning potential”—is also cognitively challenging and, we would argue, more likely to distract people from the core messages.
The benefit of the approach we used is that it delivers a single metric of performance (expected biomass, or 90% confidence interval) that allows proposed experimental options to be compared with other management options—either other options for enhancing fish biomass at the same site (e.g., physical habitat enhancement) or options for enhancing fish or other ecological endpoints at other sites. Given the reality of scarce financial resources, investments in AM usually have an opportunity cost. In our case, the most serious challenge to experimentation stemmed from the reluctance of participants to embark on a long-term experimental program that would exclude the possibility of conducting habitat enhancements in the area, due to the chance of confounding the experiment. A comparison of the benefits of flow experiments with physical habitat enhancements would have been helpful. We suggest that the quantitative approach we have outlined may be particularly relevant for decision makers seeking a transparent and replicable method of allocating resources among competing proposals for a given environmental benefit (e.g., research, monitoring, experimental flows, fixed flows, habitat enhancement, etc.) and/or across multiple sites. In addition, the method provides a means of dealing with differences in expert opinion. In our case, our experts were reasonably well aligned; in other cases, the ability to deal with competing expert opinions is more critical.The Use and Benefits of Expert Judgment
The use of expert judgment in this case was controversial. In our view, expert judgments providing explicit and quantitative information about the probability and magnitude of the response of ecological variables under alternative policy options are indispensable. They improve decision quality in three important ways. First, they help to put technical and non-technical decision makers on an equal footing, providing all stakeholders with the same degree of technical understanding about the probability and magnitude of uncertain consequences, and the magnitude of the uncertainty, as viewed by a representative set of experts. Second, the use of structured and explicit expert judgments helps shift deliberations from positional to performance-based debates. In our case, before conducting the elicitations, the debate about which flow regimes to test was largely a positional exchange among stakeholders, with regulators rejecting test flows lower than 3 cms, and insisting on flows of 9 cms or higher. Finally, when expert judgments are made explicit, the distinction between technical judgments (“what is the expected biomass response?”) and value judgments (“are benefits worth the costs?”) is clarified. When the goal is to support informed deliberation among multiple technical experts, and among technical and non-technical participants, this distinction is critical. All these effects unequivocally aid decision quality.
It is important to ensure that all participants are clear about the intended use of their judgments and the decisions that could be made as a result. Using expert judgments to scope and refine proposed experimental designs is probably broadly accepted as a scientifically defensible practice. Using them to justify the selection of a non-experimental alternative is more controversial. Other research notes that there is a risk that judgmental techniques may become a substitute for science, and cautions against this (Morgan et al. 1984). We cannot deny that it is possible that participants in the process could have used the judgments to recommend a non-experimental alternative (or that the regulatory authority may yet override the recommendation of the stakeholder group and authorize only a single fixed flow). Although we agree that judgments should not replace science altogether, we believe that judgmental/probabilistic techniques can and should be used to prevent the irresponsible expenditure of large sums of money, particularly public money, on projects with low probability of delivering commensurate benefits. We are not the first to suggest this; Walters and Green (1997) suggest that it is nonsense to claim that economic evaluation should be avoided just because an unexpected outcome may occur. They suggest that broad debate and expert review of the hypotheses can help to ensure that obvious possibilities are not missed (e.g., our 1 cms potential win-win situation), no commitment is made to irreversible physical changes (e.g., in our case, deferral of the 9 cms treatment), and that there is some reasonable possibility that the flows tested will deliver benefits sufficient, according to stakeholder values, to offset the economic costs (e.g., again, in our case, the deferral of the 9 cms treatment).
A key question is when is there enough information to make a quantitative expert judgment elicitation meaningful? Among scientists, even among those who identify themselves as Bayesians, there are a range of opinions, with some supportive of the notion that an a priori probability distribution can always be specified and forms a useful starting point for analysis, and others who feel that in extremely data-poor situations, such specification is not useful (Bier et al. 1999). From our perspective, it is hard to conceive of decision situations that would not benefit from explicit specification. If a state of true ignorance exists, then a well-elicited probability distribution should reflect this, and it is important information for decision makers to have. In some cases (such as our LBR example), it will be possible to specify upper or lower bounds to potential consequences with some confidence; even in the absence of a fully specified probability distribution, such bounds can be useful for decision making.Other Benefits for the Stakeholder Process
One of the primary benefits of the approach we used is that it exposed the different risk profiles of the alternatives, allowing decision makers to exercise their value judgments and risk tolerances. Exploring the risk profile of non-experimental alternatives can be useful in the case where experimentation (or other information collection) is not possible or desirable (e.g., predictive ability too low, or institutional complexity/barriers too great), or for selecting an interim flow while baseline information is being collected.
The analysis also imposed a discipline on the scientific process to ensure that information is not sought for information’s sake; the plausible range of benefits and the probability of realizing those benefits must be quantified or at least bounded, and decision makers must decide whether it is plausible that future decisions may change as a result of the information (i.e., is it reasonable to believe a future stakeholder group would choose one of the test flows as a permanent flow, given its estimated cost and the range of possible benefits).
An unexpected result was that the approach highlighted the reality of residual uncertainty and focused managers on differences in the learning potential of the proposed experiment. In our example, some LBR participants were surprised at the magnitude of the expected benefits of experimentation (i.e., they were small, relative to their intuitive expectations). This was caused in part by implicit anchoring on a “perfect information” model; participants failed to mentally process the information about the probability of the experiment delivering wrong or inconclusive information—a tendency consistent with research in judgment and decision making (Slovic et al. 1977, McDaniels et al. 1999). In fact, the probability of drawing correct inferences from the experiment was particularly low for high releases, due to difficulty of sampling fish at high flows in turbid water; this led to relatively poor expected results for the 9 cms flow option. The analysis was useful in exposing residual uncertainty and, more importantly, focusing managers on differences in the probability of success across different flow treatments.
Finally, the analysis supported stakeholders in making difficult value judgments. For example, an initial “willingness to pay” question asked stakeholders to provide input on the minimum acceptable level of biomass increase that would just offset the cost of a higher flow release. The responses indicated that willingness to pay ranged from $90 to $35 000 per kilogram of biomass. This question was asked relatively early in the evaluation process and, based on the range of responses, we suspect suffered from lack of context. In contrast, when faced with making recommendations on actual management choices, stakeholders unanimously rejected the 9 cms flow at an implied expected cost of $1400/kg. The “willingness to pay” question was abstract and difficult to answer; for a few stakeholders it was informative, but for most it was either meaningless or even mildly offensive. From an analytical perspective, we could not justify using the results; participants had had insufficient context to prepare thoughtful answers that reflected their values. When essentially the same question was asked in the context of an actual management choice, it became a relevant and meaningful question, which participants were willing to deliberate over. As noted by others (Keeney 1992), what is “acceptable” to stakeholders depends, appropriately, on the nature of the alternatives and the trade-offs.The Role of AM in Decision Making
Adaptive management is intuitively appealing to many stakeholders because its fundamental principles are simple—we don’t know, we don’t want to guess, let’s try it and then (we think) we’ll know for sure. It is appealing because it suggests that uncertainty can be eliminated and future decisions will be the “right” ones. It is appealing to scientists because it will increase their knowledge about the system. It is appealing to resource managers because they will feel more comfortable making recommendations about the resources for which they are responsible. Yet, from a societal perspective, the knowledge and comfort gained by scientists is useful only if it can be reasonably expected to translate into tangible improvements in the endpoints that people care about—better ecological performance, lower cost, etc. In making those judgments, decision makers must consider the probability of successfully gaining better information (based on the predictive ability of the experiment) and the likely impact of that information on decisions and endpoints of concern, relative to the costs of acquiring the information (Fig. 5). Adaptive management may be a good investment if the probability of gaining useful information is high and the consequences of the information, in terms of expected changes in the endpoints, are high. If probability is high but consequences are low, there is a risk of collecting information that does not have management relevance, as it is unlikely to affect decisions. If probability is low but consequences are high, AM is wishful thinking and managers should consider either redesigning the experiment with more funding, seeking other ways to gain the information, or accepting the existing uncertainty. Decision makers must realistically compare the costs and benefits of AM against the costs and benefits of other management alternatives. Doing this requires that AM analyses be appropriately positioned in the decision process, and that the roles of scientists and decision makers be more clearly defined.
We believe that one reason for AM’s low success rate may be the tendency to use it as the organizing framework for decision making, into which other elements of the decision—multiple stakeholders, competing (non-ecological) objectives—take a secondary role. From this starting point, it is hard to reach the conclusion that AM is not justified, and harder still to decide what to do if AM is not justified. Widespread use of the term “adaptive management” has propagated various interpretations of its meaning and, consequently, there are only vague notions about what it is, what is required for it to be successful, and how it is applied in practice. More often than not, attempts to implement it have failed. This is likely not only due to ecological surprise or persistent institutional barriers (see Walters 1997) but also because the scale of application is too large and the array of issues is too broad or complex to make the application tractable. Given the few examples of successful implementation of large-scale AM programs to date, we believe AM may best be applied or focused on critical elements of the decision, rather than the entire problem. On the LBR, we found it useful to treat AM specifically as a policy alternative, to be compared with other policy alternatives (e.g., non-adaptive flow options, habitat enhancement options, other research programs).The Role of Science and Values
Investing in an AM experiment is like purchasing insurance or a lottery ticket. The value of the information from an experimental trial is either that it decreases the risk of a bad outcome (like an insurance policy), or it increases the chance of a windfall (like a lottery ticket). When we decide whether to buy life insurance and how much to spend on it, most of us think through both the magnitude and the probability of a bad outcome. If the financial consequences of an early death are high (e.g., leaving behind young children), we may be willing to invest in an expensive policy regardless of the probability of premature death. Probability likely matters more when the consequences are not catastrophic, and the costs are significant. If a lottery ticket costs a dollar, we may buy it without much thought. If it costs a hundred dollars, most of us will ask “what are the odds of winning?” And implicitly or explicitly, most of us will think about a stopping rule—how many lottery tickets should we buy? Or in the case of insurance, how much is enough? Although the specifics of the management problem are clearly different, we can think of no reason to adopt a different thought process for considering the merits of an AM experiment: how much we are willing to spend depends on both the magnitude of the potential consequences and the probability we assign to their occurrence. When the consequences are not severe or irreversible and the costs are non-zero, probability matters. Making these trade-offs is a value judgment. Unlike insurance policies and lottery tickets however, information about the magnitude and probability of outcomes in environmental risk management is not available to most decision makers, only to technical experts. The role of technical experts in environmental risk management is not to make the value judgments, but to present information about consequences and probabilities in a manner clear enough to allow decision makers to make them.
We draw three conclusions from this work. First, it is possible and useful to conduct quantitative probabilistic analysis and have it used constructively by non-technical stakeholders to inform a decision. The requirement to go through a stakeholder process should not prevent the conduct of appropriate analysis at the level warranted by the problem. If the results are presented in an intuitive and deliberative format, stakeholders can and will make use of probabilistic information in risk-based decision making. Many of the benefits we have identified can be realized through a structured but qualitative analysis. There are additional benefits to a quantitative analysis and, with sensible simplifying assumptions, such analysis need not be onerous.
Second, it is useful to treat AM (e.g., experimentation, monitoring) as one element of a structured decision process, rather than as a decision process itself. A decision to implement an AM program has opportunity costs. As a result, an investment in AM, must, like any other investment, be evaluated on the basis of its merits (expected costs and benefits) relative to other management options.
Finally, expert judgment can be used to evaluate the merits of AM. It complements and, in some cases, may replace experimental science. It imposes a discipline on scientists, provides equal access to information to all decision makers, supports performance-based deliberations, and will help rationalize responsible investments in information.
Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.
The authors are grateful to the fisheries biologists who provided their time for elicitations and review. We thank Ben Hobbs for significant contributions to the development of the evaluation framework and for reviewing an early version of the decision model. We also thank two anonymous reviewers for helpful comments that improved the quality of the manuscript. Any errors or omissions that remain are the responsibility of the authors.
Arvai, J. L., R. Gregory and T. L. McDaniels. 2001. Testing a structured decision approach: value-focused thinking for deliberative risk communication. Risk Analysis 21(6):1065–1076.
BC Hydro. 2003. Bridge River Water Use Plan Consultative Committee Report. ISBN 0-7726-4898-0.
Bier, V. M., Y. Y, Haimes, J. H. Lambert, N. C. Matalas, and R. Zimmerman. 1999. A survey of approaches for assessing and managing the risk of extremes. Risk Analysis 19(1):83–94.
Charnley, G. 2000. Democratic science: enhancing the role of science in stakeholder based management decision making. Report prepared for the American Industrial Health Council and the American Chemistry Council.
Clemen, R. T. 1996. Making hard decisions: an introduction to decision analysis. PWS-Kent, Boston, Massachussetts, USA.
Clemen, R. T., and R. L. Winkler. 1999. Combining probability distributions from experts in risk analysis. Risk Analysis 19(2):187–203.
Gregory, R. , S. Lichtenstein, and P. Slovic. 1993. Valuing environmental resources: a constructive approach. Journal of Risk and Uncertainty 7:177–197.
Halbert, C. L. 1993. How adaptive is adaptive management? Implementing adaptive management in Washington State and British Columbia. Reviews in Fisheries Science 1:261–283.
Hilborn, R., and C. J. Walters. 1981. Pitfalls of environmental baseline and process studies. Environmental Impact Assessment Review 2(3):265–278.
Keeney, R. L. 1992. Value-focused thinking: a path to creative decision-making. Harvard University Press, Boston, Massachussetts, USA.
Keeney, R. L., and D. von Winterfeldt. 1991. Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management 38(3):191–201.
Kim, J. B., B. F. Hobbs, and J. F. Koonce. 2003. Multicriteria Bayesian analysis of lower trophic uncertainties and value of research in Lake Erie. Human and Ecological Risk Assessment 9(4):1023–1057.
Lichtenstein, S., and R. Newman. 1967. Empirical scaling of common verbal phrases associated with numerical probabilities. Psychonomic Science 9:563–564.
McDaniels, T. L., R. S. Gregory, and D. Fields. 1999. Decision aiding, not dispute resolution: creating insights through structured environmental decisions. Risk Analysis 19:497–510.
MacGregor B. W., R. M. Peterman, B. J. Pyper, and M. J. Bradford. 2002. A decision analysis framework for comparing experimental designs of projects to enhance Pacific salmon. North American Journal of Fisheries Management 22:509–527.
Maguire L. A., and L. G. Boiney. 1994. Resolving environmental disputes: a framework for incorporating decision analysis and dispute resolution techniques. Journal of Environmental Management 42:31–48.
Merkhofer, M. W. 1987. Quantifying judgmental uncertainty: methodology, experiences and insight. IEEE Transactions on Systems, Man, and Cybernetics SMC-17:741–752.
Morgan M. G., and M. Henrion. 1990. Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, Cambridge, UK.
Morgan, M. G., S. C. Morris, M. Henrion, D. A. L. Amaral, and W. R. Rish. 1984. Technical uncertainty in quantitative policy analysis—a sulfur air pollution example. Risk Analysis 4(3):201–220.
National Research Council/National Academy of Sciences (NRC/NAS). 1996. Understanding risk. Informing decisions in a democratic society. National Academy Press. Washington, D. C., USA.
Renn, O. 1999. A model for an analytic-deliberative process in risk management. Environmental Science and Technology 33:3049–3055.
Sainsbury, K. J. 1991. Application of an experimental approach to management of a tropical multi-species fishery with highly uncertain dynamics. ICES Marine Science Symposia 193:301–320.
Slovic, P., and R. Gregory. 1999. Risk analysis, decision analysis and the social context for risk decision making. Pages in J. Shanteau, B. Mellers, and D. Schum, editors. Decision science and technology: reflections on the contributions of Ward Edwards. Kluwer Academic, Norwell, Massachussetts, USA.
Slovic, P., B. Fischhoff, and S. Lichtenstein. 1977. Behavioural decision theory. Annual Review of Psychology 28:1–39.
Walters, C.J. 1986. Adaptive management of renewable resources. Macmillan Publishing, New York, New York, USA.
Walters, C. J. 1997. Challenges in adaptive management of riparian and coastal ecosystems. Conservation Ecology 1(2):1. (online) URL: http://www.consecol.org/vol1/iss2/art1.
Walters, C. J., and R. Green. 1997. Valuation of experimental management options for ecological systems. Journal of Wildlife Management 61:987–1006.
Walters, C. J., and C. S. Holling. 1990. Large-scale management experiments and learning by doing. Ecology 71(6):2060–2068.
Walters, C. J., J. Korman, L. E. Stevens, and B. Gold. 2000. Ecosystem modeling for evaluation of adaptive management policies in the Grand Canyon. Conservation Ecology 4(2):1 [online] URL: http://www.consecol.org/vol4/iss2/art1.
Address of Correspondent:
Suite 200-1260 Hamilton Street,
Vancouver, British Columbia,
Canada, V6B 2S8
|Home | Archives | About | Login | Submissions | Notify | Contact | Search|