An appraisal of adaptive management planning and implementation in ecological restoration : case studies from the San Francisco Bay Delta , USA

Adaptive management has been defined and redefined in the context of natural resource management, yet there are few examples of its successful application in ecological restoration. Although the 2009 Delta Reform Act now legally requires adaptive management for all restoration efforts in the Sacramento-San Joaquin Delta, in California, USA, projects in this region still encounter problems with implementation. We used a comparative case study analysis to examine adaptive management planning and implementation both in and around the Delta, assessing not only why adaptive management is not yet well implemented, but also what changes can be made to facilitate the adaptive management approach without sacrificing scientific rigor. Adaptive management seems to be directly and indirectly affected by a variety of challenges and convoluted by ambiguity in both planning documents and practitioner’s interpretations of the concept. Addressing these challenges and ambiguities at the project level may facilitate the adaptive management process and help make it more accessible to practitioners.


INTRODUCTION
A common conceptual basis of ecological restoration is restoring an ecosystem to its historical trajectory, a process that is inextricable from the human context in which a restoration project is situated.Each restoration project operates within a socialecological system, and the growing vulnerability of these systems, as well as failures in management, have led to calls for more adaptive governance regimes that can deal with uncertainty and change (Folke et al. 2005, Lebel et al. 2006, Olsson et al. 2006, Butler et al. 2015).
Active or passive adaptive management approaches (Box 1) are now widely recommended to govern ecological restoration because implementation often becomes a continuing series of management decisions or actions that have effects on the entire restoration site, as well as beyond its borders (Murray andMarmorek 2003, Clewell et al. 2005).Each restoration decision is strengthened if it is based on a growing pool of research information, updated measurements of ecosystem responses, and routine progress evaluations toward achieving goals and objectives (Davis andOgden 1994, Buchsbaum andWigand 2012).
The San Francisco Bay Delta region in northern California, USA, comprises the San Francisco Bay and the Sacramento-San Joaquin Delta.It is an area with sizable restoration goals, and there have been several significant attempts to implement adaptive management since 2000, when a strategy was developed for the CALFED Ecosystem Restoration Project (Delta Stewardship Council 2013a).In 2009, through the Sacramento-San Joaquin Delta Reform Act, adaptive management was incorporated as a formal legislative requirement for the Sacramento-San Joaquin Delta (henceforth referred to as the Delta).The strategy was adopted with the aim of increasing the likelihood of meeting goals successfully, effectively, and economically in the face of substantial ecological uncertainty (Delta Stewardship Council 2013a).To support this legislation, a flexible, iterative adaptive management framework was created as a tool for decision making at the program, plan, or project level (Fig. 1)

Box 1: Passive vs. active adaptive management
According to the Delta Stewardship Council (2013a:1), ' Adaptive management' means a framework and flexible decision-making process for ongoing knowledge acquisition, monitoring, and evaluation, leading to continuous improvements in management planning and implementation of a project to achieve specified objectives.Gregory et al. (2006:5-6) contrast passive and active adaptive management as, In passive adaptive management, managers typically use historical data, from the specific area under consideration or from areas considered to be ecologically comparable, to develop a 'best guess' hypothesis and to implement a preferred course of action.Outcomes are monitored and new information is used to update the historical data set and, if necessary, the hypotheses and management action… Under active adaptive management managers typically seek to define competing hypotheses about the impact of management activities on ecosystem functions and, in turn, design management experiments to test them.
However, research has documented that adaptive management is often not as successful in practice as in theory (McLain and Lee 1996, Lee 1999, Gregory et al. 2006, Walters 2007, Allen and Gunderson 2011).In 2000, the first attempt at adaptive management in the Bay Delta was unsuccessful because of barriers such as a lack of secure funding, lack of mechanisms for implementing experimentation, and inflexibility in the traditional agency approach (Healey et al. 2008).Although progress was made in 2009 with reforms and the inclusion of adaptive management into legislation, the approach is still not yet well implemented in ecological restoration of the Delta.Here, we analyze eight tidal marsh restoration case studies in and around the Delta to understand the relationship between adaptive management theory, planning, and practice.Through an analysis of adaptive management documents and in-depth interviews, we tested the hypothesis that adaptive management planning and implementation are not yet well standardized in ecological restoration projects.In doing so, our approach attempted to go beyond identifying challenges commonly faced by practitioners to determine how improved project planning might streamline and increase the ease of adaptive management implementation, without sacrificing scientific rigor.

MATERIALS AND METHODS
A case study approach was used to develop a complete picture of both the planning and practice aspects of ecological restoration projects.The planning documents from eight restoration projects were analyzed to reveal how adaptive management theory was incorporated into each project.Interviews with associated restoration practitioners were analyzed to reveal the practical realities of planning and implementation in the Delta region.

Site description
The Delta was historically a vast dendritic tidal-freshwater marsh characterized by geomorphic and hydrological heterogeneity (Whipple et al. 2012).Reclamation and development since the 1800s, primarily diking for agriculture, has removed 95% of those wetlands from tidal and floodwater inundation (Reed 2002, Delta Stewardship Council 2013a).
This extreme transformation of the landscape has resulted in the collapse of the Delta food web and affected the resilience and size of wildlife populations.Six drivers of change have been implicated in the loss of ecological functions: (1) reduction in the amount of habitat; (2) loss of heterogeneity within habitats; (3) loss of connectivity within and among habitat types; (4) degradation of habitat quality; (5) disconnection of habitats from the physical processes that form, sustain, and confer resilience to them; and (6) the invasion by ecosystem engineers such as Brazilian waterweed, invasive clams, and predatory fish (Whipple et al. 2012).
However, the Delta is still a place of extreme ecological, economic, and social importance.Its waters support > 240 special-status species, an agricultural industry valued at hundreds of millions http://www.ecologyandsociety.org/vol21/iss2/art43/ of dollars (USD) annually, and California's complex water conveyance system.Therefore, the current ecological collapse, extenuated by drought conditions, is inextricable from social and political controversy (Delta Stewardship Council 2013a).The current approach to address these issues includes a massive push toward restoring extensive areas of diked former marsh (Orr et al. 2003).Tidal marsh restoration design and planning has evolved in the last decade, with projects increasing in size and complexity.However, few undisturbed reference sites remain, and there is still considerable scientific uncertainty about how to restore marsh habitat (Callaway et al. 2011).
Of the eight tidal marsh restoration case studies examined herein (Box 2), five are located within the bounds of the Delta, where adaptive management is legally required.Three projects are located outside the Delta, either in the vicinity of Suisun Marsh or San Francisco Bay.Four projects are still in the planning phase, two are currently under construction, and two have completed construction.

Box 2: Project descriptions
Project A: The site was historically a combination of grassland, seasonal wetland, open water features, and tidal marsh until it was diked and reclaimed for agriculture in the mid-1900s.This ongoing project aims to restore physical processes (hydrodynamics and sedimentation) to increase productivity and benefit fish.
Project B: The site was historically tidal marshland until it was leveed in the early 1900s for grazing and dairy operations.This ongoing project aims to restore three parcels to create tidal marsh, riparian woodland, irrigated pasture, subtidal open water, and managed nontidal marsh.An experimental approach will be taken to compare two elevations and sizes of marsh.
Project C: The site was historically influenced by tidal flows from the Bay and backwater flooding from the Sacramento River until construction projects modified the hydrological regime and isolated a section of the slough.This completed construction project aims to reconstruct the historic channel and re-establish tidal connection to the marsh to benefit native species and water quality.
Project D: The site was historically an expansive tidal marsh until it was reclaimed for agriculture.This ongoing project aims to restore emergent wetlands and subtidal habitats, with a focus on enhancing primary and secondary productivity for native fish.
Project E: The site was historically tidal freshwater emergent wetland and was one of the last to be reclaimed for agriculture.This ongoing project aims to enhance flood protection while restoring a mosaic of tidal freshwater wetlands, seasonal floodplain, and riparian areas.
Project F: The site was historically brackish tidal marsh until the 1930s, when it was managed for livestock grazing and duck hunting activities.This completed construction project aimed to use a minimal construction and site manipulation approach to transition the site to an intertidal brackish habitat by increasing elevations over time via natural sedimentation and restoring full tidal exchange.
Project G: These sites were historically tidal wetland habitat until they were converted into salt crystallizer ponds for salt production starting in the late 1800s.This ongoing landscape-scale project aims to use applied studies to guide the restoration of a mix of tidal marsh, mudflat, and managed pond ecosystems.
Project H: The site was historically characterized by brackish tidal marsh until it was diked and drained for agriculture in the 1880s.This ongoing project aims to use the engineered placement of ~15.3 million m³ of agency-approved dredged sediment to raise site elevations and restore a mixture of tidal wetlands, seasonal wetlands, intertidal ponds, vernal pools, and upland buffer zone.

Case study selection
To settle on these eight case studies, a variety of tidal marsh restoration projects were identified that use an adaptive management approach and have publicly available planning documents (Table 1).As long as the projects had these attributes, the phase of project completion (construction planned, ongoing, or completed) was considered irrelevant.The projects were identified using information from the Sacramento-San Joaquin Delta Conservancy and the project mapping tool EcoAtlas (California Wetlands Monitoring Workgroup 2015).Prime case study candidates were then identified using a maximum variation sampling approach (Patton 2002).Current research does not agree on the ideal number of case studies to use in a multiple case study approach, but Eisenhardt (1989) suggests a maximum of seven cases, whereas Yin (1994) suggests that data should be collected until saturation.
Representatives of 12 projects were contacted by email with an interview request, and subsequently, 12 in-depth, semistructured interviews were conducted during winter 2015.Of these 12 projects, 2 fell outside the scope of this examination because the participants did not discuss a singular project, and 2 were not usable because the project did not yet have any publicly available adaptive management planning documents.

Case study analysis
Eight case studies, each with an associated interview, were usable.The interviews explored how the representative of each restoration project defined adaptive management, their attitudes toward it, experiences with implementation, and suggestions to improve its ease of use.In addition, interviews delved into project setbacks and best practices.However, the opinions and personal views shared by interviewees are not necessarily reflective of the organizations for which they work.An interview guide ensured comparability across interviews (Appendix 1).After gaining consent, all interviews were recorded using a Sony ICD-UX533F digital voice recorder and transcribed by hand in Microsoft Word.Initially, the transcripts, ranging from 11-20 pages, were uploaded in MAXQDA, a qualitative data analysis software program, to conduct a preliminary within-case analysis.Each transcript was coded according to key research themes based on both the interview question topics and a background literature review.Themes included adaptive management definitions, challenges, and recommendations.Subsequently, the results of the coding were exported to Microsoft Excel, and a cross-case analysis was Table 1.Use of key planning elements of adaptive management.Y = yes, element included; N = no, element not included; L = limited, element included but lacks detail or is hypothetical; -= unable to determine, planning has been delayed and is not finalized.Nyberg (1999), Allen and Gunderson (2011), Fischenich and Vogt (2012).
conducted.Through categorization and pattern matching, subthemes such as the different types of challenges were determined.
A similar approach was taken to analyze project planning documents.First, the number of key elements in each project's planning documents was identified.Then, the specificity and detail allotted to each key element was assessed and compared across projects.

Analysis of planning
It is clear from the evaluation of restoration project documents that the general format for adaptive management planning documents is relatively undefined, and that the inclusion of key planning elements varies by project (Table 1).A majority of the eight projects' plans did not include all of the key elements identified in adaptive management planning guides for the Delta, and only a few of the projects included additional key elements recommended by a wider array of guides.However, several plans are still in draft format, and all aspects are not yet finalized.Therefore, planning related to these projects may still evolve.As they currently stand, almost every project gave a detailed description of the key elements related to early project design: (1) a description of goals and objectives, (2) use of quantitative models to identify trade-offs of different management actions, (3) consideration of restoration design alternatives, and (4) defined performance measures.Six of the eight plans included a monitoring procedure specific to the project objectives.Two projects failed to include a detailed description of monitoring, but these plans are still in draft format.Five of the eight plans stated key uncertainties, specified reporting requirements, and included conceptual models showing cause and effect pathways.The specified reporting requirements, however, related primarily to the permit process rather than to how results would be shared with the public.
Several of the key elements potentially more relevant to adaptive management and the project postplanning were often more ambiguous.Only four projects explicitly translated the key uncertainties into hypotheses of system functions or defined who would be responsible for coordinating and carrying out specific tasks.Only three included a framework for decision making or considered potential future management actions in terms of adaptive changes.Just two projects planned out data management and documentation procedures such as data or information storage; and just two very clearly defined the role of science and technical experts in the project.Further, although all projects considered applied studies or experimentation in some way, a majority (six of eight) stated that additional resources or support were necessary to pursue them.Only two projects explicitly planned experimentation as an integral part of the project.

Analysis of practice
Practitioners' experiences revealed that in reality, adaptive management is an ambiguous concept, and project planning and implementation is complex and slow.Selected quotations regarding practitioners' interpretations of adaptive management can be found in Appendix 2. Practitioners often described the adaptive management process as fluid and organic.Beyond this, two differing views became clear.Some practitioners described adaptive management as a common-sense approach without a set form and driven purely by project objectives.As one practitioner stated, "It's just going to be completely fluid… totally driven by whatever your hypothesis is and if you're meeting your goals and objectives."In contrast, some practitioners described it as a very structured and defined strategy.For example, one practitioner stated, "It doesn't do any good to just react, you have to have a framework, a process by which you respond sensibly and productively."When considering their projects, practitioners also showed a tendency to break adaptive management into different components, mentioning adaptive management of monitoring elements, of the physical restoration configuration, and of communication.
Projects were exposed to a variety of interrelated challenges, which were similar between projects, although they varied in extent and difficulty (Table 2).Our analysis focuses less on scientific and technical challenges because a consensus was clear that these aspects could be addressed fairly easily, whereas the problems related to financial, bureaucratic, regulatory, and political aspects of the projects were harder to resolve.Supporting quotations defining these challenges can be found in Appendix 3.
The challenges that came up most frequently during interviews dealt with regulatory aspects of the projects (Table 2).Five practitioners felt that the cost and time needed to reacquire permits to make physical adaptive changes after opening the site to tidal inundation were prohibitive.Two projects did not encounter this challenge, perhaps because of the way the projects were set up, with a mind to constant scientific review, cooperation with the regulatory agencies issuing permits, and a long project timeline (e.g., 50 years).Five practitioners felt that the environmental review process was complex, costly, and slow.The main frustration was the fact that restoration projects meant to benefit the environment were faced with the same challenging permit-acquisition process as development projects.However, several practitioners felt that the existing process was necessary and straightforward if approached correctly.Finally, three practitioners detailed the challenges of conflicting regulatory priorities.Tidal marsh restoration is complex from a regulatory standpoint because it generally involves converting one wetland type into another (e.g., managed wetland to tidal marsh).The act of restoration, therefore, can have a harmful effect on preexisting species, a fact that leaves regulatory bodies, as well as the project managers, unsure of how to move forward.
Other frequent challenges that projects faced were bureaucratic, such as acquiring property or funding.Four practitioners encountered difficulty in acquiring the necessary land for restoration.Public agencies are not allowed to offer more than market value for property, and therefore, generally focus on restoring public lands.However, all four projects in question had mandated restoration targets, and agencies encountered difficulty in acquiring enough land to fulfill their obligations while still ensuring the land was conducive to tidal marsh restoration.Four practitioners found it challenging to acquire funding for specific aspects of their projects.Whereas the construction phase of projects was generally well funded, practitioners found that support was limited for aspects of the project related to evaluation and learning such as monitoring, publishing results, attending seminars, and conducting experiments.
A majority of challenges were linked to aspects of project management itself.Three practitioners described challenges in coordinating science and management, and four practitioners described challenges in coordinating the multiple agencies involved in their restoration project.The former was a challenge in trying to balance scientific rigor with the many constraints of managing a project.The later was seen as difficult because of the large number of agencies, each with their own rules, concerns, and stakes in the project.Three practitioners faced shifting internal priorities, including waxing and waning internal support for the http://www.ecologyandsociety.org/vol21/iss2/art43/restoration project, as well as a lack of priority given to fixing monitoring problems or writing management documents.Four practitioners mentioned the challenge of risk aversion.Practitioners felt that restoration projects were contentious and the people involved faced considerable pressure not to take unnecessary risks.Practitioners therefore placed extraordinary time and effort on project design and planning to avoid failure after implementation.Finally, three practitioners stated that staff turnover was a challenge.Changes in staffing were common because of the number of years, or even decades, needed to bring projects to completion.Effects included project setbacks, changes to standing agreements, or starting projects on a different track.
The final two challenges dealt with project-public interactions.
Four practitioners stated that negotiations with adjacent landowners were intricate and caused project delays.Landowners were interested in how water moved on and off the site and in ensuring that there would be no adverse effects from the restoration operations.Two practitioners found it challenging to gain public support for their projects.Miscommunications and misunderstandings between restoration staff and the public were common and at times created derisive and contentious atmospheres around these projects.
In addition to challenges, practitioners were asked to describe best practices and ideas to improve the facilitation of adaptive management.Selected quotes can be found in Appendix 4. Some of these recommendations are achievable at the project level, whereas others require structural changes at a higher level.For example, several practitioners called for a more formalized ecological restoration community and a regional monitoring framework to increase opportunities for collaboration and coordination across projects.In addition, several practitioners felt that flexible permitting, increased funding, and an improved system for acquiring land would facilitate both restoration and adaptive management.
At the project level, a majority of practitioners felt that there needed to be "more doing" so that learning would be based on experience.They spoke of the importance of proper site preparation, as well as frequent site visits to notice surprises that might affect the trajectory of the restoration.One practitioner felt that adaptive management could be facilitated better if projects incorporated a standing independent scientific review team that worked as an intermediary between the project and the regulatory agencies.Another practitioner felt that a multibenefit approach was beneficial, for example, combining flood protection and restoration goals.The most common themes were the need for increased communication between restoration participants, and the need to create a culture of patience, listening, and openness.In line with this, practitioners recommended trying to make plans as specific as possible, with defined hypotheses, organizational structures, and requirements for documentation and continuity.

DISCUSSION
The investigated case studies are essentially some of the first restoration projects in the Delta that incorporated the Delta Reform Act concept of adaptive management, which makes them a valuable source of information.It is too early to evaluate whether adaptive management is a successful management technique for each individual restoration project.However, one can start to evaluate the realities of the adaptive management planning and implementation process.
There are some limitations to the assessments presented herein.All relevant plans used in the analysis of project documents may not have been finalized because of the varying stages of project completion.In addition, not all planning documents had the same format, making comparison sometimes difficult.Planning elements were clearly defined to make the evaluations as objective as possible.In addition, the interview approach reveals perspectives specific to the practitioners' duties and experiences, which may vary somewhat between projects.Because the questions asked were often open ended, the most important, but perhaps not all, recommendations and challenges were captured.
Restoration in the Delta operates in an extremely complex social, political, and economic setting, and the nuances of each project's challenges may be lost in this overview.
However, the challenges described by restoration practitioners in and around the Delta have all been identified previously as reasons for adaptive management failure, revealing that they are not specific to these case studies or even ecological restoration (Gregory et al. 2006, Walters 2007, Allen and Gunderson 2011).
For this reason, the lessons learned from this analysis may be valuable beyond the field of tidal marsh restoration and relevant beyond the geographical context of the San Francisco Bay Delta Estuary.
While a majority of the challenges indirectly affect adaptive management by delaying the rate at which decisions and changes can be made, several directly influence the implementation of adaptive management.These challenges include: making physical adaptive changes, acquiring funding, coordinating science and management, and risk aversion.These four challenges essentially interrupt the iterative adaptive management cycle between the evaluation and planning stages or between the planning and implementation stages (Fig. 1).In doing so, they eliminate opportunities to use learning to inform key management decisions.
The feedback between learning and decision making is a defining feature of adaptive management (Williams and Brown 2012).Theoretically, it allows for action, even in the face of uncertainty, because of the understanding that new learning will fill the gaps in knowledge over time (Buchsbaum and Wigand 2012).Perhaps because the iterative process is being interrupted repeatedly, this feedback process does not seem to be occurring yet in the Delta.
Addressing the four aforementioned challenges may help streamline the feedback process.
However, compounding this problem, planning documents do not necessarily define key uncertainties or, more importantly, specify how to reduce them.The plans generally tried to account for all of the identified key elements, but often did not include enough details to provide clear guidance.Ambiguity was common, particularly in planning related to decision making, potential management actions, data management, documentation procedures, organizational structure, and scientific review.
Research suggests that project success is positively correlated with investment in defining requirements and in developing technical specifications (Dvir et al. 2003).Therefore, addressing ambiguity and the lack of detailed planning for integral adaptive http://www.ecologyandsociety.org/vol21/iss2/art43/management elements may help address the aforementioned challenges and increase chances of successful adaptive management.
In fact, practitioners called for more detailed planning, including increased specificity, improved documentation, defined hypotheses, defined organizational structures, and the inclusion of independent science review.One practitioner stated that if plans were more specific, "Then it's not me going to my boss saying, 'Oh, I think we need to add a breach here.'It's me going to my boss saying, 'This action on site is not occurring, this is what is laid out to fix and remedy that issue, and this is what the suggestion is in the restoration plan to do that.'"Projects that included detailed planning for the key elements identified in Table 1 were more able to ensure an iterative adaptive management cycle of learning.For example, two projects that defined a very specific science review process were able to work more closely with regulatory agencies to make physical adaptive changes and address risk aversion (Appendix 4).
Changes at the project level may therefore begin to address some of the internal, external, and regulatory challenges (Table 2) and help to deconstruct obstacles to the iterative cycle of learning.However, these changes may add additional expenses or may be difficult to plan early on because of a lack of information (Cabin 2007).The remaining regulatory and bureaucratic challenges cannot simply be addressed at the project level but may require changes at a governance level (Table 2).Large-scale modifications are currently being considered for ecological restoration in the Delta, and efforts are ongoing to increase structural flexibility to support better an adaptive management approach (Delta Stewardship Council 2013b, 2014, Lund and Moyle 2013).The analyses presented herein support efforts to create a formalized ecological restoration community for the San Francisco Bay Delta Estuary, which could potentially act as a centralized forum to address the challenges or knowledge-sharing issues addressed herein.
Restoration projects are immensely complex, and creating shared interpretations of adaptive management may also be essential to adaptive management success.From a theoretical standpoint, the cycle is a structured, iterative process marked by set decisionmaking points, at which time management actions or adjustments should be made (Williams 2011).Under a passive approach, the best management action is simply implemented, and monitoring and evaluation are used to introduce the appropriate adjustments.Under an active adaptive management approach, monitoring and evaluation of each alternative helps determine which one is more effective in meeting objectives, and adjustments are made in the next round of management decisions (Murray and Marmorek 2003).However, in reality, evaluation and decision making occur regularly and frequently throughout a project's timeline (Nilsson et al. 2016), suggesting that learning must be reapplied to a project at all times, not solely at set decision points.As one restoration practitioner acknowledges, decisions occur "…more by osmosis in practice, than through deliberate actions." This reality of learning and decision making may explain the interpretation of adaptive management as a common-sense approach without a set form.Managers must constantly be able to make decisions on a variety of topics, including monitoring, communication, and physical design.However, this must not take away from the particular importance of a well-structured decision-making process.Although Bart (1993) and others claim that too much planning can curtail creativity, our case study analysis suggests that specific planning, when applied as a strong framework for projects, to be updated as learning occurs, may to some degree address both the challenges and ambiguity that restoration practitioners in the Delta face when carrying out adaptive management.

CONCLUSION
Adaptive management is not yet a well-defined or easily implemented strategy for restoration managers in the Delta.Implementation seems to be directly and indirectly affected by challenges and convoluted by ambiguity.Our case study analysis suggests that planning should be very specific about potential management actions, decision making, scientific review, documentation, and responsibilities, to support the scientific underpinnings of an adaptive management approach.In addition, plans that are updated as projects progress and focus on how key uncertainties will be reduced may allow for more action on the ground.Increased attention to detail in planning can be seen clearly in projects in which adaptive management became more than just a legislative requirement, and was actually integral to the restoration trajectory.With a combination of changes at the project level and the governance level, it may be possible to make adaptive management more accessible to managers, increasing knowledge of restoration processes as well as chances for restoration success.
As restoration initiatives move toward large, landscape-scale projects, as in the Delta, adaptive management may provide a scientific framework that can support more complex projects and result in better ecological restoration, as also described by Simenstad et al. (2006) and Cabin (2007).Innovative solutions to the challenges encountered by restoration practitioners will help transition adaptive management from a promising theoretical concept to a workable management strategy.
Responses to this article can be read online at: http://www.ecologyandsociety.org/issues/responses.php/8521

Fig. 1 .
Fig. 1.The Delta Plan's nine-step framework for adaptive management.(Courtesy of the Delta Stewardship Council).

Table 2 .
Challenges described by restoration practitioners.Y = yes, challenge encountered; N = no, challenge not encountered; -= challenge not mentioned.