Resilience is a measure of the amount of perturbation or disturbance a system can withstand without crossing a critical threshold (Holling 1973). When such a threshold is exceeded, the system collapses and reorganizes. Reorganization can occur with the same driving variables and processes, in which case the original system renews, or the system can reorganize around a new set of variables and drivers in which case a new organization emerges, and quite often a very different system (Chaffin et al. 2016). When reorganization occurs around new drivers, the new system may be less desirable to humankind than the former system in terms of the provision of goods, services, and relative predictability. Therefore, it is often in managers and other stakeholders’ interest to maintain systems in desirable states, avoid critical thresholds, and enhance resilience. Alternatively, when a system is undesirable, it may be necessary to erode resilience and purposely transform the system to a more desirable state (Chaffin et al. 2016). In either case, it is important to be able to assess, at least relatively, how resilient a system is, so that managers can either foster or erode resilience, depending upon the desirability of the current system state.
Resource managers often rely on the resilience, or the capacity, of a social-ecological system (SES) to absorb and respond to a disturbance while maintaining its essential structure and functions (Holling 1973, Folke et al. 2002). A resilient system is useful to managers because it provides latitude in management options, and management is less likely to result in an unwanted regime change in a resilient, versus a nonresilient system. An improved understanding of the boundaries of resilience, i.e., the thresholds that separate one state of a system from alternative regimes, may help resource managers avoid or facilitate regime shifts so that desired ecosystem services are maintained or restored. Resilience in a complex system of people and nature has both ecological and social dimensions (Folke et al. 2002). Whereas ecological resilience is the magnitude of disturbance that an ecosystem can absorb before it shifts into a new regime, social resilience is the capacity of social systems to withstand and adapt to disturbances that result from social, political, or environmental changes (Adger 2000). Enhancing and maintaining resilience is increasingly identified as a management goal or strategy for projects focused on either ecosystems (e.g., Benson and Garmestani 2011, WWF 2012) or social systems (e.g., Godschalk 2003, Norris et al. 2008, Longstaff et al. 2010), but resilience is best understood as a product of both social and ecological, reinforcing, interactions. However, because of the different drivers in social and ecological systems, explicitly considering social versus ecological aspects of resilience can be useful and provide meaningful insight.
Unfortunately, quantifying resilience is poorly developed (Angeler and Allen 2016). Many resilience assessment approaches are based on a gestalt regarding the resilience of the system, reliant on stakeholder inputs that envision the system of interest, dominant drivers, and a limited range of scales above and below the focal scale of interest (Angeler and Allen 2016, Quinlan et al. 2016), disregarding complex cross scale structure (Allen et al. 2014). With such approaches, quantitative assessments are difficult. However, advancing resilience science is important if the concept is to have utility and application for navigating a rapidly changing Anthropocene (Biggs et al. 2012, 2015).
Although several frameworks have been proposed for conducting resilience assessments (e.g., Walker et al. 2002, 2009, Resilience Alliance 2010, Biggs et al. 2015, Quinlan et al. 2016), the application of these techniques to real-world systems continues to be a challenge because of the highly dynamic and multidimensional nature of linked social-ecological systems (Berkes and Folke 1998, Walker et al. 2002). In addition, although some practitioners may find these or similar frameworks to be useful and appropriate (see Resilience Alliance 2013 for some case studies, http://www.resalliance.org/resilience-analysis-practice), others may want to conduct a resilience assessment but lack sufficient time or information to use the often detailed and time-consuming approaches inherent in existing frameworks. In addition, an important aspect missing from existing resilience assessments are measures of uncertainty, and measures that are useful to compare relative levels of resilience across similar systems. Assessing uncertainty will help understand the systems in question better and should improve management by identifying areas of knowledge deficit, allowing the design of adaptive interventions that can further enhance understanding of the system, enhance learning, and iteratively reduce key areas of uncertainty as revealed by analyses.
Uncertainty takes many forms (Williams 2001, 2011), as does its quantification and identifying key uncertainties through resilience assessments may provide insight into how assessments can be improved, how the system itself functions, and the potential for structured experimentation and learning, which in turn can reduce uncertainties (Allen et al. 2011, 2016a, Birge et al. 2016). Therefore, methods to assess uncertainty are critical for advancing resilience theory and for the application of resilience approaches to particular systems or challenges. We present and apply a simplified approach to resilience assessment that incorporates Walker and Salt’s (2006) nine measures of resilience: ecological variability, diversity, modularity, acknowledgement of slow variables, tight feedbacks, social capital, innovation, overlap in governance, and ecosystem services.
An absolute measure of resilience has not been developed, is not likely to be developed, and may not be useful (Quinlan et al. 2016). Rather, there are two types of resilience assessments: the quantification of specific resilience, that is, the resilience of what, to what, and for whom (Carpenter et al. 2001), and general resilience of similar systems relative to one another (Nemec et al. 2013). We focus on the latter type of resilience assessment (relative resilience) to incorporate trade-offs and uncertainty. Quantitative approaches to resilience that also incorporate uncertainty, as we do, may provide new avenues to assess risk and vulnerability. Our empirical results are for illustrative uses only because our sample sizes are small. Our analyses are not meant to capture the true resilience of these watershed-based systems, but are meant to illustrate the utility of the approach.
River systems in water-stressed landscapes present some of the most challenging natural resource management issues facing the world today. Anthropogenic pressures have significantly altered river systems, affecting the provision of ecosystem services.
Our resilience assessment surveys were administered to stakeholders in the Anacostia, Columbia, Middle Rio Grande, and Platte River Basins, all located within the United States (U.S.), with the exception of the Columbia River Basin, which extends into Canada at its northern reaches. These watersheds were included in a SESYNC (National Socio-Environmental Synthesis Center) working group focused upon adaptive governance of stressed watersheds, and all share the basic similarities of being water stressed, greatly modified, the subject of intense management and frequent litigation, subject to adaptive management restoration efforts, and with unknown resilience.
The Anacostia River Basin encompasses 1140 km² of rural to urban land-cover types, including the District of Columbia, making it the most urban system of those assessed. Population density in the 1990 census was 2.66 persons/acre. As a result, questions surrounding the resilience of the Anacostia River Basin often focus on the social and ecological system’s ability to withstand ongoing or increased pollution, runoff, and flooding. This is especially true in light of the complex institutional interactions that define the ability of social and ecological components to respond to disturbances related to water quality (Arnold et al. 2014).
The Columbia River Basin covers an area of roughly 670,000 km² and extends across large parts of Idaho, Washington, Oregon, British Columbia, and smaller parts of Wyoming, California, Nevada, and Utah. The basin includes largely rural landscapes, but also encompasses a handful of metropolitan areas including Portland, Oregon, Boise, Idaho, and Spokane, Washington, in which most of the Columbia Basin’s six million human inhabitants reside (Cosens and Fremier 2014). In the Columbia River Basin, issues emerging from the intersection of climate change and water scarcity, characteristic of western river systems, include competition among hydropower, irrigation, flood control, ecological integrity, and other valuable social-ecological goods and services for dwindling water supplies (Cosens and Fremier 2014).
The Middle Rio Grande Basin encompasses roughly 8000 km² and is contained entirely in the U.S. state of New Mexico. Although sparsely populated through most of its extent, it contains the city of Albuquerque and a total population of 690,000 (Bartolino 2012). The resilience of the Middle Rio Grande River Basin generally concerns the ability of the social-ecological system to withstand ongoing human population growth and urbanization, biodiversity loss, and cyclical drought in the face of ongoing appropriation and climate change (Benson et al. 2014). Much like the other western water systems in our study (Columbia and Platte), interstate water and/or energy compact obligations strongly interact with social-ecological components in the system.
The Platte River Basin drains an area of more than 23,000 km² extending across the U.S. states of Colorado, Nebraska, and Wyoming (Palmer 2006). In the Platte Basin, the population is disproportionately distributed among a few dozen medium (e.g., Kearney, Nebraska) to very large (e.g., Denver, Colorado) metropolitan areas, many of which are in the South Platte Basin, in which population is expected to double in the next 40 years, putting additional pressure on already over-appropriated water sources (CDLF 2010). Similar to other western basins, interstate water compacts and limited flows constrain the amount of water available for appropriation for endangered species, riverine wetlands, irrigation projects that generate power and recreation opportunities, downstream users entitled to flows, and other uses. Resilience in this system is therefore largely focused on the ability of the social-ecological system as a whole to withstand flow variability, specifically droughts, but also major floods, without any integral system component losing access to water and undermining the rest of the system (Birge et al. 2014).
We used a rapid prototyping approach (Nicolson et al. 2002) to score relative resilience by having stakeholders individually respond to a series of survey questions meant to probe different aspects of the social-ecological systems inferred to confer resilience, on a Likert scale of 1 to 5, with 1 being least resilient and 5 being most resilient (Table 1). Additionally, we rated the uncertainty of each resilience score on a scale of 1 to 5, i.e., low to high. The choice of metric for resilience assessment is important. We focus on social and ecological components, but incorporating components of critical infrastructure and explicitly incorporating economics may also be useful. Organizations such as the Resilience Alliance (http://www.resalliance.org/) have spent considerable time and resources in determining relevant aspects of resilience. Although it would be appropriate for assessments to develop particular aspects, or axes, of resilience, we utilize a modification of those presented in Nemec et al. (2013), because they are both useful and illustrative. Our method is based on surveys of stakeholders in the social-ecological systems of interest and based on 26 questions focused on social and ecological aspects of resilience. The selection of particular variables (survey questions) is important and necessarily varies based on the systems of interest and the aspects of those systems that are important.
We surveyed government, researchers, end users, and NGOs within each of the four watersheds of interest to illustrate our methods for assessing relative resilience of the watersheds, trade-offs among social and ecological components of resilience, and uncertainty in the assessments. To identify participants, each of the basin research leads invited 40 people they identified as members of each of the 4 stakeholder groups (10 people per group) to participate in the survey using an institutional review board (IRB) approved email. When 10 people per group could not be identified, basin leads/the survey coordinators used search engines to locate additional participants. Potential participants were informed that they were identified as users belonging to certain groups by the basin leads, but that they could reassign themselves to other groups should they agree to participate. Names and email addresses of those who gave affirmative consent to participate were sent to the survey coordinator. Participants were then sent an email from the coordinators with informed consent, survey instructions, and an anonymous link to the survey, administered online using SNAP software.
In total, 200 (5 basins, 4 user groups each, 10 surveys per user group) surveys were administered, 30 were completed for a response rate of 15%. More than 30 responses were initiated with more than 1 question completed, but we only included fully completed surveys in our analyses.
Because we relied on interpersonal connections of academic researchers, there is bias toward participants who are already likely to engage with the material in the survey, and they may have lower uncertainty and stronger opinions than the population we hope our participants represent. Further, we used a search engine to identify NGO and end-user participants more than government and academic participants, potentially leading to a more random sampling of those groups. However, because our analyses were meant to explore a new methodology and not to draw inference regarding the user groups or the basins of interest per se, the bias in our participants does not affect the objective of our study.
Although social and ecological resilience are intertwined, we evaluated the properties for social and ecological resilience separately as well as in combination to provide a clearer assessment of the elements of resilience (Allen et al. 2003). Because not all of the properties applied to both kinds of resilience, we assessed eight with regard to social resilience and three for ecological resilience (Table 1). We concurred with Walker and Salt’s (2006) creation of a property for ecological variability but not social variability because, as they defined it, variability refers to variability in the occurrence and magnitude of ecological phenomena, such as flooding and wildfire that do not have a social equivalent. Similar aspects of a system, such as social and cultural heterogeneity, are incorporated into the social diversity and social modularity variables. Likewise, the social properties of social capital, innovation, and overlapping governance do not have an ecological equivalent. We decided that slow ecological variables and tight ecological feedbacks are so closely related to ecological variability that these properties are encompassed by the ecological variability property, and ecosystem services are a social and not an ecological construct because they refer to the benefits that humans obtain from nature.
Within each of these 5 basins, we invited 40 survey participants that we identified as representatives of end users (farmer, rancher, and/or private citizen), state or federal government, nongovernment organizations, or research/extension stakeholder groups, but allowed respondents to reassign themselves at the outset of the survey to the stakeholder category with which they most closely identified. This, along with the survey hyperlink being nonspecific to user, and the collection of no identifiable information beyond stakeholder group and basin, assured that the anonymity of the participants was carefully preserved.
We sent consenting participants a hyperlink to an online questionnaire. The questions we included in the survey were designed to assess stakeholder perception and uncertainty concerning various aspects of their system’s social-ecological resilience. Specifically, we designed our questionnaire to analyze stakeholders’ assessments of Walker and Salt’s (2006) nine properties of a resilient world, but within individual contexts (Table 1). We used a Likert scale of 5 points for the questionnaire, which included 25 content questions each followed by a question asking respondents to rank their level of certainty in their response to the previous question (see Appendix 1 for the full list of questions). Some definitions (e.g., modularity) were provided to respondents via hyperlinks embedded in the online survey (see Appendix 2 for definitions provided).
Our questions addressed both how different stakeholders from the five basins perceive the level of (1) biological (species, response, and trait) diversity, (2) ecological variability, (3) modularity. (4) acknowledgement of slow variables, (5) feedback length, (6) social capital, (7) innovation, (8) overlap in critical social roles, and (9) ecosystem services provisioning in their social-ecological system. We asked stakeholders explicitly to assess these properties (i.e., “assess the ecological diversity in your system”), but also designed questions to address the properties for a stressed river basin context.
These nine resilience variables likely represent an incomplete and subjective list, but their use in prior assessments (e.g., Nemec et al. 2013, Allen et al. 2016b), ease of interpretation, and applicability across different systems lend them well to a rapid comparative approach and for uncertainty assessments, as we have done here. Practitioners should design their own survey questions based upon the context of their study.
To assess resilience and uncertainty, we calculated the mean scores reported for each question by assigning values corresponding with the Likert score for each question (i.e., very high = 5, high = 4, moderate = 3, low = 2, very low = 1, etc.). To assess uncertainty, we calculated mean uncertainty across individual stakeholders as reported in the questionnaires (explicit uncertainty) as well as the degree of variance across both response to content questions and the explicit uncertainty (implicit uncertainty). Only completed surveys were included in our analysis.
Relative resilience is inferred from the total area under the curve in the radar plots of results (refer to Figs. 1-8), that is, by summing the individual axis scores. Ecological and social areas can be compared to assess trade-offs among different components of resilience. Relative resilience scores and degree of uncertainty can be compared among user groups and basins. Therefore, a suite of empirical results follows from our survey: relative resilience of each basin; the sum of the individual axis scores, and uncertainty therein; relative measures of resilience of individual axes, and uncertainty therein; relative resilience scores and their uncertainty for different user groups and for social versus ecological components; and trade-offs among axes of resilience, measured by the axes scores and in particular the relative strength of scores in relationship to one another. Additional measures are possible, as is the selection of different axes for measuring resilience.
We received completed responses (n = 30) from representatives of all four user groups and four watersheds (Figs. 1-9). Total resilience scores (the sum of average response to questions) was similar among NGO, end user, and government user groups, but markedly lower for research/extension respondents (Table 2.). Among river basins, respondents from the Anacostia reported the highest scores (Fig. 5), followed by Columbia, Platte, and Middle Rio Grande (Table 2).
No individual questions from the survey appeared to drive or diverge from total resilience trends across user groups, but research/extension respondents consistently reported lower average scores (Fig. 2). Variance in our assessment is meant as an index of implicit, group-level uncertainty. Although most user groups had relatively low variance relative to their mean resilience scores, there are departures from this trend for the end users in their responses to questions on flow variability, ecosystem services production, and trust among stakeholders. Spikes in variance (Fig. 2) for some questions and user groups indicate uncertainty in those areas and opportunities for learning and further probing to determine the sources of uncertainty.
In addition to this group level, implicit uncertainty (i.e., variance in responses), we asked respondents to report their self-assessed individual level of certainty in their responses (Fig. 3) as a second approach to quantify uncertainty. In our survey, self-assessed uncertainty varied relatively little and was moderate across all questions, although there appears to be more uncertainty in social and governance questions than for ecological questions. End users reported higher mean certainty in their responses; we also assessed variance in user identified uncertainty and found relatively low variance with interesting departures of high variance for some questions and user groups, particularly for social modularity among end users. (Fig. 4).
Resilience scores for each question across basins (Fig. 5) parallel total resilience scores (Table 2). However, it is noteworthy that the shape of the resilience scores in Figure 1 is rounder than that of Figure 5, even though they draw from the same source data. This is perhaps reflected in the significantly higher variance, or implicit group level uncertainty in resilience scores across basins (Fig. 6). Individual uncertainty was also higher across basins (lower certainty scores; Fig. 7) than across user groups, but this difference is nominal and likely attributed to the redistribution of end users across basins in the former analysis. Similar to the pattern of variance across user groups, there was no discernible pattern of variance in level of response certainty by basin (Fig. 8).
When the variance of mean response to questions was plotted against certainty (i.e., group level uncertainty versus individual certainty), we expected a negative relationship. However, this relationship is weak (Fig. 9) and only significant (P < 0.043) when comparing variance versus certainty of user groups.
Quantifying absolute, general resilience of complex systems of people and nature is problematic, but assessing relative resilience and specific resilience (resilience of what to what) is a realistic goal that may provide useful tools for managers and policy makers, as well as other stakeholders. Furthermore, the search for better methods to quantify resilience will lead to greater understanding of the drivers of resilience in complex systems. We expanded upon the properties of resilience forwarded by Walker and Salt (2006) and modified by Nemec et al. (2013) to develop a straightforward survey of stakeholders in SES and approaches to quantify relative resilience (Nemec et al. 2013), trade-offs among social and ecological (or economic or infrastructure) components, and uncertainty. We addressed uncertainty in two ways: user self-assessment and analysis of variance within responses and across watersheds and user groups. Although we developed and implemented our survey simply to demonstrate the approach, with sufficient randomization and sample size, quantitative comparison of our metrics is straightforward, and amenable to analysis of variance or a number of similar approaches to determine if significant differences are present in responses, across basins or user groups.
It is clear that resilience in complex systems of people and nature encompasses both social and ecological components of the system. Furthermore, purely social components can be differentiated from economic components. Among social, ecological, and economic components, clear trade-offs are often apparent. For example, the Platte River Basin, encompassing nearly all of Nebraska and parts of Colorado and Wyoming, is a system that has been heavily altered by human activities (primarily agriculture), with extensive hydrological alteration of surface and groundwater in support of agriculture and development. In support of agriculture, and for other reasons including energy production, the Platte River itself has been dammed and hydrological variability greatly reduced. The riparian corridor has undergone a regime shift (Birge et al. 2014). Prior to damming, pulsing floods created bare sandbar habitat necessary for currently endangered least terns (Sternula antillarum) and piping plovers (Charadrius melodus). With damming, sandbars became vegetated and eventually armored by herbaceous and woody vegetation, and channels became reduced and incised. The riparian corridor, ecologically, is in an undesirable state with deep hysteresis. However, the social and economic aspects of the Platte Basin are highly desirable, and, unconsciously at least, a decision has been made to sacrifice the ecological component in favor of the social and economic components (Birge et al. 2014). Such imbalance is likely to lessen overall resilience. Our described methods allow for assessing trade-offs in system subcomponents, which, in addition to social and ecological, could include economic and infrastructure components.
Identifying individual components of resilience that are weak or highly uncertain, either at high levels (ecological versus social, example above) or at the survey question (axis) level, should be beneficial for practitioners. Explicitly considering trade-offs and identifying the areas of highest uncertainty will allow for focus on those aspects or components in most need of intervention or further understanding. These types of results can also provide baseline information against which the success of interventions can be assessed over time. Time series data following interventions that affect resilience (e.g., Nemec et al. 2013) provides an especially valuable opportunity to assess trade-offs.
Resilience approaches are maturing, and better, objective, measures of resilience are being developed (Angeler and Allen 2016). It is critical that quantitative methods be developed to complement qualitative measures, especially given that the concept of resilience has expanded to mean many things to many people. Resilience can be considered a process (as in building resilience), a rate (as in return time following perturbation), or, most appropriately, as an emergent property of complex systems of people and nature. The latter definition avoids normative determinations and is appropriate for objective quantitative measures. Return time is also amendable to quantification, but return time is a stationary concept that also discounts the potential for thresholds to alternative regimes and is thus at best a partial measure of resilience (measuring transient behavior occurring below thresholds). Understanding sources of uncertainty is the first step in reducing that uncertainty over time, the ultimate goal for environmental management.
We presented a simplified approach for resilience assessment, which incorporates measures of resilience (Walker and Salt 2006). This approach was designed to reduce uncertainty in resilience assessments, as well as to compare the relative resilience of different, large-scale watersheds. We tested the approach on four watersheds in North America, using a rapid prototyping approach that generated responses from stakeholders in each of the respective watersheds. The results of the study are for illustrative purposes only, due to small sample sizes, but the results indicate that the approach has significant potential for assessing relative resilience, trade-offs, and uncertainty in complex systems of people and nature.
The Nebraska Cooperative Fish and Wildlife Research Unit is jointly supported by a cooperative agreement between the U.S. Geological Survey, the Nebraska Game and Parks Commission, the University of Nebraska-Lincoln, the United States Fish and Wildlife Service, and the Wildlife Management Institute. Reference to trade names does not imply endorsement by the authors or the U.S. government. The views and opinions expressed in this article are those of the individual authors and the U.S Geological Survey, but not those of the US EPA and other sponsor organizations.
Adger, W. N. 2000. Social and ecological resilience: are they related? Progress in Human Geography 24:347-364. http://dx.doi.org/10.1191/030913200701540465
Allen, C. R., D. G. Angeler, G. S. Cumming, C. Folke, D. Twidwell, and D. R. Uden. 2016a. Quantifying spatial resilience. Journal of Applied Ecology 53:625-635. http://dx.doi.org/10.1111/1365-2664.12634
Allen, C. R., D. G. Angeler, A. S. Garmestani, L. H. Gunderson, and C. S. Holling. 2014. Panarchy: theory and application. Ecosystems 17:578-589. http://dx.doi.org/10.1007/s10021-013-9744-2
Allen, C. R., H. E. Birge, S. L. Bartelt-Hunt, R. A. Bevans, J. L. Burnett, B. A. Cosens, X. Cai, A. S. Garmestani, I. Linkov, E. A. Scott, M. D. Solomon, and D. R. Uden. 2016b. Avoiding decline: fostering resilience and sustainability in midsize cities. Sustainability 8:844. http://dx.doi.org/10.3390/su8090844
Allen, C. R., J. J. Fontaine, K. L. Pope, and A. S. Garmestani. 2011. Adaptive management for a turbulent future. Journal of Environmental Management 92:1339-1345. http://dx.doi.org/10.1016/j.jenvman.2010.11.019
Allen, T. F. H., J. A. Tainter, and T. W. Hoekstra. 2003. Supply-side sustainability. Columbia University Press, New York, New York, USA.
Angeler, D. G., and C. R. Allen. Quantifying resilience. 2016. Journal of Applied Ecology 53:617-624. http://dx.doi.org/10.1111/1365-2664.12649
Arnold, C. A., O. O. Green, D. Decaro, A. Chase, and J.-G. Ewa. 2014. The social-ecological resilience of an eastern urban-suburban watershed: the Anacostia River Basin. Idaho Law Review 51:29-90.
Bartolino, J. 2012. Middle Rio Grande Basin study. USGS New Mexico Water Science Center, Albuquerque, New Mexico, USA. [online] URL: http://nm.water.usgs.gov/projects/middleriogrande/
Benson, M. H., and A. S. Garmestani. 2011. Can we manage for resilience? The integration of resilience thinking into natural resource management in the United States. Environmental Management 48:392-399. http://dx.doi.org/10.1007/s00267-011-9693-5
Benson, M. H., D. Llewellyn, R. Morrison, and M. Stone. 2014. Water governance challenges in New Mexico’s Middle Rio Grande Valley: a resilience assessment. Idaho Law Review 51:195. http://dx.doi.org/10.2139/ssrn.2464387
Berkes, F., and C. Folke. 1998. Linking social and ecological systems for resilience and sustainability. Pages 1-25 in F. Berkes and C. Folke, editors. Linking social and ecological systems: management practices and social mechanisms for building resilience. Cambridge University Press, New York, New York, USA.
Biggs, R., M. Schlüter, D. Biggs, E. L. Bohensky, S. BurnSilver, G. Cundill, V. Dakos, T. M. Daw, L. S. Evans, K. Kotschy, A. M. Leitch, C. Meek, A. Quinlan, C. Raudsepp-Hearne, M. D. Robards, M. L. Schoon, L. Schultz, and P. C. West. 2012. Toward principles for enhancing the resilience of ecosystem services. Annual Review of Environment and Resources 37:421-448. http://dx.doi.org/10.1146/annurev-environ-051211-123836
Biggs, R., M. Schlüter, and M. L. Schoon. 2015. Principles for building resilience: sustaining ecosystem services in social-ecological systems. Cambridge University Press, Cambridge, United Kingdom. http://dx.doi.org/10.1017/CBO9781316014240
Birge, H. E, C. R. Allen, R. K. Craig, A. S. Garmestani, J. A. Hamm, C. Babbitt, K. Nemec, and E. Schlager. 2014. Social-ecological resilience and law in the Platte River Basin. Idaho Law Review 51:229-256.
Birge, H. E., C. R. Allen, A. S. Garmestani, and K. L. Pope. 2016. Adaptive management for ecosystem services. Journal of Environmental Management 183:343-352. http://dx.doi.org/10.1016/j.jenvman.2016.07.054
Carpenter, S., B. Walker, J. M. Anderies, and N. Abel. 2001. From metaphor to measurement: resilience of what to what? Ecosystems 4:765-781. http://dx.doi.org/10.1007/s10021-001-0045-9
Chaffin, B. C., A. S. Garmestani, L. H. Gunderson, M. H. Benson, D. G. Angeler, C. A. Arnold, B. Cosens, R. K. Craig, J. B. Ruhl, and C. R. Allen. 2016. Transformative environmental governance. Annual Review of Environment and Resources 41:399-423. http://dx.doi.org/10.1146/annurev-environ-110615-085817
Colorado Department of Local Affairs (CDLF). 2010. Colorado population forecasts. Colorado Department of Local Affairs, Denver, Colorado, USA. [online] URL: https://raqc.org/postfiles/board_meetings/2010/may07/StateDemographer050710.pdf
Cosens, B., and A. Fremier. 2014. Assessing system resilience and ecosystem services in large river basins: a case study of the Columbia. Idaho Law Review 51:91-125.
Folke, C., S. Carpenter, T. Elmqvist, L. Gunderson, C. S. Holling, B. Walker, J. Bengtsson, F. Berkes, J. Colding, K. Danell, M. Falkenmark, L. Gordon, R. Kasperson, N. Kautsky, A. Kinzig, S. Levin, K.-G. Mäler, F. Moberg, L. Ohlsson, P. Olsson, E. Ostrom, W. Reid, J. Rockström, H. Savenije, and U. Svedin. 2002. Resilience and sustainable development: building adaptive capacity in a world of transformations. ICSU Series on Science for Sustainable Development No. 3. International Council for Scientific Unions, Paris, France or Swedish Environmental Advisory Council, Ministry of the Environment, Stockholm, Sweden.
Godschalk, D. R. 2003. Creating resilient cities. Natural Hazards Review 4:136-143. http://dx.doi.org/10.1061/(ASCE)1527-6988(2003)4:3(136)
Holling, C. S. 1973. Resilience and stability of ecological systems. Annual Review of Ecology and Systematics 4:1-23. http://dx.doi.org/10.1146/annurev.es.04.110173.000245
Longstaff, P. H., N. J. Armstrong, K. Perrin, W. M. Parker, and M. A. Hidek. 2010. Building resilient communities: a preliminary framework. Homeland Security Affairs VI(3):1-23. [online] URL: https://www.hsaj.org/articles/81
Nemec, K. T., J. Chan, C. Hoffman, T. L. Spanbauer, J. A. Hamm, C. R. Allen, T. Hefley, D. Pan, and P. Shrestha. 2013. Assessing resilience in stressed watersheds. Ecology and Society 19(1):34. http://dx.doi.org/10.5751/ES-06156-190134
Nicolson, C. R., A. M. Starfield, G. P. Kofinas, and J. A. Kruse. 2002. Ten heuristics for interdisciplinary modeling projects. Ecosystems 5:376-384. http://dx.doi.org/10.1007/s10021-001-0081-5
Norris, F. H., S. P. Stevens, B. Pfefferbaum, K. F. Wyche, and R. L. Pfefferbaum. 2008. Community resilience as a metaphor, theory, set of capacities, and strategy for disaster readiness. American Journal of Community Psychology 41:127-150. http://dx.doi.org/10.1007/s10464-007-9156-6
Palmer, T. 2006. America by rivers. Island, Washington, D.C., USA.
Quinlan, A. E., M. Berbés-Blázquez, L. J. Haider, and G. D. Peterson. 2016. Measuring and assessing resilience: broadening understanding through multiple disciplinary perspectives. Journal of Applied Ecology 53:677-687. http://dx.doi.org/10.1111/1365-2664.12550
Resilience Alliance. 2010. Assessing resilience in social-ecological systems: workbook for practitioners. Version 2.0. Resilience Alliance. [online] URL: https://www.resalliance.org/files/ResilienceAssessmentV2_2.pdf
Resilience Alliance. 2013. Resilience assessment projects. Resilience Alliance. [online] URL: http://www.resalliance.org/resilience-analysis-practice
Walker, B., S. Carpenter, J. Anderies, N. Abel, G. Cumming, M. Janssen, L. Lebel, J. Norberg, G. D. Peterson, and R. Pritchard. 2002. Resilience management in social-ecological systems: a working hypothesis for a participatory approach. Conservation Ecology 6(1):14. http://dx.doi.org/10.5751/ES-00356-060114
Walker, B. H., N. Abel, J. M. Anderies, and P. Ryan. 2009. Resilience, adaptability, and transformability in the Goulburn-Broken Catchment, Australia. Ecology and Society 14(1):12. http://dx.doi.org/10.5751/ES-02824-140112
Walker, B., and D. Salt. 2006. Resilience thinking: sustaining ecosystems and people in a changing world. Island, Washington, D.C., USA.
Williams, B. K. 2001. Uncertainty, learning, and the optimal management of wildlife. Environmental and Ecological Statistics 8:269-288. http://dx.doi.org/10.1023/A:1011395725123
Williams, B. K. 2011. Passive and active adaptive management: approaches and an example. Journal of Environmental Management 92:1371-1378. http://dx.doi.org/10.1016/j.jenvman.2010.10.039
World Wide Fund for Nature (WWF). 2012. Resilience-building: a new approach to protecting Arctic ecological values. [online] URL: http://wwf.panda.org/who_we_are/wwf_offices/denmark/index.cfm?uProjectID=9Z1414