Ecology and SocietyEcology and Society
 E&S Home > Vol. 24, No. 2 > Art. 29
The following is the established format for referencing this article:
Fernández-Giménez, M. E., D. J. Augustine, L. M. Porensky, H. Wilmer, J. D. Derner, D. D. Briske, and M. Olsgard Stewart. 2019. Complexity fosters learning in collaborative adaptive management. Ecology and Society 24(2):29.
https://doi.org/10.5751/ES-10963-240229
Research

Complexity fosters learning in collaborative adaptive management

1Department of Forest and Rangeland Stewardship and Center for Collaborative Conservation, Colorado State University, Fort Collins, CO, USA, 2USDA-ARS Rangeland Resources and Systems Research Unit, Fort Collins, CO, USA, 3USDA-ARS Rangeland Resources and Systems Research Unit, Fort Collins, CO, USA., 4USDA-ARS Rangeland Resources and Systems Research Unit, Cheyenne, WY, USA, 5Department of Ecosystem Science and Management Texas A&M University, College Station, TX, USA

ABSTRACT

Learning is recognized as central to collaborative adaptive management (CAM), yet few longitudinal studies examine how learning occurs in CAM or apply the science of learning to interpret this process. We present an analysis of decision-making processes within the collaborative adaptive rangeland management (CARM) experiment, in which 11 stakeholders use a structured CAM process to make decisions about livestock grazing and vegetation management for beef, vegetation, and wildlife objectives. We analyzed four years of meeting transcripts, stakeholder communications, and biophysical monitoring data to ask what facilitated and challenged stakeholder decision making, how challenges affected stakeholder learning, and whether CARM met theorized criteria for effective CAM. Despite thorough monitoring and natural resource agency commitment to implementing collaborative decisions, CARM participants encountered multiple decision-making challenges born of ecological and social complexity. CARM was effective in achieving several of its management objectives, including reduced ecological uncertainty, knowledge coproduction, and multiple-loop social learning. CARM revealed limitations of the idealized CAM cycle and challenged conceptions of adaptive management that separate reduction of scientific uncertainty from participatory and management dimensions. We present a revised, empirically grounded CAM framework that depicts CAM as a spiral rather than a circle, where feedback loops between monitoring data and management decisions are never fully closed. Instead, complexities including time-lags, trade-offs, path-dependency, and tensions among stakeholders’ differing types of knowledge and social worlds both constrain decision making and foster learning by creating disorienting dilemmas that challenge participants’ pre-existing mental models and relationships. Based on these findings, we share recommendations for accelerating learning in CAM processes.
Key words: adaptive management; collaboration; environmental governance; knowledge coproduction; North American Great Plains; social learning

INTRODUCTION

Public rangeland managers in the American West are tasked with managing for multiple ecosystem services and balancing sometimes incompatible goals of stakeholders who range from ranchers to recreationists to environmental advocates, among others. Diverging expectations for public rangelands have bred decades of conflict (McGreggor Cawley 1993, Walker 2016); yet these sometimes fraught social and political landscapes also generate and serve as a testing ground for emerging management approaches grounded in collaboration and evidence-based adaptive management (Caves et al. 2013, Charnley et al. 2014, Wilmer et al. 2018a). Collaborative adaptive management (CAM) is one such emerging approach. Focused on learning through management experiments, CAM aims to improve environmental and social outcomes of resource management by engaging multiple stakeholders in structured, deliberative learning and decision making (Innes and Booher 2010). Learning is central to CAM, yet few longitudinal studies examine how learning occurs in CAM. Building on the adaptive management framework, CAM incorporates multiple interests and knowledge sources to reduce uncertainty and foster shared understanding of ecosystem dynamics and management outcomes. This process ideally leads to reduced conflict among stakeholders and identification of novel solutions that maximize mutual benefits. In reality, the benefits of CAM remain elusive (Susskind et al. 2012, Beratan 2014).

Our objective is to evaluate CAM processes, learning outcomes, and overall criteria for effectiveness hypothesized by existing CAM theory. We do so through a structured longitudinal ethnographic case study focused on decision-making processes within the collaborative adaptive rangeland management (CARM) project, a ranch-scale grazing management experiment ongoing in northeastern Colorado. We observed and collaborated with a group of stakeholders who practiced CAM as one experimental treatment in the CARM study. The design of CARM removed two important barriers to the success of CAM: lack of adequate monitoring and insufficient institutional commitment to implementing management decisions. This provided an opportunity to observe CAM under “best case” conditions. CARM differed from previous research on CAM by directly comparing the process and outcomes of stakeholder-managed treatments with those of a traditional rangeland management (TRM) treatment in matched and randomly assigned pastures. Using a qualitative, ethnographic approach, we traced the stakeholder group’s interaction with monitoring data and their decision-making processes to compare the observed empirical CAM processes and outcomes with those theorized in the literature (as synthesized in Fig. 1). We focus on four research questions: (1) What facilitated and challenged the CAM decision-making process within CARM? (2) How did these challenges affect learning within CARM? (3) How well does a “best case” CAM scenario with intensive monitoring and agency commitment to implement stakeholder decisions fulfill the theorized criteria for effective CAM? (4) How may lessons from the CARM experiment enhance the CAM theoretical framework and lead to greater effectiveness of future CAM efforts? This research fills a critical gap in our understanding of how theorized CAM processes operate in practice.

We worked from a comprehensive CAM theoretical framework, and used the concept of “social worlds” to understand how individuals from different social contexts interact. Adaptive management (AM) addresses simultaneous needs to take management action and reduce ecological uncertainties through experimentation and learning (Holling 1978, Lee 1993, Williams 2011a, Westgate et al. 2013). AM is typically depicted as a six-step cycle (Nyberg 1999, Rist et al. 2013; Fig. 1a) involving framing the resource problem; making an explicit conceptual model; identifying management objectives and alternatives; and designing management experiments and monitoring protocols. In the iterative phase managers use these elements in a cyclical manner to learn about the system and make decisions based on inferences from monitoring results. The AM literature theorizes a relatively direct pathway from scientific monitoring data to new knowledge, and subsequently to decision making (Fig. 1a).

The clarity and logical order of the iterative AM cycle may contribute to its wide adoption as a management framework, but investigations of the underlying conceptual models, monitoring protocols, or comparisons to other management approaches are rare (Aldridge et al. 2004, Westgate et al. 2013). Absence of feedbacks among monitoring, learning, and management, often referred to as “closing the loop,” is a major reason AM often fails to meet expectations in practice (Moir and Block 2001, Aldridge et al. 2004, Westgate et al. 2013). Barriers to successful AM include a mismatch of spatial or temporal scales of information and decision making (Williams and Brown 2016), cooptation by extractive interests, lack of institutional commitment (Stankey et al. 2003), insufficient monitoring (Moir and Block 2001, Williams and Brown 2016), failure to act, and insufficient stakeholder participation (McLain and Lee 1996, Allen and Gunderson 2011).

Collaborative Adaptive Management (CAM), also known internationally as adaptive comanagement (Armitage et al. 2009), aims to overcome limited stakeholder involvement (Kusel et al. 1996). Drawing on the broader trend toward greater collaboration and participation on natural resource management (Wondolleck and Yaffee 2000), CAM seeks to achieve desired management objectives while also fostering social learning and other collaboration benefits. Diverse stakeholder participation in AM has several potential benefits. First, stakeholders bring unique backgrounds, knowledge, and experiences with the local ecosystem (Innes and Booher 2010). Second, engaging stakeholders broadens the opportunity for shared learning within the group and diffusion through the larger community (Fernandez-Gimenez et al. 2008). Third, CAM offers benefits of increased trust, social capital, and collaborative capacity, and potential for novel and legitimatized solutions to jointly defined problems (Daniels and Walker 2001, Innes and Booher 2010, Lubell 2015). Stakeholder engagement in the process of AM offers a potential solution to the fundamental tension in environmental governance between science-based and participatory decision making. Increased involvement of stakeholders may also help distinguish goals and objectives from actions and indicators, and data interpretation from learning (Fig. 1b; Delta Stewardship Council 2013).

Criteria for assessing CAM effectiveness include (1) reducing ecological uncertainty, (2) achieving desired management outcomes, (3) using monitoring data in adaptive decision making, (4) incorporating multiple stakeholder interests and perspectives, (5) building trust and collaborative capacity, and (6) multiple-loop learning (defined below). Although examples of successful CAM exist (Caves et al. 2013), CAM has its own set of challenges. These include difficulties surrounding stakeholder participation and engagement; institutional barriers to implementation; poorly designed management processes; and insufficient monitoring of management outcomes, learning processes, and learning outcomes (Armitage et al. 2009, Cundill et al. 2012, Beratan 2014, Fabricius and Cundill 2014).

CAM seeks to advance social learning, yet “learning” is often used vaguely and uncritically, with little attention to the social processes or the learning mechanisms (Armitage et al. 2008). Social learning has origins in psychology (Bandura 1977) and organizational management (Argyris and Schon 1978), and broad application in education. In the natural resource context social learning has been variously defined as a process of learning through collective, iterative reflection (Keen et al. 2005, Fernandez-Gimenez et al. 2008), or a learning outcome, such as a change in conceptions or attitudes, which becomes social learning when it is diffused through a larger network through social interactions (Reed et al. 2010). Here, we follow Baird et al. (2014) in defining social learning as conceptual, normative, or relational changes that occur through interactions among individuals within a group. Cognitive learning is the acquisition or restructuring of knowledge, including the process of conceptual change (Baird et al. 2014). The science of learning provides insights into how learning occurs, often overlooked in the AM literature, which typically depicts simple feedbacks among scientific data, new knowledge, and decision making (Fig. 1a). Yet, all learning builds on prior experience (Kolb 1984, Novak 2010), and occurs in a social and cultural context (Lave and Wenger 1991). Further, changes in understanding whereby existing conceptions are replaced by new ones often involve “cognitive struggles” (Bransford et al. 2006), comprising disorienting dilemmas, critical reflections, and reflective discourses (Pennington et al. 2013). Conceptual change is a shift in an individual’s “mental model,” or personal, internal, tacit representation of reality used to navigate the world (Jones et al. 2011). Normative learning denotes changes in values, norms, or paradigms, and convergence of group opinions. Relational learning refers to improved understanding of others’ worldviews, relationship-building, and increased trust and cooperation (Baird et al. 2014). These three types of learning are sometimes associated with the three levels of multiple-loop learning: (1) learning about cause and effect relationships (single-loop learning), (2) reassessing assumptions or mechanisms (double-loop learning; Petersen et al. 2014), and (3) revising values, norms, or governing structures that underlie assumptions and actions (triple-loop learning; Keen et al. 2005). Whereas single-loop learning generally represents change in cognition or conceptual change, double- and triple-loop learning represent normative and relational shifts (Baird et al. 2014).

The notion of social learning implies the interaction of individuals with potentially divergent knowledge or attitudes, whose conceptions, norms, and relationships may shift as a result of their interactions. In complex natural resource settings, CAM entails the encounter of individuals from different “social worlds,” or “universes of discourse” where meaning is created through interaction of actors (Strauss 1978, Clarke and Star 2008). When these worlds encounter each other in the arena of collaboration, conflict often results, creating opportunities to negotiate and construct shared meaning across different social worlds. Although encounters of different knowledge types, users, and sources have been addressed in the CAM and adaptive comanagement literature (Berkes 2009, Plummer et al. 2012), these dynamics have not been understood or analyzed as an encounter of social worlds.

Studies of social learning in natural resources and CAM are proliferating, drawing on a range of methodologies from comparative case studies (Fernandez-Gimenez et al. 2008, Petersen et al. 2014); experiments (Fujitani et al. 2017); and qualitative (Bentley Brymer et al. 2018), quantitative (Leys and Vanclay 2011, Baird et al. 2016), and participatory (Leys and Vanclay 2011) assessments of learning in a series workshops or collaborative meetings. Increasingly, such research includes a control group of stakeholders who did not receive a treatment aimed to increase social learning. However, few studies trace a CAM process closely over multiple years, with close and continual engagement between the research team and participating stakeholders and detailed observations of decision-making processes. Here, we report on an ethnographic case study of decision-making processes within a single case of CAM, the CARM project, in which 11 stakeholders used a structured CAM process to make decisions about livestock grazing and vegetation management for beef, vegetation, and wildlife objectives. We analyzed four years of meeting transcripts, stakeholder communications, and biophysical monitoring data to ask what facilitated and challenged stakeholder decision making, how challenges affected stakeholder learning, and whether CARM met theorized criteria for effective CAM. Rather than an experiment, this study is an in-depth, longitudinal case study grounded in observation and qualitative analysis of stakeholder decision making and outcomes within an intensively monitored instance of CAM.

METHODS

The experimental component of the project compares two different livestock grazing management strategies (CARM versus TRM, described in detail below). The experiment was motivated by the contrast between the findings of small-scale experimental studies that have consistently shown that rotational, i.e., multipaddock, grazing does not enhance vegetation or animal performance compared to continuous, season-long grazing (reviewed by Briske et al. 2008, Hawkins et al. 2017) versus the findings of some rangeland and social scientists, working at the scale of ranching enterprises, who report ecological and economic benefits from various forms of rotational grazing (Sherren et al. 2012, Teague and Barnes 2017). A key unanswered question is whether the adaptive component of rotational grazing, whereby managers move livestock in response to short-term variation in weather and forage availability across a landscape, can effectively generate desired ecosystem services (Teague and Barnes 2017). We designed an experiment at the United States Department of Agriculture (USDA) Agricultural Research Service’s Central Plains Experimental Range, a Long-Term Agro-ecosystem Research (LTAR) network site located on the shortgrass steppe of eastern Colorado, which could address this question at the scale of a property that is representative of many livestock production operations within the broader region (Kachergis et al. 2013, Wilmer et al. 2018b). The majority of this study site consists of two soil-plant community associations (the Loamy Plains and Sandy Plains ecological sites as defined by USDA-NRCS 2007a, b) which are broadly distributed across a 5.2 million ha region of the North American Great Plains (USDA-NRCS 2006).

Our analyses of stakeholders’ decision making and learning in CARM are based on qualitative analysis of ethnographic data on stakeholder interactions and activities. These analyses relate to the experimentally derived ecological data from paired CARM and TRM grazing treatments. We focus on the first three cycles of collecting, analyzing, and interpreting ecological monitoring data from the grazing experiment, which includes a year of group formation and objective setting (2012), a baseline pretreatment year (2013), and two and a half years of adaptive management (2014 and 2015), including stakeholder evaluation of 2015 monitoring data and planning for the 2016 grazing season. Extensive qualitative and biophysical data produced during this time span provide robust answers to our questions about decision making within CARM and implications for the CAM theoretical framework, even as the broader CARM project continues. Our ethnographic case study methodology does not include carrying out observations of a control or counterfactual group that makes decisions using a different, non-CAM process. Further, our social data collection does not control for the influence of outside factors and participant experiences on our results, but rather traces such influences through our qualitative analysis, in keeping with case study methodology that examines a phenomenon in context (Yin 1981).

The CARM stakeholder group

We identified potential organizations to participate in this study from researchers’ existing professional networks, with the aim of creating an 11-member stakeholder group that would include balanced representation from three main stakeholder types: livestock producers, conservation organizations, and natural resource management agencies. We invited the Crow Valley Livestock Cooperative, whose members are all ranchers that graze livestock on the adjacent Pawnee National Grassland and also provide the cattle for experiments at the USDA - Agricultural Research Service’s Central Plains Experimental Range (CPER), to appoint four of their members to serve as stakeholders for the CARM experiment. We also invited three NGOs that promote biodiversity conservation in eastern Colorado rangelands (Environmental Defense Fund, Bird Conservancy of the Rockies, and The Nature Conservancy), three natural resource management agencies (USDA-Forest Service, USDA-Natural Resources Conservation Service, and Colorado State Land Board), and Colorado State University Extension to each appoint one representative to the stakeholder board. These ranchers and organizations represented regionally important perspectives on ranching, rangeland, and wildlife management. All individuals had a professional stake in the management of the shortgrass steppe, an ecosystem increasingly challenged by changing land-use, public demands, and variable climate and markets.

Stakeholders were first charged with establishing and prioritizing management goals and objectives for land that they would be collectively managing. Stakeholders and researchers met for two days in fall 2012 to establish an overall goal and set of objectives. Objectives for vegetation composition and structure centered around increasing the abundance of C3 perennial grasses (particularly western wheatgrass, Pascopyrum smithii), which produce more forage early in the growing season than C4 grasses and can provide nesting cover for certain bird species, but can also decline with increasing grazing intensity (Milchunas et al. 2008, Skagen et al. 2018). Objectives for wildlife conservation focused on a suite of grassland birds that require habitat across a spectrum of structures, resulting from interactions of fire, grazing, rest, and precipitation. Species occurring along this spectrum of habitat types were recognized by the group as facing population declines across their ranges in the Western Great Plains (Brennan and Kuvlevsky 2005). Ranch profitability objectives emphasized drought resilience and livestock production, following key concerns from the local ranching context (Table 1).

CARM study design

The research team established 10 pairs of experimental pastures (130 ha per pasture; 2600 ha total area of the experiment), matched on soil and vegetation characteristics (Fig. 2). One pasture in each pair was randomly assigned to a stakeholder-managed CARM treatment, and the other to a traditionally managed (TRM) treatment. Thus, the CARM field experiment used an “active” adaptive management approach (Williams 2011b, Murray and Marmorek 2003) that compared two alternative management treatments, a stakeholder-managed CARM treatment, and a traditionally managed TRM treatment. Within the CARM treatment, stakeholders used within-season and annual biophysical monitoring data to set “triggers” for livestock movement decisions before each grazing season, evaluate the outcomes of using those triggers after each grazing season, and adaptively alter the triggers for the next grazing season based on these outcomes. In this way, CARM incorporates elements of both “active” adaptive management that evaluates multiple management treatments simultaneously, and “passive” or sequential adaptive management that adjusts treatments over time based on monitoring.

In January 2013, stakeholders outlined the initial grazing management approach for the CARM pastures to include the following: (1) management of CARM cattle as a single herd rotated among pastures within a given year, (2) resting (not grazing) two of 10 CARM pastures each year, (3) annually establishing the sequence in which the remaining eight pastures are grazed, along with measurable triggers to determine exact timing of cattle rotations among pastures, and (4) annually deciding whether to apply additional vegetation treatments such as prescribed fire or herbicide. Stakeholders also helped identify initial monitoring indicators and measures to assess progress toward stakeholder-defined objectives. After baseline data collection in 2013, the grazing plan was implemented in 2014. Following the initiation of treatments in 2014, stakeholders and scientists met at least four times per year (October, January, April, June), to review monitoring data, observe pastures during and after each grazing season, and make annual decisions on whether and how to adjust stocking rate, sequence of cattle use of each of the 10 CARM pastures, the criteria/triggers used to determine when to rotate cattle in response to forage conditions, and whether and under what conditions to implement other vegetation treatments, e.g., prescribed fire (Fig. 3).

Management of collaborative processes requires attention to factors shaping engagement outcomes, including group power dynamics, local political contexts, and science-manager relationships (Connell 1997, Daniels and Walker 2001, Reed et al. 2018). After the initial 2012 workshop, facilitated by an outside facilitator, the research team facilitated all stakeholder meetings to ensure long-term continuity in facilitation and build on existing relationships of trust between researchers and participants. Facilitation aimed to ensure stakeholders had ample time to discuss management issues and that all voices were heard during meetings. The stakeholders reviewed monitoring data, discussed management outcomes, toured experimental pastures, and made decisions about future management approaches and collaborative processes (Fig. 3). Detailed analysis of the participatory process will be evaluated in a future manuscript; here we focus on stakeholder decision making. The group sought consensus on all decisions, but agreed to take an action if 75% of a quorum of stakeholders supported it. During the study period, one NGO stakeholder seat turned over, and one government seat was vacant from 2014-2015. Researchers implemented stakeholder decisions, monitored outcomes, communicated outcomes to stakeholders, and documented decision-making processes.

In all years, each of the 10 TRM pastures were grazed season-long by 10% as many cattle as the CARM herd number. Thus, the two treatments differed in the spatiotemporal pattern of grazing but not annual stocking rate, with CARM pastures experiencing 10-fold greater stocking density on days when they are grazed as compared to TRM pastures.

Ecological data

To measure vegetation and bird responses (Table 1), we established four monitoring plots in each pasture. In six pastures with unique lowlands that support salt-tolerant grasses (Alkalai sacaton and Distichlis spicata; “saltflats”), we established two additional plots per pasture in the saltflats. Plots (N = 92) were located on the same dominant soil types and topographic positions within each CARM/TRM pair. Bird point counts (N = 92 points monitored annually) were conducted by an observer standing at the center of each plot, who recorded the species, sex, and distance to each bird detected during a six-minute interval (Hanni et al. 2013). Counts were repeated on two occasions each June (corresponding to the peak of the breeding season) by two observers, where one observer was constant across years. Here, we report on two indicies of bird diversity (the Shannon diversity index [H'; Spellerberg and Fedor 2003] and species eveness [H'/Richness; Tuomisto 2012]), based on detections of birds, excluding raptors and waterfowl, truncated at 125 m radius around points. We did not adjust counts for detection rates because our focus is on relative differences in diversity indices between treatments rather than absolute abundance of any given bird species. For plants, we measured foliar and basal cover in June along four transects per plot using line point intercept (Herrick et al. 2009). Because one objective of the stakeholders was to enhance the abundance of C3 relative to C4 grasses, we also measured densities of tillers of the most abundant C3 perennial grass in June in 32, 0.25 m² circular quadrats per plot. Vegetation height-density was measured at 64 locations per plot at the conclusion of the grazing season (early October) using the vegetation visual obstruction method (Robel et al. 1970), modified to use increments of centimeters instead of decimeters. Cattle were weighed at the beginning and end of each grazing season, with weight gains per animal (kg/steer) determined using shrunk weights (Derner et al. 2016).

Social data

We selected an ethnographic approach to evaluate the collaborative management process because the approach is well-suited to study the cultures and contexts of land management communities and decision makers. For example, McGoodwin (2001) and Nightengale (2013) both used this approach to investigate the subjective and cultural aspects of fishing communities, and were able to offer clear recommendations for policy makers, government agencies, and outreach professionals to enhance culturally relevant engagement with these communities. To conduct our ethnography, we gathered primary qualitative data sources, including transcribed audio recordings and in-depth field notes of each stakeholder meeting, stakeholder surveys and interviews, and a transcripts from a focus group (2016). Other data included email correspondence, multiple versions of grazing management plans, and presentations from researchers (CSU IRB protocol 12-3381H). Interviews covered stakeholder backgrounds, reflections on their experiences and learning in CARM, feedback on CARM processes, and stakeholder perceptions of ecological factors influencing progress toward CARM objectives. During the focus group, researchers relayed initial patterns from the interviews to the stakeholders, and stakeholders discussed their views of CARM processes, uncertainties, and changes in CARM facilitation and decision-making processes. We used directional content analysis to examine meeting notes, and meeting, stakeholder interview, and focus group transcripts (Hsieh and Shannon 2005). We also constructed a detailed time-line of CARM events and decisions. Details of qualitative analysis and the graphical timeline are available in Wilmer et al. (2018a).

Linking social and ecological data

We held a series of interdisciplinary research team workshops to answer the four research questions identified for this case study. To understand what facilitated and challenged the CARM decision-making process (Question 1) and how these challenges affect learning within CARM (Question 2), we used the project timeline and research team discussion to identify five complexities observed in CARM. These included ambiguities, trade-offs, time lags, and uncertainties in biophysical data and the correlated stakeholder discussions and decisions. As we identified and reflected on these complexities as a research team, our interpretation of the complexities evolved from viewing them as barriers to decision making to understanding them as complexities that ultimately facilitated learning precisely because they challenged stakeholders’ (and research team members’) existing mental models/conceptions of specific biophysical relationships and the CAM process overall. To test this emergent theory, we traced decisions associated with each identified complexity to observe instances of single- and multiple-loop learning and indicators of growing trust and collaborative capacity in transcripts of the group discussion and stakeholder interviews (George and Bennett 2005). We synthesized the outcomes from these complexities to evaluate how well a best case CAM scenario of intensive monitoring and agency commitment fulfilled the theoretical criteria for effective CAM (Question 3). Finally, we revised the CAM theoretical framework based on our learning and developed recommendations for future CAM practice from our experiences (Question 4).

Our single case study and purposeful sampling of rangeland management stakeholder organizations was designed for deep and nuanced understanding of the qualitative aspects of the CAM process, and thus precludes wide-scale generalization to all rangeland management contexts. We enhanced rigor of the qualitative analyses by considering alternative explanations for our findings, reviewing initial results with research participants, and documenting our decisions in a series of memos, meeting notes, and emails (Lincoln and Guba 1986, Moon et al. 2016). From the outset of our study, we used an ethnographic approach to evaluate the collaborative management process. However, we note that researchers’ view of our role in CARM events evolved from a view of ourselves as outside observers to one recognizing our interests and (nonvoting) involvement in learning and decision making. Our individual positions and team experiences shaped the interpretation of those events as reported in this study. Thus we consider the researchers as nonvoting stakeholders in the study, and our results refer to learning by the full CARM team consisting of both stakeholders and researchers.

RESULTS

Complexity inherent in the CARM process challenged stakeholders’ ability to “close the adaptive management loop” by interrupting the feedbacks among monitoring, single-loop learning, and management theorized by the existing AM and CAM conceptual frameworks (Fig. 1). More importantly, various forms of complexity enabled multiple-loop social learning by stimulating normative and relational learning. Below we describe five forms of complexity observed in CARM and the learning processes spurred by each. These complexities frame the partial effectiveness of CAM observed in the case of CARM.

Trade-offs between multiple objectives

During the first three years, clear trade-offs emerged among stakeholder-defined objectives (Table 1). We focus here on the trade-off between a profitability objective (beef production) and a vegetation/wildlife objective (increasing heterogeneity in vegetation structure to enhance grassland bird habitat). CARM steers gained ~15% less weight per animal per day than TRM steers in both 2014 and 2015 (Fig. 4). The CARM treatment created more among-pasture heterogeneity in vegetation structure than the TRM treatment (Fig. 5). Because of two consecutive years of above-average spring precipitation, forage production exceeded livestock demand such that the CARM herd’s forage requirements were met by grazing only 7 of 10 pastures in 2014 and 4 of 10 pastures in 2015. Thus, 30–60% of the landscape under CARM management was ungrazed in 2014 and 2015. The combination of ungrazed and intensively grazed pastures led to high variability in vegetation height across pastures (Fig. 5). Based on their familiarity with recent research on heterogeneity in rangelands (Fuhlendorf et al. 2006, Toombs et al. 2010), some stakeholders argued that structural heterogeneity indicated improved habitat in ungrazed pastures for grassland bird species such as Lark Buntings (Calamospiza melanocorys) and Grasshopper Sparrows (Ammodramus savannarum). Stakeholders hypothesized this habitat would lead to more overall bird diversity in the CARM pastures. Stakeholders and researchers interpreted these monitoring results as evidence of a clear trade-off, whereby objectives for vegetation and bird habitat were achieved at the cost of reduced beef production and ranch income.

This trade-off among objectives challenged the assumption that beef production, vegetation, and wildlife objectives could be met simultaneously and it had clear impacts on decision making in 2014 and 2015. Stakeholders were concerned about reduced per animal weight gain and worked to mitigate the reduction. After the 2014 and 2015 grazing seasons, stakeholders made incremental adjustments to the stocking rate, the sequence in which CARM pastures were grazed, and the triggers used for moving the CARM herd among pastures with the goal of improving beef production (Wilmer et al. 2018a). These changes were indicative of the classic AM cycle, or single-loop learning (Fig. 1a). After two growing seasons, the CARM team found that incremental adjustments were not achieving the expected or desired results.

When incremental adjustments to stocking rates and grazing rotations failed to fully close the beef production gap between treatments, researchers asked stakeholders to clarify and communicate their mental models of how beef production should respond to different management approaches (Wilmer et al. 2018a). This process exposed key areas of uncertainty and disagreement about underlying biological processes. For example, stakeholders had different hypotheses about how stock density and rotation speed among pastures influence cattle grazing behavior, and in turn, beef production and vegetation outcomes. These discussions represented an example of double-loop learning in which the CARM team began to examine and challenge their assumptions about underlying biophysical relationships. With insights into the ways different stakeholders justified CARM decisions, researchers designed new subexperiments to test key uncertainties.

After the second growing season, monitoring evidence demonstrated that a previous prescribed burn had both improved forage quality and enhanced bird habitat. Several conservation NGO representatives interpreted these data as indicating that burning could potentially mitigate the problematic trade-off between beef production and wildlife habitat objectives. Although some ranchers initially expressed willingness to experiment with burning, they ultimately voted unanimously against burning any pastures in the fall of 2015. This decision frustrated some conservation stakeholders who perceived that factors other than scientific evidence influenced the no-burn decision, which contradicted their understanding of how the CAM “learning to action” feedback should work (Fig. 1). Interviews with stakeholders, followed by reflective discussion by the research team and stakeholders, led researchers to better understand how ranchers’ social world influenced their decisions. Ranchers’ decision about prescribed burning was based on their experiences, values, and shared intergenerational knowledge of how to manage cattle in a drought-prone semiarid grassland. Rancher stakeholders perceived that burned grass was “wasted” forage, and that burning presented an unnecessary economic risk, in terms of loss of forage quantity, to the ranch enterprise in the event of future drought. Ranchers perceived that the stakeholder group had privileged vegetation and wildlife objectives over beef production objectives during the first two years of the experiment (Wilmer et al. 2018a). Monitoring data indicating that burning improved forage quality was insufficient to offset rancher perceptions of the risk associated with burning forage that might be needed in a future drought.

By 2016, stakeholder discussions about trade-offs between wildlife habitat and beef production, and associated discussions about burning, led to a need to clarify decision-making procedures. Some stakeholders were unclear whether their role was to advocate for specific interests, e.g., ranchers advocate for cattle production objectives, or to contribute toward achieving all shared objectives. After discussion, stakeholders formally agreed that the goal is to achieve all objectives rather than advocating for individual interests, and to consider how each alternative for a given decision might influence each of the three objective categories. This decision indicates the creation of a “CARM social world,” where stakeholders from previously disparate social worlds brought knowledge and values from their individual social worlds into a group discussion designed to weigh options and work toward achieving multiple objectives. This shift represented a significant instance of multiple-loop learning, underpinned by a change in group norms (normative learning) and relationships (relational learning).

Time lags constrain science-based decision making

Decision making was continually constrained by different parts of the biophysical system responding at varying rates to specific management actions. This led to disparate availability of monitoring data for various management objectives. For example, impacts of grazing management on beef production were clear at the end of each grazing season, but impacts of a given season’s management on vegetation composition and bird diversity had lag times of over one year. Results for vegetation composition were statistically uncertain (Fig. 6) and considered “in progress.” Thus, although data from the first two years suggested trade-offs between beef production and vegetation composition (Figs. 4 and 6), time lags made it difficult to assess the strength of this trade-off. The same was true for bird diversity. Monitoring data collected in 2015 provided information about potential responses of bird habitat to management actions, but these data were inconclusive (Table 2). Because of ecologically driven time lags, levels of uncertainty about outcomes of specific management actions varied across objectives (Fig. 7).

Lags in data availability for vegetation composition and birds had important consequences for decision making. Meeting transcripts indicated that stakeholders prioritized closing the beef production gap between the CARM and traditional treatments (Fig. 4). In the absence of definitive results, when discussing how management might affect vegetation composition outcomes, stakeholders referred to their pre-existing hypotheses and professional knowledge about management-vegetation relationships, e.g., resting a pasture early in the growing season will benefit C3 grasses, rather than to knowledge generated within the experiment. Stakeholders made incremental changes to their management approach that were consistent with their hypotheses. Overall, different rates of learning and adaptive action possible for different objectives and decisions created a complex set of decision-making trade-offs that interrupted the simple feedbacks among management, monitoring, and learning hypothesized by the CAM conceptual diagram (Fig. 1b). These trade-offs mirror those that challenge many rangeland managers.

Spatial trade-offs among objectives

Stakeholder-developed objectives included enhancing within- and among-pasture heterogeneity in vegetation structure (Table 1). These objectives were advocated by stakeholders from conservation NGOs on the basis that maximizing habitat heterogeneity allows grasslands to support the greatest diversity of plant and animal taxa (Tews et al. 2004), including grassland birds (Fuhlendorf et al. 2006, Derner et al. 2009), and has additional potential benefits for beef production (Fynn 2012). After the 2014 grazing season, among-pasture variance was 58% greater and mean within-pasture variance was 259% greater for CARM than for TRM pastures (Fig. 5a,c). However, following treatment implementation in 2015, grazed CARM pastures were uniformly defoliated to low vegetation structure, while rested CARM pastures retained uniformly taller vegetation than paired TRM pastures. The contrast between intensively grazed versus rested CARM pastures in 2015 led to 107% greater among-pasture variance in CARM versus TRM. Within-pasture variance in CARM pastures declined to a level 14% below that of TRM pastures (Fig. 5b,d). Differences between treatments in among-pasture variance cannot be compared statistically, but the difference in within-pasture variance was marginally significant in 2014 (Fig. 5c; paired t9 = 1.66, P = 0.065), and insignificant in 2015 (Fig. 5d; paired t9 = 0.11, P = 0.45). Results from 2015 suggested that CARM can sustain high among-pasture heterogeneity via intensive grazing in some pastures and no grazing in others, but this strategy may be incompatible with the objective of increasing within-pasture heterogeneity. This spatial trade-off reflects the real-world complexity involved in adaptively managing landscapes.

Stakeholders grappled with this spatial trade-off extensively. Discussions about within- vs. among-pasture heterogeneity led to more explicit discussion of the pasture-scale implications of management actions relative to ranch-scale goals and objectives. Stakeholders asked for novel data analyses to address the heterogeneity objective at multiple scales. During meetings preceding the 2015 and 2016 grazing seasons, stakeholder management decisions favored among-pasture over within-pasture heterogeneity. As this tendency became clear to the group, stakeholders openly questioned whether it was possible to achieve both heterogeneity objectives and converged on the opinion that among-pasture heterogeneity may be more important to CARM, demonstrating a clear example of normative learning.

Temporal trade-offs among objectives reveal learning opportunity costs

One objective identified by stakeholders was drought resilience (Table 1), which is a key element of long-term ranch financial sustainability in highly variable semiarid environments (Hamilton et al. 2016). The main strategy stakeholders used to increase drought resilience was to store reserve forage in ungrazed pastures. Ungrazed CARM pastures (three in 2014 and six in 2015) contained reserve forage that was hypothesized to mitigate the impacts of a future drought by enabling ranchers to maintain herd size and limit supplemental feed expenses. Some stakeholders recognized that reduced beef production, i.e., short-term loss, from the CARM herd compared to the TRM herd during the high-rainfall years of 2014 and 2015 might trade off with longer term financial viability achieved via reserve forage in rested pastures. However, the strength and importance of this trade-off remained highly uncertain because drought did not occur during the study (Fig. 6).

Stakeholders and researchers returned to this hypothetical temporal trade-off during decision-making discussions throughout the study period. Although this trade-off has not yet been observed in CARM, the possibility of its existence motivated stakeholders to continue with management strategies that maintain forage reserves. One strategy for increasing beef production in CARM would be to dramatically increase the stocking rate. Yet stakeholders were reluctant to take this action and lose the forage reserve. Thus, despite data showing lower ranch-scale beef production in CARM than TRM herds, stakeholders did not take major actions to increase beef production, i.e., close the loop, but instead took an approach expected to be less successful in the short-term to increase hypothesized long-term financial stability in the face of future drought. Temporal trade-offs characterized by participants balancing and prioritizing uncertain long-term outcomes over more certain short-term outcomes challenge the simplistic, sequential decision-making process portrayed in AM and CAM cycles (Fig. 1).

The identification of this temporal trade-off between short-term losses and hypothetical future drought resilience led stakeholders and researchers to identify a learning opportunity cost, or the trade-off between learning opportunities and actions to improve ecological or economic outcomes. After two growing seasons, some stakeholders wanted to alter the CARM grazing strategy to improve livestock weight gains. One option was to split the single CARM herd into two herds, based on the hypothesis that the experimental stocking density was too high, but that an intermediate stocking density could create the triple-win, or as one stakeholder described it, a “sweet spot,” for all objectives. A reduction in stocking density should increase individual animal weight gains, which would result in more total beef production in the short-term. However, reducing the stocking density (by splitting the single CARM herd into two or more herds) would eliminate the opportunity to learn whether reserve forage created by resting pastures in high-rainfall years would increase drought resilience and long-term ranch financial viability. Reducing stocking density would also eliminate the opportunity to learn how the higher stocking density treatment performs over the long term (Fig. 8).

The recognition of additional hidden costs associated with changing management due to short-term findings represented another example of multiple-loop learning within the CARM project. Path dependency is an inherent feature of CAM and AM because actions taken early in the process constrain future decision-making opportunities related to doing (achieving objectives) and learning (about current or alternative management strategies). Stakeholders repeatedly struggled with learning opportunity costs at meetings, with transcripts suggesting that stakeholders wished to reduce the stocking density, but also wanted to observe outcomes of the current strategy during a drought year. Learning opportunity costs are yet another feature of CARM that challenges the narrative of a simple, sequential AM/CAM cycle.

Complexities of collaboration among diverse social worlds

CAM is anchored on the idea that stakeholders use monitoring data to make informed management decisions. Decision-making processes in CARM, however, were strongly influenced by the diverse histories, experiences, types of knowledge, and values of participating stakeholders, whose social worlds shaped how they related to and valued scientific data. Stakeholders also brought a variety of learning styles, information sources, and ways of visualizing and interpreting ecological processes to the project. The extent to which stakeholders’ social worlds influenced their encounters with collaborative processes are illustrated in three touchstone examples of conceptual, normative, and relational change, i.e. multiple-loop learning.

Early in the project, stakeholders recognized the need for frequent monitoring of cattle behavior and condition during the grazing season. They discussed how to incorporate local knowledge of cattle condition into the formal monitoring protocols. This prompted technicians employed on the project to develop a protocol for cattle behavior monitoring based on their local knowledge of cattle ranching. They used this protocol to regularly document animal behavior and body condition. The technicians presented the protocol and initial data to stakeholders in both a descriptive format and a scientific format, which allowed for more effective communication with stakeholders from different backgrounds.

Meeting transcripts indicated that the creation and use of the cattle monitoring protocol facilitated collective learning and led to stakeholders’ increased engagement with and understanding of other quantitative monitoring data. At the January 2016 meeting, when cattle behavior monitoring data were displayed alongside data for diet quality and forage biomass, stakeholders engaged more meaningfully with monitoring data than they had in previous meetings. Stakeholders began to discuss a more integrated picture of forage-livestock interactions and their implications for cattle rotation decisions. Stakeholder discussions at this meeting demonstrated to researchers, disappointed by the lack of reference to scientific data in previous meetings, that stakeholders were gaining interest in the scientific evidence, and increasingly valuing and using it for decision making. Following this discussion, stakeholders requested additional “data day” meetings to further explore the value and meaning of monitoring data. This touchpoint indicates a feedback between valuing local knowledge salient to stakeholders, and stakeholders’ increased engagement with and use of scientific monitoring data.

Several acts of reciprocal sharing among stakeholders and researchers occurred outside of regular meetings between fall 2015 and summer 2016. A rancher co-led a wildlife conference field trip with two researchers, an agency stakeholder invited researchers on an agency field trip, and a rancher invited stakeholders and researchers to attend his family’s spring branding. In each case, stakeholders invited others to participate in their social world and shared their knowledge in a different context. These acts of reciprocity began to lay a foundation of mutual respect and understanding for different stakeholders’ social worlds. Transcripts show that in meetings, stakeholders increasingly were willing to see each other’s points of view on decisions and objectives, illustrating relational learning.

After the 2015 grazing season, researchers began to engage deliberately in their own process of reflective adaptive management (Roux et al. 2010) via discussions about epistemological differences and challenges of working with multiple disciplines and stakeholders. Through this dialogue, researchers situated themselves and the stakeholders within the social worlds and social learning frameworks. This led to discussions of researcher influence on stakeholder decision making and how to facilitate collaboration more effectively. Researchers facilitated a reflective focus group at the stakeholder meeting in April 2016. This resulted in a constructive discussion about group goals, roles of stakeholders and researchers, and the nature and process of CARM research. At this meeting, stakeholders collectively agreed that they would strive to meet the needs of all stakeholders rather than advocating for individual or subgroup interests. They further decided that collaborative learning should be an additional, explicit objective for the CARM project, in addition to the vegetation, wildlife, and beef production objectives. This reflective process and the resulting decisions exemplify normative learning.

Partial effectiveness of CARM

Under a best case scenario of intensive monitoring and commitment to implementing stakeholder decisions, CARM partially met each of the criteria for effective CAM. CARM reduced ecological uncertainties at some spatial and temporal scales, but also revealed additional complexities and uncertainties, especially regarding trade-offs over space and time within and among objectives. CARM made progress toward vegetation and wildlife, but not beef production objectives. Stakeholders succeeded in closing the loop by basing decisions on monitoring data in some cases, e.g., livestock movement decisions, but not in others, e.g., no-burn decision. Stakeholders improved over time at incorporating multiple interests and types of knowledge resulting in increased trust and meaningful social learning. Multiple-loop (normative) learning was observed when stakeholders revised their collective understanding of their roles and committed to seeking mutually beneficial solutions rather than advocating individual positions. Together, the five complexities we describe from CARM illustrate that intensive monitoring and institutional commitment may be necessary, but not sufficient, conditions for successful CAM. Instead, successful CAM may depend on authentic and prolonged engagement of participants with real social and ecological complexity.

DISCUSSION

A revised CAM conceptual framework

CARM’s partial progress toward effective CAM reveals that participants and researchers should not underestimate ecological or social complexity of CAM, even in situations that initially appear bounded, simple, or controlled. To illustrate the nature of this complexity, we present a revised CAM framework (Fig. 9), based on evidence from CARM. Our “learning-doing spiral” illustrates a more empirically grounded, nonlinear process than earlier conceptualizations (Fig. 1), summarized in four key revisions. First, as others have proposed (Montambault et al. 2015), CAM is best represented as a path-dependent spiral, not a closed circle. Trade-offs in time and space, time lags in data (see also Appendix 1), and interacting social worlds drive learning over time and constrain fully closing the loop. Second, key concepts from learning science enhance our understanding of data evaluation and learning proposed in the original CAM framework. Third, public participation is the intersection of many different mental models and social worlds as opposed to a “unitary” public. Fourth, the CAM process is situated within economic, social, historical, environmental, and climatic contexts that shape decision making. Together these four revisions to the CAM conceptual diagram suggest that systems concepts apply to the CAM process because excluding from consideration these or other subsystems when analyzing a complex systems problems may lead to misinterpretations of the barriers to successful CAM (Ackoff 1971).

The five complexities described above make it impossible to fully close the loop in CAM, i.e., to rapidly and iteratively adjust management actions based on monitoring data as hypothesized in the simple AM/CAM cycle (Fig. 1). From a biophysical perspective, this is because management decisions affect multiple, interconnected aspects of the biophysical template, different effects manifest at different rates, and certain management actions preclude other, future management actions. Grazing management decisions made in one year affect the options available for and potential outcomes of grazing management decisions made in subsequent years. It may therefore be more appropriate to envision CAM as a path-dependent spiral, rather than a circle, and to explicitly represent time lags in the conceptual diagram (Fig. 9, Appendix 1). A spiral better represents the way linkages between monitoring data and management decisions vary over the course of multiple years of adaptive management (Montambault et al. 2015). In CARM, we found that linkages between monitoring and management could change over time because of trade-offs among objectives, time lagged monitoring information, opportunities to learn more about current strategies, stakeholders’ differential perceptions of risk, and temporal and spatial trade-offs. Although trade-offs among objectives are not new (Ager et al. 2016), the literature does not often discuss how and why management objectives come to be recognized as conflicting and how this influences decision making and learning in CAM processes. As noted by Montambault et al. (2015), visualizing the adaptive management process as a spiral can also facilitate recognition of complex history and path dependency.

Trade-offs in space also made it difficult to close the loop in CARM, as participants became more aware of the challenges of managing for spatial heterogeneity at multiple scales for multiple management goals. Traditional grazing management in the Great Plains emphasized optimal livestock production for the latter half of the 20th century. Now, conservation concerns for grassland birds have spurred efforts to re-establish heterogeneity at regional scales (Fuhlendorf et al. 2006, Derner et al. 2009). Recognition of trade-offs in spatial heterogeneity at different scales provides opportunities for managers to minimize trade-offs and identify potential synergies among multiple objectives.

Depicting CAM as a spiral also represents how decision-making processes are shaped by social learning processes. Our qualitative analysis documented how complexities resulted in nonlinear and incomplete learning about various management actions. Knowledge from outside the experiment and management decisions made early in the process conditioned decisions and learning opportunities encountered later. Stakeholders engaged more fully with quantitative monitoring data when these data were linked to their existing experiential and local knowledge. This observation is consistent with learning science that demonstrates how new knowledge is constructed on a scaffolding of past experience (Kolb 1984, Lave and Wenger 1991, Novak 2010).

The CAM theoretical framework must account for the ways that stakeholder encounters with complexity contribute to increased trust and multiple-loop learning. We expanded upon the learning steps theorized by the existing CAM framework, specifically the “interpret data and evaluate” and “learn and adjust” (Fig. 1b), to more appropriately describe the learning processes observed in CARM (Fig. 9), supported by learning science.

Learning depends on an individual or group’s dissatisfaction with existing conceptions (Posner et al. 1982). This process may require a disorienting dilemma (Pennington et al. 2013), and a subsequent cognitive struggle (Bransford et al. 2006). The complexities and trade-offs we described in the CARM process created disorienting dilemmas and cognitive struggles that challenged stakeholders’ and researchers’ existing conceptions and set the stage for meaningful cognitive, normative, and relational learning-multiple-loop learning (Keen et al. 2005, Baird et al. 2014). It is uncertain whether learning diffused through social networks to a broader community during the initial study period (Reed et al. 2010).

Previous conceptualizations of public participation in CAM fail to account for the ways that collaboration facilitates the intersection of many different mental models and social worlds. Thus, stakeholders’ social worlds (represented in Fig. 9 by Venn diagram) are inherent to the CAM learning-doing spiral. Throughout the CAM process, differing individual mental models and social worlds collided, engaged, and collectively reconstructed new meanings in a shared space. This affected not only stakeholder decisions made based on monitoring data, but what data researchers collected and how, stakeholder interpretation and application of data, and the scope of available management options. Such knowledge coproduction does not imply absence of conflict, but rather learning about sources of differences and ways to understand and appreciate them through CAM.

If CAM practitioners do not account for the social nature of knowledge creation, wherein new knowledge is built upon participants’ existing experiences, learning within CAM may be ineffective. Lack of recognition or understanding of the social worlds that give rise to particular conceptions, norms, and relationships can lead to perceptions by some stakeholders that others are disregarding scientific evidence (Cote and Nightingale 2011). Researchers and some stakeholders initially explained ranchers’ apparent reluctance to burn as a failure to learn from monitoring data. Analysis of stakeholder interviews and meeting transcripts helped clarify that burning held a different meaning in ranchers’ social world, leading to different perceptions of risk. This suggests that relational learning may create conditions for later cognitive and normative learning in CAM. In CARM, some researchers and stakeholders initially held overly simplistic notions about the role of scientific monitoring data in CAM decision making, and underestimated the complexity of knowledge negotiation across multiple knowledge systems born of different social worlds. However, over the longer term, a new, shared, coproduced knowledge system emerged to help manage these complexities.

Finally, the entire CAM learning-doing spiral operates within economic, social, political, historical, environmental, and climatic contexts. These contexts inform stakeholder management priorities and help stakeholders evaluate trade-offs in management decision making. For example, the local management context helps explain why stakeholders prioritized learning about drought resilience by keeping CARM cattle in a single large herd, rather than splitting them into several smaller herds. This allowed for the development of a drought forage reserve by resting several pastures, even though it was associated with a reduction in daily livestock gains. The semiarid shortgrass ecosystem experienced several severe droughts in the early 2000s, and a regionally extensive drought in 2012 that caused major financial hardship for ranches. This temporal environmental context influenced how stakeholders valued learning about drought resilience over potential short-term financial gain from reduced stock density.

Acknowledging that CAM is shaped by social and environmental contexts at multiple scales may provide a pathway for the application of CAM-produced knowledge. The use of new knowledge depends on its salience, credibility, and legitimacy to users (Cash et al. 2003, Kristjanson et al. 2009, Beier et al. 2017). In multistakeholder collaborations, prolonged engagement and mutual knowledge exchange is often required to develop respect, trust, and ultimately the credibility and legitimacy of coproduced knowledge. In CARM, the emerging shared culture of learning contributed to coproduced knowledge driven by stakeholder questions relevant to the local natural resource management context. Coproduction was further enhanced by comparing CARM outcomes to those from the traditional rangeland management (TRM) treatment, the long-time status quo management on local rangelands outside of the project. This shared learning culture also led to the use of methods that stakeholders increasingly codesigned, in an environment of transparent, deliberative decision making to achieve shared goals. As a result, stakeholder use and ownership over CARM-produced data increased as they experienced situated and meaningful learning (Kolb 1984, Lave and Wenger 1991, Novak 2010) and conceptual change (Posner et al. 1982). In CARM, stakeholders and researchers increasingly recognized that knowledge coproduction holds greater potential for “actionable science” that can inform management decisions both within CARM and beyond (Beier et al. 2017).

Recommendations for CAM practice

To maximize learning and management effectiveness in the inherently complex context of CAM, we recommend the following practices. (1) Codevelop goals, objectives, management strategies and monitoring indicators in an inclusive manner, involving stakeholders in each step. (2) Accelerate relational and normative aspects of social learning by providing opportunities for participants to develop understanding and respect for each other’s social worlds, ways of learning, and the knowledge they practice daily. Invite nonscientist stakeholders to share/present their knowledge, including their interpretations of scientific data; develop researcher capacity to present and communicate scientific data clearly to nonscientists; and take time to build nonscientist stakeholders’ literacy in scientific terminology and data visualization methods. (3) Make social learning an explicit objective for stakeholders and researchers. Develop a common understanding of how learning occurs generally and for different stakeholders specifically, and discuss what kinds of evidence lead to conceptual change for different stakeholders, and what knowledge is deemed salient, credible and legitimate and why. Engage in reflexive practice within stakeholder-research teams to promote multiple-loop learning. (4) Anticipate and discuss spatial, temporal, and learning-doing trade-offs explicitly, so that stakeholders recognize when outcomes are not being achieved. Recognizing trade-offs may assist in reprioritizing management actions to seek synergistic (win-win) outcomes, or to balance multiple objectives over time. (5) Recognize path dependency in management and learning decisions as early decisions condition later ones making it more difficult to change course and adapt. The experience and empirical evidence derived from CARM indicate that these procedures will collectively enhance both the success and cost effectiveness of CAM, albeit with a considerable time commitment. Implementing these recommendations may require institutional commitment and investment that enable or incentivize managers and researchers to prioritize collaborative processes. However, CARM learning outcomes suggest that investment in CAM can lead to improved relationships, shared understanding, and credible, salient, and legitimate knowledge and solutions.

RESPONSES TO THIS ARTICLE

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.

ACKNOWLEDGMENTS

We gratefully acknowledge the participation of the CARM stakeholder group. We thank Nick Dufek, Melissa Johnston, Matt Mortenson, Pam Freeman, and many seasonal employees of USDA-ARS for excellence in project management and field data collection. We thank Emily Kachergis for her contributions initiating this study, and Julie (Jules) Kray for assistance creating figures 1, 5, and 8. We than the Crow Valley Livestock Cooperative, Inc. for providing steers. This work was supported by the USDA-ARS, USDA AFRI awards 2009-04442 and 2012-38415-20328, Colorado Agricultural Experiment Station project COLO0698, and the Center for Collaborative Conservation at Colorado State University.

LITERATURE CITED

Ackoff, R. L. 1971. Towards a system of systems concepts. Management Science 17:661-671. https://doi.org/10.1287/mnsc.17.11.661

Ager, A. A., M. A. Day, and K. Vogler. 2016. Production possibility frontiers and socioecological tradeoffs for restoration of fire adapted forests. Journal of Environmental Management 176:157-168. https://doi.org/10.1016/j.jenvman.2016.01.033

Aldridge, C. L., M. S. Boyce, and R. K. Baydack. 2004. Adaptive management of prairie grouse: How do we get there? Wildlife Society Bulletin 32:92-103. https://doi.org/10.2193/0091-7648(2004)32[92:AMOPGH]2.0.co;2

Allen, C., and L. H. Gunderson. 2011. Pathology and failure in the design and implementation of adaptive management. Journal of Environmental Management 92:1372-1384.

Argyris, C., and D. Schon. 1978. Organizational learning: a theory of action perspective. Addison-Wesley Publishing Company, Reading, Massachusetts, USA.

Armitage, D., M. Marschke, and R. Plummer. 2008. Adaptive co-management and the paradox of learning. Global Environmental Change 18:86-98. https://doi.org/10.1016/j.gloenvcha.2007.07.002

Armitage, D. R., R. Plummer, F. Berkes, R. I. Arthur, A. T. Charles, I. J. Davidson-Hunt, A. P. Diduck, N. C. Doubleday, D. S. Johnson, M. Marschke, P. McConney, E. W. Pinkerton, and E. K. Wollenberg. 2009. Adaptive co-management for social-ecological complexity. Frontiers in Ecology and the Environment 7:95-102. https://doi.org/10.1890/070089

Baird, J., R. Plummer, C. Haug, and D. Huitema. 2014. Learning effects of interactive decision-making processes for climate change adaptation. Global Environmental Change 27:51-63. https://doi.org/10.1016/j.gloenvcha.2014.04.019

Baird, J., R. Plummer, M.-L. Moore, and O. Brandes. 2016. Introducing resilience practice to watershed groups: What are the learning effects? Society and Natural Resources 29:1214-1229. https://doi.org/10.1080/08941920.2015.1107788

Bandura, A. 1977. Social learning theory. Prentice-Hall, Englewood Cliffs, New Jersey, USA.

Beier, P., L. J. Hansen, L. Helbrecht, and D. Behar. 2017. A how-to guide for coproduction of actionable science. Conservation Letters 10:288-296. https://doi.org/10.1111/conl.12300

Bentley Brymer, A. L., J. D. Wulfhorst, and M. W. Brunson. 2018. Analyzing stakeholders’ workshop dialogue for evidence of social learning. Ecology and Society 23(1):42. http://doi.org/10.5751/ES-09959-230142

Beratan, K. K. 2014. Summary: addressing the interactional challenges of moving collaborative adaptive management from theory to practice. Ecology and Society 19(1):46 http://dx.doi.org/10.5751/ES-06399-190146

Berkes, F. 2009. Evolution of co-management: role of knowledge generation, bridging organizations and social learning. Journal of Environmental Management 90:1692-1702. https://doi.org/10.1016/j.jenvman.2008.12.001

Bransford, J., N. Vye, R. Stevens, P. Kuhl, D. Schwartz, P. Bell, A. Meltzoff, B. Barron, R. Pea, and B. Reeves. 2006. Learning theories and education: toward a decade of synergy. Handbook of Educational Psychology 2:209-244.

Briske, D. D., J. D. Derner, J. R. Brown, S. D. Fuhlendorf, W. R. Teague, K. M. Havstad, R. L. Gillen, A. J. Ash, and W. D. Willms. 2008. Rotational grazing on rangelands: reconciliation of perception and experimental evidence. Rangeland Ecology and Management 61:3-17. https://doi.org/10.2111/06-159R.1

Brennan, L. A., and W. P. Kuvlesky, Jr. 2005. North American grassland birds: an unfolding conservation crisis? Journal of Wildlife Management 69:1-13 https://doi.org/10.2193/0022-541x(2005)069<0001:nagbau>2.0.co;2

Cash, D. W., W. C. Clark, F. Alcock. N. M. Dickson, N. Eckley, D. H. Guston, J. Jäger, and R. B. Mitchell. 2003. Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences 100(14):8086-8091. https://doi.org/10.1073/pnas.1231332100

Caves, J. K., G. S. Bodner, K. Simms, L. A. Fisher, and T. Robertson. 2013. Integrating collaboration, adaptive management, and scenario-planning: experiences at Las Cienegas National Conservation Area. Ecology and Society 18(3):43. http://dx.doi.org/10.5751/ES-05749-180343

Charnley, S., T. E. Sheridan, and G. P. Nabhan, editors. 2014. Stitching the West back together. University of Chicago Press, Chicago, Illinois, USA. https://doi.org/10.7208/chicago/9780226165851.001.0001

Clarke, A. E., and S. L. Star. 2008. The social worlds framework: a theory/methods package. Handbook of Science & Technology Studies 3:113-137.

Connell, D. 1997. Participatory development. Development in Practice 7:248-259. https://doi.org/10.1080/09614529754486

Cote, M., and A. J. Nightingale. 2011. Resilience thinking meets social theory: situating social change in socio-ecological systems (SES) research. Progress in Human Geography 36:475-489. https://doi.org/10.1177/0309132511425708

Cundill, G., G. S. Cumming, D. Biggs, and C. Fabricius. 2012. Soft systems thinking and social learning for adaptive management. Conservation Biology 26:13-20. https://doi.org/10.1111/j.1523-1739.2011.01755.x

Daniels, S. E., and G. B. Walker. 2001. Working through environmental conflict: the collaborative learning approach. Praeger, Westport, Connecticut, USA.

Delta Stewardship Council. 2013. The Delta Plan, Appendix C Adaptive Management and the Delta Plan. Delta Stewardship Council, Sacramento, California, USA. [online] URL: http://deltacouncil.ca.gov/sites/default/files/documents/files/AppC_Adaptive%20Management_2013.pdf

Derner, J. D., W. K. Lauenroth, P. Stapp, and D. J. Augustine. 2009. Livestock as ecosystem engineers for grassland bird habitat in the Western Great Plains of North America. Rangeland Ecology and Management 62:111-118. https://doi.org/10.2111/08-008.1

Derner, J. D., J. L. Reeves, M. C. Mortenson, M. West, J. Gonzalo Irisarri, and M. Durante. 2016. Estimating overnight weight loss of corralled yearling steers in semiarid rangeland. Rangelands 38:101-104. https://doi.org/10.1016/j.rala.2016.02.004

Fabricius, C., and G. Cundill. 2014. Learning in adaptive management: insights from published practice. Ecology and Society 19(1):29. http://dx.doi.org/10.5751/ES-06263-190129

Fernandez-Gimenez, M. E., H. L. Ballard, and V. E. Sturtevant. 2008. Adaptive management and social learning in collaborative and community-based monitoring: a study of five community-based forestry organizations in the western USA. Ecology and Society 13(2):4. https://doi.org/10.5751/ES-02400-130204

Fuhlendorf, S. D., W. C. Harrell, D. M. Engle, R. G. Hamilton, C. A. Davis, and D. M. Leslie. 2006. Should heterogeneity be the basis for conservation? Grassland bird response to fire and grazing. Ecological Applications 16:1706-1716.

Fujitani, M., A. McFall, C. Randler, and R. Arlinghaus. 2017. Participatory adaptive management leads to environmental learning outcomes extending beyond the sphere of science. Science Advances 3:e1602516. https://doi.org/10.1126/sciadv.1602516

Fynn, R. W. S. 2012. Functional resource heterogeneity increases livestock and rangeland productivity. Rangeland Ecology & Management 65:319-329. https://doi.org/10.2111/REM-D-11-00141.1

George, A. L., and A. Bennett. 2005. Case studies and theory development in the social sciences. MIT Press, Cambridge, Massachusetts, USA.

Hamilton, T. W., J. P. Ritten, C. T. Bastian, J. D. Derner, and J. A. Tanaka. 2016. Economic impacts of increasing seasonal precipitation variation on southeast Wyoming cow-calf enterprises. Rangeland Ecology & Management 69:465-473. https://doi.org/10.1016/j.rama.2016.06.008

Hanni, D. J., C. M. White, N. J. Van Lanen, J. J. Birek, J. M. Berven, and M. A. McLaren. 2013. Integrated monitoring of bird conservation regions (IMBCR): field protocol for spatially-balanced sampling of landbird populations. Rocky Mountain Bird Observatory, Brighton, Colorado, USA.

Hawkins, H.-J., A. Short, and K. P. Kirkman. 2017. Does holistic planned grazing™ work on native rangelands? African Journal of Range & Forage Science 34:59-63. https://doi.org/10.2989/10220119.2017.1367328

Herrick, J. E., J. W. Van Zee, K. M. Havstad, L. M. Burkett, and W. G. Whitford. 2009. Monitoring manual for grassland, shrubland and savanna ecosystems. USDA - ARS Jornada Experimental Range, Las Cruces, New Mexico, USA.

Holling, C. S. 1978. Adaptive environmental assessment and management. John Wiley and Sons, Chichester, UK.

Hsieh, H.-F., and S. E. Shannon. 2005. Three approaches to qualitative content analysis. Qualitative Health Research 15:1277-1288. https://doi.org/10.1177/1049732305276687

Innes, J. E., and D. E. Booher. 2010. Planning with complexity: an introduction to collaborative rationality for public policy. Routledge, London, UK.

Jones, N. A., H. Ross, T. Lynam, P. Perez, and A. Leitch. 2011. Mental models: an interdisciplinary synthesis of theory and methods. Ecology and Society 16(1):46. https://doi.org/10.5751/ES-03802-160146

Kachergis, E., J. Derner, L. Roche, K. Tate, M. Lubell, R. Mealor, and J. Magagna. 2013. Characterizing Wyoming ranching operations: natural resource goals, management practices and information sources. Natural Resources 4(1):45-54. 10.4236/nr.2013.41005 https://doi.org/10.4236/nr.2013.41005

Keen, M., V. A. Brown, and R. Dyball. 2005. Social learning in environmental management: towards a sustainable future. Earthscan, London, UK.

Kolb, D. A. 1984. Experiential learning. Prentice Hall, Englewood Cliffs, New Jersey, USA.

Kristjanson, P., R. S. Reid, N. Dickson, W. C. Clark, D. Romney, R. Puskur, S. MacMillan, and D. Grace. 2009. Linking international agricultural research knowledge with action for sustainable development. Proceedings of the National Academy of Sciences 106:5047-5052. https://doi.org/10.1073/pnas.0807414106

Kusel, J., S. C. Doak, S. Carpenter, and V. E. Sturtevant. 1996. The role of the public in adaptive ecosystem management. Pages 611-624 in Sierra Nevada Ecosystem Project: Final report to Congress, Vol II, Assessments and scientific basis for management options. University of California, Centers for Water and Wildland Resources, Davis, California, USA.

Lave, J., and E. Wenger. 1991. Situated learning: legitimate peripheral participation. Cambridge University Press, Cambridge, UK.

Lee, K. N. 1993. Compass and gyroscope: integrating science and politics for the environment. Island Press, Washington, D.C., USA.

Leys, A. J., and J. K. Vanclay. 2011. Social learning: a knowledge and capacity building approach for adaptive co-management of contested landscapes. Land Use Policy 28:574-584. https://doi.org/10.1016/j.landusepol.2010.11.006

Lincoln, Y. S., and E. G. Guba. 1986. But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation 1986:73-84. https://doi.org/10.1002/ev.1427

Lubell, M. 2015. Collaborative partnerships in complex institutional systems. Current Opinion in Environmental Sustainability 12:41-47. https://doi.org/10.1016/j.cosust.2014.08.011

McGoodwin, J. 2001. Understanding the cultures of fishing communities: a key to fisheries management and food security. FAO Report No. 401. Food and Agriculture Organization, Rome, Italy. [online] URL: http://www.fao.org/3/Y1290E/y1290e00.htm

McGreggor Cawley, R. 1993. Federal land, western anger: the Sagebrush Rebellion and environmental politics. University of Kansas Press, Lawrence, Kansas, USA.

McLain, R. J., and R. G. Lee. 1996. Adaptive management: promises and pitfalls. Environmental Management 20:437-448. https://doi.org/10.1007/BF01474647

Milchunas, D. G., W. K. Lauenroth, I. C. Burke, and J. K. Detling. 2008. Effects of grazing on vegetation. Pages 389-446 in W. K. Lauenroth and I. C. Burke, editors. Ecology of the shortgrass steppe: a long-term perspective. Oxford University Press, New York, New York, USA.

Moir, W. H., and W. M. Block. 2001. Adaptive management on public lands in the United States: commitment or rhetoric? Environmental Management 28:141-148. https://doi.org/10.1007/s002670010213

Montambault, J. R., S. Wongbusarakum, T. Leberer, E. Joseph, W. Andrew, F. Castro, B. Nevitt, Y. Golbuu, N. W. Oldiais, C. R. Groves, W. Kostka, and P. Houk. 2015. Use of monitoring data to support conservation management and policy decisions in Micronesia. Conservation Biology 29:1279-1289. https://doi.org/10.1111/cobi.12542

Moon, K., T. D. Brewer, S. R. Januchowski-Hartley, V. M. Adams, and D. A. Blackman. 2016. A guideline to improve qualitative social science publishing in ecology and conservation journals. Ecology and Society 21(3):17. https://doi.org/10.5751/ES-08663-210317

Murray, C., and D. Marmorek. 2003. Adaptive management and ecological restoration. Pages 417-428 in P. Freiderici, editor. Ecological restoration of southwestern ponderosa pine forests. Island Press, Washington, D.C., USA.

Nightingale, A. 2013. Fishing for nature: the politics of subjectivity and emotion in Scottish inshore fisheries management. Environment and Planning A 45:2362-2378. https://doi.org/10.1068/a45340

Novak, J. D. 2010. Learning, creating, and using knowledge: concept maps as facilitative tools in schools and corporations. Second edition. Routledge, New York, New York, USA. https://doi.org/10.4324/9780203862001

Nyberg, B. 1999. An introductory guide to adaptive management for project leaders and participants. BC Forest Service, Victoria, British Columbia, USA.

Pennington, D. D., G. L. Simpson, M. S. McConnell, J. M. Fair, and R. J. Baker. 2013. Transdisciplinary research, transformative learning, and transformative science. BioScience 63:564-573. https://doi.org/10.1525/bio.2013.63.7.9

Petersen, B., J. Montambaulrt, and K. Koopman. 2014. The potential for double-loop learning to enable landscape conservation efforts. Environmental Management 54:782-794. https://doi.org/10.1007/s00267-014-0337-4

Plummer, R., B. Crona, D. R. Armitage, P. Olsson, M. Tengö, and O. Yudina. 2012. Adaptive comanagement: a systematic review and analysis. Ecology and Society 17(3):11. http://dx.doi.org/10.5751/ES-04952-170311

Posner, G. J., K. A. Strike, P. W. Hewson, and W. A. Gertzog. 1982. Accommodation of a scientific conception: toward a theory of conceptual change. Science Education 66:211-227. https://doi.org/10.1002/sce.3730660207

Reed, M. S., A. C. Evely, G. Cundill, I. Fazey, J. Glass, A. Laing, J. Newig, B. Parrish, C. Prell, C. Raymond, and L. C. Stringer. 2010. What is social learning? Ecology and Society 15(4):r1. https://doi.org/10.5751/es-03564-1504r01

Reed, M. S., S. Vella, E. Challies, J. de Vente, L. Frewer, D. Hohenwallner-Ries, T. Huber, R. K. Neumann, E. A. Oughton, J. Sidoli del Ceno, and H. van Delden. 2018. A theory of participation: What makes stakeholder and public engagement in environmental management work? Restoration Ecology 26(S1):S7-S17. https://doi.org/10.1111/rec.12541

Rist, L., A. Felton, L. Samuelsson, C. Sandstrom, and O. Rosvall. 2013. A new paradigm for adaptive management. Ecology and Society 18(4):63. http://dx.doi.org/10.5751/ES-06183-180463

Robel, R. J., J. N. Briggs, A. D. Dayton, and L. C. Hulbert. 1970. Relationships between visual obstruction measurements and weight of grassland vegetation. Journal of Range Management 23:295-297. https://doi.org/10.2307/3896225

Roux, D. J., R. J. Stirzaker, C. M. Breen, E. C. Lefroy, and H. P. Cresswell. 2010. Framework for participative reflection on the accomplishment of transdisciplinary research programs. Environmental Science and Policy 13:733-741. https://doi.org/10.1016/j.envsci.2010.08.002

Sherren, K., J. Fischer, and I. Fazey. 2012. Managing the grazing landscape: insights for agricultural adaptation from a mid-drought photo-elicitation study in the Australian sheep-wheat belt. Agricultural Systems 106:72-83. https://doi.org/10.1016/j.agsy.2011.11.001

Skagen, S. K., D. J. Augustine, and J. D. Derner. 2018. Semi-arid grassland bird responses to patch-burn grazing and drought. Journal of Wildlife Management 82(2):445-456. https://doi.org/10.1002/jwmg.21379

Spellerberg, I. F., and P. J. Fedor. 2003. A tribute to Claude-Shannon (1916-2001) and a plea for more rigorous use of species richness, species diversity and the ‘Shannon-Wiener’ Index. Global Ecology and Biogeography 12(3):177-179. https://doi.org/10.1046/j.1466-822x.2003.00015.x

Stankey, G. H., B. T. Bormann, C. Ryan, B. Shindler, V. Sturtevant, R. N. Clark, and C. Philpot. 2003. Adaptive management and the Northwest Forest Plan: rhetoric and reality. Journal of Forestry 101:40-46.

Strauss, A. L. 1978. A social world perspective. Pages 119-128 in N. Denzin, editor. Studies in symbolic interaction. JAI Press, Greenwich, Connecticut, USA.

Susskind, L., A. E. Camacho, and T. Schenk. 2012. A critical assessment of collaborative adaptive management in practice. Journal of Applied Ecology 49:47-51. https://doi.org/10.1111/j.1365-2664.2011.02070.x

Teague, R., and M. Barnes. 2017. Grazing management that regenerates ecosystem function and grazingland livelihoods. African Journal of Range & Forage Science 34:77-86. https://doi.org/10.2989/10220119.2017.1334706

Tews, J., U. Brose, V. Grimm, K. Tielbörger, M. C. Wichmann, M. Schwager, and F. Jeltsch. 2004. Animal species diversity driven by habitat heterogeneity/diversity: the importance of keystone structures. Journal of Biogeography 31:79-92. https://doi.org/10.1046/j.0305-0270.2003.00994.x

Toombs, T. P., J. D. Derner, D. J. Augustine, B. Krueger, and S. Gallagher. 2010. Managing for biodiversity and livestock: a scale-dependent approach for promoting vegetation heterogeneity in western Great Plains grasslands. Rangelands 32:10-15. https://doi.org/10.2111/rangelands-d-10-00006.1

Tuomisto, H. 2012. An updated consumer’s guide to evenness and related indices. Oikos 121(8):1203-1218. https://doi.org/10.1111/j.1600-0706.2011.19897.x

United States Department of Agriculture, Natural Resources Conservation Service (USDA-NRCS). 2006. Land resource regions and major land resource areas of the United States, the Caribbean, and the Pacific Basin. U.S. Department of Agriculture Handbook 296. USDA-NRCS, Washington, D.C., USA.

United States Department of Agriculture, Natural Resources Conservation Service (USDA-NRCS). 2007a. Ecological site description for Loamy Plains (R067BY002CO). USDA-NRCS, Washington, D.C., USA. [online] URL: https://esis.sc.egov.usda.gov/ESDReport/fsReport.aspx?approved=yes&id=R067BY002CO

United States Department of Agriculture, Natural Resources Conservation Service (USDA-NRCS). 2007b. Ecological site description for Sandy Plains (R067BY024CO). USDA-NRCS, Washington, D.C., USA. [online] URL: https://esis.sc.egov.usda.gov/ESDReport/fsReport.aspx?approved=yes&id=R067BY024CO

Walker, P. 2016. Malheur occupation is over, but the war for America’s public lands rages on. The Conversation 19 February. [online] URL: https://theconversation.com/malheur-occupation-is-over-but-the-war-for-americas-public-lands-rages-on-54943

Westgate, M. J., G. E. Likens, and D. B. Lindenmayer. 2013. Adaptive management of biological systems: a review. Biological Conservation 158:128-139. https://doi.org/10.1016/j.biocon.2012.08.016

Williams, B. K. 2011a. Adaptive management of natural resources-framework and issues. Journal Environmental Management 92(5):1346-1353. https://doi.org/10.1016/j.jenvman.2010.10.041

Williams, B. K. 2011b. Passive and active adaptive management: approaches and an example. Journal of Environmental Management 92(5):1371-1378. 10.1016/j.jenvman.2010.10.039 https://doi.org/10.1016/j.jenvman.2010.10.039

Williams, B. K., and E. D. Brown. 2016. Technical challenges in the application of adaptive management. Biological Conservation 195:255-263. https://doi.org/10.1016/j.biocon.2016.01.012

Wilmer, H., D. J. Augustine, J. D. Derner, M. E. Fernández-Giménez, D. D. Briske, L. M. Roche, K. W. Tate, and K. E. Miller. 2018b. Diverse management strategies produce similar ecological outcomes on ranches in Western Great Plains: social-ecological assessment. Rangeland Ecology & Management 71:626-636. https://doi.org/10.1016/j.rama.2017.08.001

Wilmer, H., J. D. Derner, M. E. Fernández-Giménez, D. D. Briske, D. J. Augustine, L. M. Porensky, and the CARM Stakeholder Group. 2018a. Collaborative adaptive rangeland management fosters management-science partnerships. Rangeland Ecology & Management 71:646-657. https://doi.org/10.1016/j.rama.2017.07.008

Wondolleck, J. M., and S. L. Yaffee. 2000. Making collaboration work: lessons from innovation in natural resource management. Island Press, Washington, D.C., USA.

Yin, R. 1981. The case study as a serious research strategy. Knowledge: Creation, Diffusion, Utilization 3(1):97-114. https://doi.org/10.1177/107554708100300106

Address of Correspondent:
Hailey Wilmer
1701 Centre Ave
Fort Collins, CO
USA
80526
hailey.wilmer@usda.gov
Jump to top
Table1  | Table2  | Figure1  | Figure2  | Figure3  | Figure4  | Figure5  | Figure6  | Figure7  | Figure8  | Figure9  | Appendix1