Collaborative adaptive management (CAM) was developed to address complex natural resource management problems that are hard to resolve with standard problem-solving approaches. CAM combines adaptive management’s learning and experimental aspects with collaborative management’s participatory aspects (Monroe et al. 2013, Smedstad and Gosnell 2013). However, CAM has proved difficult to implement successfully (Mitroff and Featheringham 1974, Catton 1989, George 1995, Miller 1999, Susskind et al. 2010, Spradlin 2012, Chevallier 2016). It is inherently difficult to develop effective management plans for these messy problems (Bellamy et al. 2001), and the specific skills and capabilities needed to establish workable collaborations are not available in most natural resource management organizations (Yaffee and Wondolleck 2003). As a consequence, CAM efforts commonly get stuck in model development, plans fail to get implemented, and failure to make difficult management decisions leads to disappointing or even counterproductive results (Hernandez and Hodges 2006, Allen and Gunderson 2011, Berkley and Gunderson 2015, Head and Alford 2015).
One factor contributing to disappointing outcomes is lack of attention to process design (Camacho et al. 2010). In particular, the process of problem definition usually gets very little attention despite being arguably the most important single influence on a project’s ultimate success or failure (Posner 1973, Bardwell 1991, Miller 1999, Head and Alford 2015). Problem definition identifies which conditions, system elements, and relationships are considered most important to the problem-solving effort, and thus which interests will be prioritized (Bardwell 1991, Miller 1999) and what approach will be used to address the problem (Rittel 1972).
The most consequential decisions of the problem definition process, including the basic structure and scope of the problem situation and who should be involved, are made at the very start of a project when the project manager knows the least about the problem situation (Reed et al. 2009). Recognizing and filling important knowledge gaps is not simple because much of the needed knowledge is tacit and distributed among diverse individuals and organizations. New project managers thus face a major challenge: how to learn enough about the problem situation quickly enough to be of use in process planning.
I present a pragmatic approach, exploratory problem assessment (EA), that can help CAM project managers address this challenge and achieve better outcomes. This facilitated conceptual modeling approach can quickly provide project managers with contextualized information that is useful for problem definition and process planning, collected and presented in ways that foster engagement of key actors in the CAM process. The approach is based on two main ideas: (1) development of an effective problem definition and strategic plan can benefit from knowledge-focused facilitation, and (2) knowledge integration and fostering of common understandings can be enhanced by application of information design concepts to conceptual system modeling.
I begin with a discussion of the potential value of knowledge-centered facilitation and information design in the problem definition process. I then describe the EA approach and methodology and present two examples that illustrate how it can be applied in practice.
Problem definition can be much more difficult than expected in the type of complex problem situations suitable for CAM (Allen and Garmestani 2015, Murray et al. 2015). A particularly consequential misstep in project planning is insufficient engagement of crucial partners in problem definition. Decision makers in the organization convening the CAM process may believe that they already know what the focus should be. However, any single organization is likely to be expert only on aspects of the problem directly related to its mission and specific responsibilities. The broader knowledge needed for effective problem definition and project planning requires input from a diverse group of participants (Beratan 2007).
The choice of who should take part in problem definition influences project outcomes in two ways. First, the problem definition may reflect an inaccurate or incomplete understanding of the problem situation, including the social dynamics of the management subsystem. Second, key actors’ perceptions of a project’s salience, credibility, and legitimacy (Cash et al. 2003) are strongly influenced by who has a say in defining the problem.
A collaborative or inclusive problem definition process can address these issues. Such processes are rarely used (Davidoff 1965, George 1995, Allen and Kilvington 1999, van Cleve et al. 2004, Morss 2005, Stankey et al. 2005) because they can be time consuming and it can be very difficult to demonstrate progress to supervisors, funding agencies, and multiple decision makers with their separate time frames, i.e., election cycles and agency reporting requirements. Project managers rarely have training in participant engagement even though the very first interactions with potential project participants are particularly influential in shaping their attitudes toward the project.
Appropriate facilitation can help a new CAM project manager avoid early missteps and set the project up for success. Effective process facilitation requires someone able to elicit, filter, and integrate information from diverse sources (van Noordwijk et al. 2001, Carr and Wilkinson 2005, McNie 2007, Kristjanson et al. 2009), including tacit and experiential knowledge. The term “knowledge broker” has been applied to such people in service- and policy-related fields such as science policy, knowledge management, health care, and technology transfer. Knowledge brokers are boundary-spanning persons or organizations that facilitate information exchange within and between sectors, disciplines, and groups (Sverrisson 2001, Cash et al. 2003, Holgate 2012). Knowledge brokers not only move knowledge around but also transform it in ways that help diverse participants engage with each other in mutual understanding (Barnett 2003, Jackson 2003, Meyer 2010) and collectively identify problems that need to be addressed (Sverrisson 2001, Dobbins et al. 2009, Ward et al. 2009, Klerkx et al. 2012). Dobbins et al. (2009) classify knowledge-brokering activities into seven categories: initial and ongoing needs assessments; “scanning the horizon” to look for new evidence and resources of interest to participants; knowledge management; knowledge translation and exchange; network development, maintenance, and facilitation; facilitation of individual capacity development; and facilitation of and support for organizational change. Table 1 lists specific activities relevant to CAM project planning.
A knowledge broker for a CAM process needs to have (1) sufficient knowledge of biophysical systems across a range of scales to be able to find, read, and understand relevant scientific research and technical reports; (2) interpersonal communication skills and experiential knowledge of facilitation and conflict resolution concepts and methods; and (3) information design and graphic design skills. Not all facilitators have all of these knowledge-brokering skills, whereas some people who are not professional facilitators do have them.
Well-designed diagrams can be a powerful means of visually communicating information about complex problem situations that can be difficult for most people to understand (Joffe and Mindell 2006, Parsons and Sedig 2014). Good visual designs make use of humans’ inherent and highly developed spatial reasoning skills to reduce the cognitive effort of understanding a concept or set of relationships (Kirsh 2004, Heer and Agrawala 2008, Corter et al. 2011, Nickerson et al. 2013). For example, we readily understand the meaning of a diagram when the spatial organization on the page matches our nonconscious spatial expectations, e.g., time flows in the direction in which we read text and objects placed higher are larger or more powerful than those below them. In contrast, a poor match with expectations can confuse thinking, convey false relationships, or lead viewers to focus on unproductive concepts (Nickerson et al. 2013). Diagrams that are too complex can overwhelm viewers’ cognitive-processing capabilities (Fischer et al. 2012) and create confusion and negative feelings toward the diagram’s creators (Michela and Floricel 2012).
Application of general principles of information design (e.g., Tufte 1990) and visual design guidelines (e.g., Watzman 2002) can be employed to produce information-rich graphics that can be readily interpreted by the human brain. The approach I present makes particular use of the principles of separation and layering. “Separation” is distinguishing among types or layers of information through distinctions of texture, weight, shape, value, size, or color, and “layering” is visual stratification or ordering of data to establish a proper relationship among types of information (Tufte 1990).
A key element of the EA approach is development of readily understandable conceptual models of the problem situation showing problem-relevant system elements (polygons) and critical linking relationships (arrows, Fig. 1). Elements may be organizations; people; structures, e.g., the road system; ecosystems; biophysical features, e.g., a river or the soil; or a certification or requirement, e.g., a permit. Arrows can represent any type of linkage, including the following: causal relationships, e.g., more intense precipitation events because of global climate change will result in a greater incidence of flood events, or more people moving to an area will result in an increase in development pressure; and communication channels, e.g., A contacts B under certain circumstances. Another type of linkage occurs when changes in one element influence another element. For example, a citizen outreach effort may influence the way some people feel about an issue without directly causing collective action.
The conceptual models are developed using a visual design strategy in which every aspect, i.e., color, shape, and placement, of each element conveys problem-relevant information and requires relatively little cognitive effort to understand. Visual separation is achieved through differentiating categories of elements and linkages by differences in object shape, line style/width, and color of elements and arrows (Fig. 2). Layering is done by placement of elements and actions on the page. Placement along the horizontal axis is governed by the temporal order in which things are expected or need to occur. This makes it simple to highlight assumed causal relationships. Placement along the vertical axis is based on a spatial scale, grading from smaller at the bottom to larger at the top. This can be used to show cross-scale linkages and contextual factors. Using the scope of decision making to categorize the vertical scale can enhance diagrams’ subsequent usefulness in project planning. For instance, it can help identify which actors need to take what actions to achieve the project goals. The specifics of the problem situation determine which human system categories should be used in a particular assessment. As an example, Bellamy et al. (2001) identified four scales of application for Australian catchment management case studies: individual/enterprise, local community, catchment, and national/state policy. The vertical placement of natural system elements is determined by roughly matching their spatial scale with the human system element that influences an area of roughly the same size.
In addition to informing the problem definition process, the EA approach is designed to provide the project manager with information that is both useful and usable in project planning. The conceptual models show how proposed actions are supposed to lead to desired outcomes and thus provide a graphic summary of change theories linking causal factors to emergent system properties. This makes the diagrams particularly useful for developing evaluation metrics that can provide feedback in time to permit adjustment of management actions. Because human behavior is a major causal factor in most natural resource problems, changes in human behavior tend to be useful leading indicators.
The first step in the EA process (Table 2) is for the knowledge broker to have an orienting conversation with the project manager. This discussion should center on two major topics: (1) the objectives of, constraints on, and resources available to the project; and (2) a description of the problem situation in terms of problematic conditions, causal factors, and key actors. The knowledge broker should ask clarifying questions that can help the project manager structure information in new and useful ways. This conversation is arguably the most important step in the entire process because it provides an opportunity for the project manager to think strategically and to engage in reflective exploration of tacit knowledge and unexamined assumptions.
During the conversation, the knowledge broker works with the project manager to capture important relationships and causal connections in the form of short diagrammatic “sentences” using the visual “grammar” described previously (Fig. 1). These building blocks will be the basis for the initial conceptual model, and the process introduces the visualization approach to the project manager.
Key informants, starting with those identified by the project manager, are interviewed to gain additional information and different views of the problem situation. Interviews are semistructured; informants are prompted to tell stories about relevant experiences and to bring to light assumptions about future system behavior. The knowledge broker works with the interviewee to summarize the information in graphic form; this assists in keeping the conversation focused on problem-relevant information and helps visual thinkers work through contexts and linkages. After informants have finished providing their independent information, they are asked to review the current version of the conceptual model and help integrate their information into it. They are likely to provide additional information while doing so because many people find it easier to articulate experiential knowledge through pointing out errors in the draft diagram than in answer to direct questions.
It is common to find discrepancies when integrating information provided by different informants even though each tells the truth as he or she sees it. The following rules of thumb are useful in such cases:
The following two examples illustrate how the EA approach works in practice and how it can be adapted for different decision contexts. Both were elements of larger resource management efforts. Although neither of these larger efforts was identified as a CAM project by its project manager, they exhibit many characteristics of a CAM process: the objectives of management activities were changes in biophysical conditions requiring changes in human behavior; there was uncertainty about which management activities would best achieve the management objectives; the active participation of many individuals, government agencies, and organizations was needed; and there was intent to monitor the results of management actions to determine if adjustments were needed. The included quotes are from follow-up interviews with the two project managers.
Study of relatively small-scale natural resource management efforts such as these can provide useful insights applicable to the larger scale projects that are the usual focus of CAM scholarship. Smaller efforts are easier to analyze because they generally involve fewer key actors and because management actions produce identifiable impacts relatively rapidly.
I worked with the executive director (ED) of a nonprofit watershed association in a small city that works to protect and restore a stream flowing through the most urbanized part of the city. The organization has leveraged limited resources by building strong relationships with neighborhoods and by developing project-centered partnerships with city government offices, the business community, and other environmental organizations. The board of directors initiated a planning process aimed at revising the organization’s committee structure in response to a recent expansion of the organization’s activities and resources. The ED requested my help in coordinating this with his ongoing strategic planning activities.
Our initial discussion centered on how the organization might operate more effectively given its resource limitations and current capabilities. I created a draft conceptual model and then interviewed staff and board members about the stream system and the organization’s activities. I used the visual design strategy described previously to structure the elicited information and keep the conversations focused on feasible pathways to tangible objectives. Following each interview, the graphic notes were integrated into the conceptual model.
The resulting diagrams, one of which is shown in Appendix 1 (see the online supplementary materials), highlighted some issues and relationships that were not included in the ED’s initial framing. For example, staff were increasingly stressed by trying to meet the divergent expectations of grants providing staff funding and those of the board tied to the specific mission of the organization. This stress was subsequently reduced through hiring of additional staff and redefinition of staff responsibilities.
In our follow-up interview, the ED said that our discussions had helped him recognize why he had experienced difficulty conducting his strategic planning effort in parallel with the board’s very different and more limited process:
What [the board] wanted was something very, very practical. Which was, how should our board and committees be set up? That’s not where I was. I think I was more open-ended than that. I was more thinking about what does [the organization] want to achieve? What’s the big picture? What are some of the partnerships needed there? What’s in our way?
He found that his conversations with me were useful for his long-term strategic planning but were less relevant to the board’s process:
At the same time I had this board committee going through a process where they’re basically trying to come up with a vision of how to rearrange our committees, to make it a better fit for our current reality and for our goals moving forward. I was using this thinking that we were doing to help me, I think, in a deeper way, inform myself while we were going through that process. I have to admit I found it difficult to use it that way. Because here I was with you, talking about … it’s not just the process but the product of what we did is very outward focused. Because you’re looking for a way through, a path forward in complex situations. You’re thinking outside the organization by definition. You’re thinking outside of the current reality by definition. I found the process that I was going through with the board at the same time to be more insular. We were definitely trying to find solutions, like putting the structures in place. But for us it was more about, "Well, what are our committees like now?" and, "Are they really the committees that we need to do what we want to do in the future?" Somehow that doesn’t beg the question of how do we need to be forming partnerships and how do we need to be thinking outside the organization.
The ED noted that he had not fully understood my assessment process and its potential value at the outset. He said that if he had it to do over again, he would have narrowed the focus of the assessment somewhat, emphasizing a particular problem or issue. However, his statements during our initial discussion suggested that he was not ready to focus more narrowly at that time. Over the course of the project I observed that he was increasingly able to articulate his tacit understandings and assumptions about the problem situation and the organization’s potential roles, and that this allowed him to more quickly identify particular organizational issues relevant to the topics being discussed.
The ED concluded that working with me had provided a useful opportunity to mentally step back from routine concerns to regain a broader, longer term view of the problem situation and how the organization functions within the management network. This helped him think more strategically about the organization’s operational priorities:
Your role again was holding up a mirror. But with a filter. The filter allowed us, allowed me, to look with a very specific eye toward the processes that we needed to be thinking about, the potential connections, the impediments that might be in the way of making necessary connections, taking the steps forward, accomplishing goals or objectives, the partnerships that might be needed. In that way, you forced me to think about those things that maybe I would have not consciously been trying to think about without a framework. And so I just didn’t have a framework, and I like the framework you got me.
A multicounty land trust and partners received funding to protect the endangered longleaf pine (Pinus palustris) ecosystem by increasing private landowners’ willingness and capacity to conduct controlled burns on their land. The project manager described this pilot project’s goal in this way:
There is so much negative perception about fire. How could we create a process that would allow people to develop the skills, but more importantly, develop the confidence to be the change agents [for what] they wanted to see happen on their land?
The project manager requested my assistance in dealing with implementation challenges: the grant provided less funding than requested, the reduced funding increased the need for coordination among partner agencies and organizations, and project evaluation metrics were needed that could capture early impacts of relationship-building efforts to both permit adaptive modifications to the project design and inform the project funder. I began by developing a conceptual model based on the project manager’s input, the proposal text, and published reports on longleaf pine management. I obtained additional information through meetings with the project manager and representatives from two partner organizations. Interestingly, all three said that they learned useful information from each other while explaining the project to me even though they had previously worked together on similar projects.
The project manager said that the process was particularly useful in identifying information gaps in the presentations to be given to landowners:
So when we were kind of mapping out with you, and [interviewee 2] was there and trying to figure out where to go, where it might be sort of obvious to us, or we think it’s obvious, because we’re down in it. As a person that doesn’t have that insider knowledge necessarily or you just have a way of being objective, you are able to say, but wait, there’s, there’s other ways that they need to engage before they get … We think this is point A, but actually this is point C, because you were able to see that there’s a previous action step that needed to be taken.
Most of these information gaps were because of differences in expertise between the resource management professionals preparing the presentations and the landowners who were their target audience. This type of knowledge gap is very common because it is difficult for an expert to see things from the perspective of a nonexpert (Wieman 2007). Because I lacked specific expertise in fire-based management of longleaf pine, I was able to serve as a surrogate for the target audience.
The project manager recognized that lack of clear and specific guidance about procedures and risk management had contributed to disappointing outcomes in many previous efforts to get private landowners to adopt fire management practices. A priority for him was presenting information in a way that encouraged rather than discouraged landowners:
Yeah, how to sequence that information so that it makes sense to the landowner and they can see a progression. Because you need that positive feedback loop to say, I made this step, this resulted in a positive affirmation of that step that allows me the confidence to move to the second step. … What you don’t want to do, and I have a fear of this happening with other agencies, if you get a landowner who goes out and tries to make something happen and it happens in a negative way, they get a negative feedback loop from it. You’ve actually not gone back to the zero. You’ve gone back to negative 50.
Therefore, I developed use-specific diagrams based on the conceptual model, including a decision-tree handout and checklist for landowners listing the steps in conducting controlled burns as well as a project flowchart to assist the project manager with strategic planning (Appendix 2 in the online supplementary materials).
The process manager found both the diagrams and the process as a whole to be useful:
Having a process that can create a seat for a wide variety of experiences becomes very important. And that’s what your process allowed us to do, is to see in our own head how to roll it out in a way that makes sense to the folks that we want to see adopt the change.
The value of the process was demonstrated within the project’s first 6 months. Our conversations helped the project manager identify an outreach activity that became the centerpiece of the project, and several landowners have subsequently moved forward with fire management planning. The decision-tree handout proved useful enough that other management organizations have adopted it for their own landowner outreach efforts. The “leading indicator” metrics derived from the conceptual model proved useful for communicating the project’s progress to the granting agency.
Caution must be used when interpreting and generalizing from the illustrative examples presented previously because they were not designed as research projects. Nonetheless, they can provide insights into how the EA approach might be usefully applied in other complex management situations.
The two examples differed in the type of planning process involved (Table 3). The watershed management process was part of a general rethinking of the growing organization’s structure and strategic plan. The process therefore was broadly concerned with “what should we do” questions, with a focus on identifying and prioritizing potential interventions and key actors, i.e., intermediate management level, throughout a relatively small watershed that is entirely within a single county, i.e., low to intermediate geographic extent.
In contrast, the longleaf management process centered on implementation of a particular project. It thus was more narrowly concerned with “how should we do it” questions, with a focus on identifying specific activities, i.e., low management level, that could overcome barriers to adoption of promoted management practices at a landscape scale, i.e., broad geographic extent.
Both project managers had experienced difficulty in organizing their extensive but amorphous and largely tacit knowledge into a more ordered and shareable framework from which a logical and defensible change theory could be derived. In both cases, I found that the initial orienting conversation was the most productive step of the EA process. Explaining the situation to me allowed the project managers to make explicit their extensive tacit knowledge, identify information gaps, and recognize opportunities and limitations. This helped both project managers increase the specificity and usefulness of their problem definitions.
Both project managers found construction of the diagrammatic representations to be useful for organizing their thinking into a guiding framework. This aided them in prioritizing among possible actions to make effective use of limited resources. In addition, it helped them engage with and communicate important information and ideas to staff, key partners, and stakeholders. Rather than disseminating the conceptual models, both project managers requested simpler derived diagrams designed for specific audiences and events. This follows a foundational concept of information design: A presentation needs to be designed to meet specific objectives and for a particular audience to effectively communicate information.
The EA approach I have described is similar to “problem structuring methods” (PSMs) from the field of operational research (e.g., Checkland 1981, Checkland and Scholes 1990, Friend and Hickling 2005). PSMs are characterized by use of diagrammatic models as a temporary focus for structuring engagement among a diverse group of participants looking for solutions in complex problem situations. These models show connections between concepts, activities or stakeholders, and cause-and-effect relationships (Eden and Sims 1979, Eden and Ackermann 2006, Horlick-Jones and Rosenhead 2007, Cronin et al. 2014) to help participants develop a shared understanding of the nature and boundaries of the problems to be addressed (Rosenhead and Mingers 2001). The EA approach differs from PSMs and other forms of group model building in its focus on knowledge gathering and integration at the start of process planning rather than on direct engagement among stakeholders during the decision process. In fact, EA could be a useful precursor activity to group model building exercises.
The EA approach is also similar to the rapid stakeholder and conflict assessment (RSCA) approach developed by Hjortsø et al. (2005). Both approaches draw on diverse knowledge sources to gain a broader understanding of the problem situation. The RSCA approach focuses on assessment of stakeholder perceptions and conflicts and is “aimed at preparing facilitators of participatory inquiry and decision-making processes by providing them with a way to obtain an in-depth understanding of the situation in which they will engage before conducting explicit participatory activities” (Hjortsø et al. 2005:150). In contrast, the EA aims to assist a CAM project manager in gaining a sufficiently broad and complete understanding of both the biophysical, socioeconomic, and management aspects of a problem situation quickly enough to guide problem definition and project planning. It focuses on causal relationships and development of practical change theories. The EA approach draws information from each interviewee’s technical and/or experiential knowledge of the particular problem situation rather than considering specialist and stakeholder knowledge separately.
EA’s visual design strategy has some similarities to other modeling approaches and tools, such as influence diagrams (e.g., Howard and Matheson 2005) and situation mapping (Daniels and Walker 2012). The role of the facilitator differs: In the EA approach, the facilitator develops the model based on stakeholder knowledge collected through interviews rather than facilitating model development by stakeholders during stakeholder meetings. Another difference is in the degree of systematization in design and structure of the resulting diagrams. These differences reflect differences in intended application. Most other conceptual modeling approaches and tools are intended as “a means of designing and implementing a series of events to promote creative thought, constructive debate and the effective implementation of proposals that the stakeholders generate” (Daniels and Walker 2012:113, in reference to situation mapping). The emphasis thus is more on diagramming as a tool to enable collective learning through discourse in a collaborative process. In contrast, the EA approach focuses more on the utility and usability of the resulting diagrams: They are designed to be readily interpretable by people who did not participate in their creation and to assist in the transfer of problem-relevant knowledge from diverse sources to the project manager and partners for use in project planning. In addition, the readily interpretable diagrams can preserve a “snapshot” view of what was understood about the problem situation at a given time, and periodic updates can help document significant changes in the problem situation and in people’s perceptions of the situation.
The EA approach was designed with a practical aim: to help CAM project managers quickly gain knowledge necessary for problem definition and initial project planning, and to do so in a way that promotes productive engagement with potential partners and stakeholders. The EA approach is based on recognition that the very first decisions and actions when setting up a new CAM effort have a disproportionate impact on the effort’s ultimate success.
The two example projects illustrate the two main concepts at the core of the EA approach: the problem definition process can benefit from knowledge-focused facilitation, and application of sound information design principles can enhance the usefulness of problem-relevant information. More research is needed for definitive assessment, but the project managers’ comments support the supposition that the EA approach can provide project managers with useful information that can lead to valuable insights into the nature and structure of the problem to be addressed. This can help them develop a more effective problem definition and strategic plan.
A process involving face-to-face interactions among stakeholders is likely to be more effective than the EA approach in furthering social outcomes like trust building. However, a skilled knowledge broker conducting an EA process can introduce a project to potential participants in a way that encourages productive engagement and limits defensive responses. This can help foster acceptance of the gathered and integrated knowledge by the key stakeholders and decision makers and can prepare the ground for future collaboration by encouraging key actors to consider how their knowledge and point of view intersects with those of other stakeholders.
In conclusion, a new CAM effort is more likely to achieve desired outcomes if the project manager conducts a thoughtful problem definition process rather than assuming that “we know what the problem is.” The EA approach provides a practical and cost-effective way for CAM project managers to set their projects up for success.
My thanks to the Department of Forestry and Natural Resources at North Carolina State University, where some of the work described in this paper was done. I am grateful to the project leads for working with me and for agreeing to inclusion of their projects in this paper. Special thanks to Jim Berkley and Jennifer Pratt Miles for their encouragement and their helpful comments on early versions of this paper. I appreciate the helpful recommendations of the editor and two anonymous reviewers.
Allen, C. R., and A. S. Garmestani. 2015. Adaptive management. Pages 1-10 in C. R. Allen and A. S. Garmestani, editors. Adaptive management of social-ecological systems. Springer, Dordrecht, The Netherlands. https://doi.org/10.1007/978-94-017-9682-8_1
Allen, C. R., and L. H. Gunderson. 2011. Pathology and failure in the design and implementation of adaptive management. Journal of Environmental Management 92(5):1379-1384. https://doi.org/10.1016/j.jenvman.2010.10.063
Allen, W. J., and M. J. Kilvington. 1999. Why involving people is important: the forgotten part of environmental information system management. Pages 1-6 in Proceedings of the Second International Conference on Multiple Objective Decision Support Systems (MODSS ’99) for Land, Water and Environmental Management (Brisbane, Queensland, Australia, 1999). Queensland Department of Natural Resources, Brisbane, Queensland, Australia.
Bardwell, L. V. 1991. Problem-framing: a perspective on environmental problem-solving. Environmental Management 15(5):603-612. https://doi.org/10.1007/BF02589620
Barnett, R. 2003. Foreword. Pages xvi-xviii in N. Jackson, editor. Engaging and changing higher education through brokerage. Aldershot, Ashgate, UK.
Bellamy, J. A., D. H. Walker, G. T. McDonald, and G. J. Syme. 2001. A systems approach to the evaluation of natural resource management initiatives. Journal of Environmental Management 63(4):407-423. https://doi.org/10.1006/jema.2001.0493
Beratan, K. K. 2007. A cognition-based view of decision processes in complex social–ecological systems. Ecology and Society 12(1):27. https://doi.org/10.5751/ES-02103-120127
Berkley, J., and L. Gunderson. 2015. Practical resilience: building networks of adaptive management. Pages 201-216 in C. R. Allen and A. S. Garmestani, editors. Adaptive management of social-ecological systems. Springer, Dordrecht, The Netherlands. https://doi.org/10.1007/978-94-017-9682-8_11
Camacho A. E., H. Doremus, J. S. McLachlan, and B. A. Minteer. 2010. Reassessing conservation goals in a changing climate. Issues in Science and Technology 26:21-26.
Carr, A., and R. Wilkinson. 2005. Beyond participation: boundary organizations as a new space for farmers and scientists to interact. Society and Natural Resources 18(3):255-265. https://doi.org/10.1080/08941920590908123
Cash, D., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, and J. Jäger. 2003. Salience, credibility, legitimacy and boundaries: linking research, assessment and decision making. KSG Working Papers Series RWP02-046. Harvard University, Cambridge, Massachusetts, USA. https://doi.org/10.2139/ssrn.372280
Catton, W. R. 1989. Choosing which danger to risk. Society 27(1):6-8. https://doi.org/10.1007/BF02695443
Checkland, P. 1981. Systems thinking, systems practice. Wiley, Chichester, UK.
Checkland, P., and J. Scholes. 1990. Soft systems methodology in action. Wiley, New York, New York, USA.
Chevallier, A. 2016. Strategic thinking in complex problem solving. Oxford University Press, New York, New York, USA. https://doi.org/10.1093/acprof:oso/9780190463908.001.0001
Corter, J. E., D. L. Mason, B. Tversky, and J. V. Nickerson. 2011. Identifying causal pathways with and without diagrams. Pages 2662-2667 in L. Carlson, C. Hoelscher, and T. F. Shipley, editors. Proceedings of the 33rd Annual Conference of the Cognitive Science Society (Boston, Massachusetts, 20-23 July 2011). Cognitive Science Society, Austin, Texas, USA.
Cronin, K., G., Midgley, and L. S. Jackson. 2014. Issues mapping: a problem structuring method for addressing science and technology conflicts. European Journal of Operational Research 233:145-158. https://doi.org/10.1016/j.ejor.2013.08.012
Daniels, S. E., and G. B. Walker. 2012. Lessons from the trenches: twenty years of using system thinking in natural resource conflict situations. Systems Research and Behavioral Science 29:104-115. https://doi.org/10.1002/sres.2100
Davidoff, P. 1965. Advocacy and pluralism in planning. Journal of the American Institute of Planners 31:331-338. https://doi.org/10.1080/01944366508978187
Dobbins, M., P. Robeson, D. Ciliska, S. Hanna, R. Cameron, L. O’Mara, K. DeCorby, and S. Mercer. 2009. A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies. Implementation Science 4:23. https://doi.org/10.1186/1748-5908-4-23
Eden, C., and F. Ackermann. 2006. Where next for problem structuring methods. Journal of the Operational Research Society 57(7):766-768. https://doi.org/10.1057/palgrave.jors.2602090
Eden, C., and D. Sims. 1979. On the nature of problems in consulting practice. Omega 7(2):119-127. https://doi.org/10.1016/0305-0483(79)90099-9
Fischer, A., S. Greiff, and J. Funke. 2012. The process of solving complex problems. Journal of Problem Solving 4(1):19-42. https://doi.org/10.7771/1932-6246.1118
Friend, J., and A. Hickling. 2005. Planning under pressure: the strategic choice approach. Third edition. Elsevier, New York, New York, USA.
George, R. V. 1995. Expert support for problem formulation in urban planning. Environment and Planning B: Planning and Design 22(2):175-190. https://doi.org/10.1068/b220175
Head, B. W., and J. Alford. 2015. Wicked problems: implications for public policy and management. Administration and Society 47(6):711-739. https://doi.org/10.1177/0095399713481601
Heer, J., and M. Agrawala. 2008. Design considerations for collaborative visual analytics. Information Visualization 7:49-62. https://doi.org/10.1057/palgrave.ivs.9500167
Hernandez, M., and S. Hodges. 2006. Applying a theory of change approach to interagency planning in child mental health. American Journal of Community Psychology 38(3-4):183-190. https://doi.org/10.1007/s10464-006-9079-7
Hjortsø, C. N., S. M. Christensen, and P. Tarp. 2005. Rapid stakeholder and conflict assessment for natural resource management using cognitive mapping: the case of Damdoi Forest Enterprise, Vietnam. Agriculture and Human Values 22:149-167. https://doi.org/10.1007/s10460-004-8275-z
Holgate, S. A. 2012. Emerging professions: knowledge broker. Science, 8 June. https://doi.org/10.1126/science.caredit.a1200064
Horlick-Jones, T., and J. Rosenhead. 2007. The uses of observation: combining problem structuring methods and ethnography. Journal of the Operational Research Society 58:588-601. https://doi.org/10.1057/palgrave.jors.2602271
Howard, R. A., and J. E. Matheson. 2005. Influence diagram retrospective. Decision Analysis 2(3):144-147. https://doi.org/10.1287/deca.1050.0050
Jackson, N. 2003. Introduction. Pages xvi-xviii in N. Jackson, editor. Engaging and changing higher education through brokerage. Ashgate, Aldershot, UK.
Joffe, M., and J. Mindell. 2006. Complex causal process diagrams for analyzing the health impacts of policy interventions. American Journal of Public Health 96(3):473-479. https://doi.org/10.2105/AJPH.2005.063693
Kirsh, D. 2004. Metacognition, distributed cognition, and visual design. Pages 147-179 in P. Gardenfors and P. Johansson, editors. Cognition, education, and communication technology. Routledge, New York, New York, USA.
Klerkx, L., M. Schut, C. Leeuwis, and C. Kilelu. 2012. Advances in knowledge brokering in the agricultural sector: towards innovation system facilitation. IDS Bulletin 43(5):53-60. https://doi.org/10.1111/j.1759-5436.2012.00363.x
Kristjanson, P., R. S. Reid, N. Dickson, W. C. Clark, D. Romney, R. Puskur, S. MacMillan, and D. Grace. 2009. Linking international research knowledge with action for sustainable development. Proceedings of the National Academy of Sciences of the United States of America 106(13):5047-5052. https://doi.org/10.1073/pnas.0807414106
McNie, E. C. 2007. Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature. Environmental Science and Policy 10(1):17-38. https://doi.org/10.1016/j.envsci.2006.10.004
Meyer, M. 2010. The rise of the knowledge broker. Science Communication 32(1):118-127. https://doi.org/10.1177%2F1075547009359797
Michela, J. L., and S. Floricel. 2012. Collaboration for innovation through knowledge representation. Pages 1-14 in The XXIII International Society for Professional Innovation Management (ISPIM) Conference Proceedings (Barcelona, Spain, 17-20 June 2012). International Society for Professional Innovation Management, Manchester, UK.
Miller, A. 1999. Environmental problem solving: psychosocial barriers to adaptive change. Springer, New York, New York, USA.
Mitroff, I. I., and T. R. Featheringham. 1974. On systemic problem solving and the error of the third kind. Behavioral Science 19(6):383-393. https://doi.org/10.1002/bs.3830190605
Monroe, M. C., R. Plate, and A. Oxarart. 2013. Intermediate collaborative adaptive management strategies build stakeholder capacity. Ecology and Society 18(2):24. https://doi.org/10.5751/ES-05444-180224
Morss, R. E. 2005. Problem definition in atmospheric science public policy: the example of observing-system design for weather prediction. Bulletin of the American Meteorological Society 86:181-192. https://doi.org/10.1175/BAMS-86-2-181
Murray, C. L., D. R. Marmorek, and L. A. Greig. 2015. Adaptive management today: a practitioners’ perspective. Pages 181-199 in C. R. Allen and A. S. Garmestani, editors. Adaptive management of social-ecological systems. Springer, Dordrecht, The Netherlands. https://doi.org/10.1007/978-94-017-9682-8_10
Nickerson, J. V., J. E. Corter, B. Tversky, Y. J. Rho, D. Zahner, and L. Yu. 2013. Cognitive tools shape thought: diagrams in design. Cognitive Processing 14(3):255-272. https://doi.org/10.1007/s10339-013-0547-3
Parsons, P., and K. Sedig. 2014. Common visualizations: their cognitive utility. Pages 671-691 in W. Huang, editor. Handbook of human centric visualization. Springer, New York, New York, USA. https://doi.org/10.1007/978-1-4614-7485-2_27
Posner, M. I. 1973. Cognition: an introduction. Scott, Foresman, Oxford, UK.
Reed, M. S., A. Graves, N. Dandy, H. Posthumus, K. Hubacek, J. Morris, C. Prell, C. H. Quinn, and L. C. Stringer. 2009. Who’s in and why? A typology of stakeholder analysis methods for natural resource management. Journal of Environmental Management 90:1933-1949. https://doi.org/10.1016/j.jenvman.2009.01.001
Rittel, H. W. J. 1972. On the planning crisis: systems analysis of the "first and second generations." Institut für Grundlagen der Planung I.A., Universität Stuttgart, Stuttgart, Germany.
Rosenhead, J., and J. Mingers. 2001. Rational analysis for a problematic world revisited: problem structuring methods for complexity, uncertainty and conflict. Wiley, Chichester, UK.
Smedstad, J. A., and H. Gosnell. 2013. Do adaptive comanagement processes lead to adaptive comanagement outcomes? A multicase study of long-term outcomes associated with the National Riparian Service Team’s place-based riparian assistance. Ecology and Society 18(4):8. https://doi.org/10.5751/ES-05793-180408
Spradlin, D. 2012. Are you solving the right problem? Asking the right questions is crucial. Harvard Business Review 90(9):84-93.
Stankey, G. H., R. N. Clark, and B. T. Bormann. 2005. Adaptive management of natural resources: theory, concepts, and management institutions. General Technical Report PNW-GTR-654. U.S. Forest Service, Pacific Northwest Research Station, Portland, Oregon, USA. https://doi.org/10.2737/PNW-GTR-654
Susskind, L., A. E. Camacho, and T. Schenk. 2010. Collaborative planning and adaptive management in Glen Canyon: a cautionary tale. Colombia Journal of Environmental Law 35:1-55.
Sverrisson, Á. 2001. Translation networks, knowledge brokers and novelty construction: pragmatic environmentalism in Sweden. Acta Sociologica 44(4):313-327.
Tufte, E. R. 1990. Envisioning information. Graphics, Cheshire, Connecticut, USA.
van Cleve, F. B., C. Simenstad, F.Goetz, and T. Mumford. 2004. Application of “best available science” in ecosystem restoration: lessons learned from large-scale restoration efforts in the USA. Puget Sound Nearshore Partnership Report 2004-1. Washington Sea Grant Program, University of Washington, Seattle, Washington, USA. [online] URL: http://www.pugetsoundnearshore.org/technical_papers/lessonslearned.pdf
van Noordwijk, M., T. P. Tomich, and B. Verbist. 2001. Negotiation support models for integrated natural resource management in tropical forest margins. Conservation Ecology 5(2):21. http://dx.doi.org/10.5751/ES-00344-050221
Ward, V., A. House, and S. Hamer. 2009. Knowledge brokering: the missing link in the evidence to action chain? Evidence and Policy 5(3):267-279. https://doi.org/10.1332/174426409X463811
Watzman, S. 2002. Visual design principles for usable interfaces. Pages 263-285 in K. Stanney, editor. Human computer interaction handbook: design, implementations, and applications. Lawrence Erlbaum, Mahwah, New Jersey, USA.
Wieman, C. 2007. The “curse of knowledge” or why intuition about teaching often fails. American Physical Society News 16(10). [online] URL: https://www.aps.org/publications/apsnews/200711/backpage.cfm
Yaffee, S. L., and J. M. Wondolleck. 2003. Collaborative ecosystem planning processes in the United States: evolution and challenges. Environments 31(2):59-72.