APPENDIX 3. Dealing with frame diversity: evaluations and observations Four different kinds of interventions in the project have been observed from a social learning perspective. We used participant evaluations and our own observations during these workshops to assess the impact of these four kinds of intervention. How do they enable to deal with the ambiguity provoked by frame diversity? And do they foster constructive ways to connect knowledge from different backgrounds? Organizing interactive workshops: evaluations and observations To assess the impact of organizing interactive workshops, rather than relying on written communication, we have assembled evaluations and observations from 6 project workshops. We report the results thematically hereafter. Participants in interactive workshops valued them for several reasons: "People could see each other face to face" "Communication and work in the break-out groups, and small internal meetings" "We have communications that would not be possible through e-mail, e.g. integration" "People got to know what they can expect from each other" "People recognized where they fit in, they became aware of linkages" "We have tangible and intangible outcomes, like working with the group, which we will realize later" In this sense, the workshops contributed to the relational side of connecting the people using the different frames: getting to know each other and mutual expectations, finding a place and a role in the project, starting to work as a group. That the participants referred to the more interactive parts of the workshops as the more fruitful ones adds to this picture. Working in smaller groups was experienced as the most helpful to explore and define new concepts. "Break-out groups were the most interesting and useful discussions, the most interactive" "We need longer slots for small productive groups" "Smaller groups were really positive, working on a clear product keeps you focused" "In the discussion we were questioning a lot of things, ..., we were questioning each point we elaborated before, generating new ideas" "Products were better because of the small groups, I had fun moments in the break-out groups" Also working with an open format ("market") to stimulate intensive interactions was valued: "At the end of the market, I felt that something started to happen, I didn’t want to get back to the plenary" From our observations it was also clear that small group activities during the workshops generated much more lively and in depth discussions than the usual presentation plus questions format. According to the experiences of the participants, open communication contributes to the value of interactive workshops. "I experienced as positive that everybody is very willing to listen to each others views and opinions" "Very good atmosphere, trial and error approach without feeling bad about it is very constructive" One of the participants reported that his way of framing adaptive systems was changed through a workshop ("my mental model about adaptive systems evolved"). From our observations, an interactive discussion episode during a workshop about the difference between current water management, integrated water resources management and AM allowed an exploration and redefinition of these concepts among the participants in ways that would be very hard to achieve through other means. As a downside, frustration was also expressed with respect to the high investment of time and resources that workshops require. It seems like there is never enough time in a workshop: "It is not possible in one meeting" "There was not so much time as I expected" Furthermore, the important relational effects a workshop can have, impacts only the people who were there, and the results that are generated depend on the group of people that went through that process together. “At the same time I feel a bit uncomfortable about randomness: the composition of the group does matter in what we have as result, we should be very conscious about it” “I had a low moment this afternoon, because it seems like every meeting is starting anew” In this vein, a number of participants experienced as problematic that "people come in and jump out" or are “concerned about earlier leaving”. Facilitation: evaluations and observations Evaluations and observations from 4 project workshops served as a basis for assessing the impact of facilitation. At the end of the kick-off meeting of the project, the difficulty of taking the double role of presenting a lot of information and at the same time facilitating the discussion was identified in a debriefing between the organizers and observers of the meeting. Presenting calls for clear explanation and information flow from presenter to public. Facilitating the meeting calls for stimulating participation, checking comprehension, following up on comments and explicating (and checking) the goal of the meeting and what we expect as outputs. It’s not an easy job to combine both these roles in one person, and presenters seemed to have difficulties with it. This was one of the reasons why a next workshop was specifically designed and facilitated for exploring the different frames of reference of the participants and dealing with them in constructive ways, in order to reduce confusion and increase mutual understanding. This took the form of inviting participants to prepare a 2-slide presentation on their view of the workshop’s central topic. These presentations were then given at the start-up of the workshop. In this way, a number of differences in conceptions came to the fore, which might otherwise have remained implicit. The way this phase of the workshop was designed and facilitated allowed people to better understand the background of the others around the table and use that knowledge throughout the further discussions. Comments of the participants when evaluating the workshop show they valued the influence of the facilitators on the design of the workshop as well as the role they took during the workshop. "The moderation was very good, everyone was able to look beyond one's own perspective in the open break-out groups. We can now position ourselves better." "The complementary roles of presenting versus facilitating worked well." By contrast, in another workshop where only the presenters' role was taken and important differences in interpretations and views between the presenter and the other participants emerged but remained unaddressed, the following comment was given at the end of the day: "Did we have to meet for this ? We could have read this. I expected more guidance." Sometimes one person is able to fulfill both the roles of participating and facilitating. In that case process skills are combined with a good scientific understanding, the latter increasing the credibility of the facilitator. This situation was present in a workshop that was organized to change the dynamic of a rather polarized discussion between two research groups. A third group of researchers was brought in. They had their own alternative approach and thus contributed to the discussion at the content level. At the same time interventions on the procedural level were made by this third party, leading to shared action plans and thus improving the interaction process. The example shows that the facilitator’s role can be taken in an informal way if this is accepted by the group. One of the group members confirmed the effectiveness of this setting: "So in the meeting yesterday we brought in X, who hadn't been part to that, I mean, been part of some of the discussions. A new one can and could immediately see how it could work, could see a role for themselves. The rest of us were willing to allow this." Group model building: evaluations and observations To assess the impact of group model building, we analyzed observations and evaluations from the 2 first project workshops where UML was intensively used for building up joint conceptual frameworks. Several graphical notations for the graphical visualisation of the framework were discussed at project workshops and bilateral meetings. UML (unified modelling language) was finally adopted as the modelling notation for the project framework on transition towards AM. UML is not a running model itself. It is rather a notation system that allows documenting or specifying knowledge about objects, relations and associations, workflows and processes, responsibilities, information flows, interfaces, etc. UML supports different views or frames on the same part of the world. All ‘views’ or diagrams share the same terminology and their level of detail depends on the degree of information that is required to understand a certain problem. Due to this approach a diagram is a view into a model presented from the aspect of a particular perspective (e.g. a stakeholder), it provides a partial representation of the system, and it is semantically consistent with other views. Using UML in group discussions helps to make mutual assumptions explicit, because everybody attempts to translate his or her concepts into a common language. In selecting aspects, labelling them, drawing the relations and labelling the relations, differences between participants’ frames can emerge and can be discussed. It also helps to keep the attention focussed on the developing diagram and it results in a tangible output of the discussion (one or more diagrams). “making these diagrams is useful to access knowledge that’s in someone else’s head” “with the diagrams you have to become more precise” “very useful for the discussion because it focuses the discussion on a visual image” “we were talking and talking until now but now we have a clear result” “it is positive that each of the groups made good products” Making the diagrams helps to identify where knowledge of a system or process is incomplete. Information about some elements or links may be missing. “I think it helps to identify knowledge gaps, to see where our description is incomplete” “It was interesting to see the bits of information that were lacking” It’s important to embed the making of diagrams in a larger process and clarify the goals of making them (knowledge representation, integration, facilitating discussions, ...). Otherwise, people may not see the point of making the diagrams, or at the other extreme, people may start using UML indiscriminately. Keeping track of the developed UML diagrams and limiting their number by prioritizing will not be an easy task. “The first big discussion was: how we can use these diagrams or why do we need them in the process, and if we now start with this are they really used later” “we should be clear about what we want to get out of it from the beginning so that we don’t make a diagram for the sake of the diagram” “we should be very critical about which ones we make” In terms of finding a mutually workable representation, the top level representation (the “matrix”) itself may be the biggest challenge, since this captures world views on a high level of abstraction. In some cases, starting from representations of concrete subsystems may be easier (e.g. starting from the most important issues in a specific case). “We spent much time on the matrix and which cell was the most relevant for us. We ranked all the cells independently, and there were very different perceptions” A workable equilibrium between technically correct UML and easily understandable UML should be found. The best diagrams may be those where “UML-experts” watch the formal correctness, while “UML-laypersons” assure it is easily understandable for non-experts as well. “formally correct UML diagrams are not necessarily the most easily understandable ones” “we need set quality standards, if not we are prone to ambiguities in interpretation, we need quality check” A characteristic of UML as a formal language is that the visual lay-out of elements and relations in a specific view is technically meaningless: as long as the entities, attributes and relationships remain the same, you can rearrange the visual representation as you like without affecting what it means in UML. People are used however to derive meaning from the visual aspects of a diagram (above-below, left-right, close-distant, ...), and this may affect the meanings that are connected to the diagram. The advantage is that UML does not rely of these often implicit and possibly diverging meanings of the visual aspects of the diagram. The disadvantage is that people may still read those meanings into the visual aspects of the diagram and make diverging conclusions. A general concern with respect to the integration of different theories into a conceptual model, is the difficult distinction between (1) connecting concepts that represent different parts of reality, and (2) connecting concepts that categorize differently the same part of reality. The problem is that our concepts to some extent define what we take to be the reality. The more cautious approach may be to allow for parallel representations of parts of reality where or when necessary (different ways of framing the issue), and try to identify overlapping parts and look for complementarities where possible. Using concrete case contexts: evaluations and observations To analyze the impact of using concrete case contexts we used evaluations and observations from 3 project workshops. The third workshop was explicitly designed to profit as much as possible from the use of concrete case contexts. The idea of using concrete case contexts when confronted with diverging theoretical frameworks, is that the meaning of a concept, theory or method can be easier understood when we look at specific contexts and illustrate there what the concept means. The concrete case context can then provide the necessary common ground to discuss concepts that come from different backgrounds. In one of the facilitated workshops, case presentations were deliberately used for clarifying concepts related to AM. Two people who were each very familiar with a specific case, prepared an overview of the situation. The other participants were then invited to draw upon these inputs, and the presenters, to clarify and discuss the various concepts. A number of evaluations which were voiced during and at the end of the workshop referred to this approach, evaluating it as motivating and helpful. “we need examples for understanding what we mean, we may not have enough case studies in the project” “the early exemplary case approach was very helpful, it was good to have case study people around” “the case example taught me about complex adaptive systems, I understood better the meaning of transition as a natural process versus change as a result of deliberative actions” “the break-out groups on a very concrete case generated a lot of attention and energy, focusing on a concrete reality was enough to keep on contributing” “the case studies as background were useful, but could have been used even more” However, concerns were also raised that scientific rigor may suffer when focusing on concrete cases. “NeWater is an Integrated Project and needs to develop integrated concepts first before applying it to cases. Otherwise the quality of generalizable scientific insights will suffer.” In another project workshop, efforts to integrate different approaches for vulnerability assessment didn't succeed. A direct comparison of different research methods seemed to be too sensitive. Probably the position of both research groups as well as the differing contexts in which these methods have been applied, made a comparison very difficult. The decision to choose one “stylized situation” on which the different approaches under investigation would be applied, created some commonality to easier interpret the differences in the results. This approach uses case contexts not in their full complexity and detail, but through a simplified representation. The stylized situation still refers to a specific situation in a specific case and is thus different from an abstracted theoretical model. In the studied project, very different frames about uncertainty exist among scientists coming from different research traditions. In a workshop on uncertainty, concrete situations in which uncertainty was experienced by water management practitioners were collected through dialogue sessions with decision-makers in several case study contexts. By means of a few open questions they were invited to tell about specific uncertainty related situations they had experienced, and these were summarized as short stories or vignettes. The most striking illustrations of different types of uncertainty were selected from the stakeholder dialogues and presented at workshop among scientists. Scientists from different fields discussed in break-out groups how they would deal with the uncertainties in specific case situations. In the discussion participants could easily refer back to these case situations, which provided a common focus for the group discussions. The cases allowed them to present and explain better their concepts and the approach they would take by applying them to the case examples. In the evaluations of this workshop, participants referred to this way of working in positive terms. “it was positive that it was linked to practical examples, so we could combine the different approaches” “it was quite good to work very concretely, in small groups” “it was very interesting to see that you get other views on the vignettes, from very different perspectives”