Ecology and SocietyEcology and Society
 E&S Home > Vol. 19, No. 3 > Art. 10
The following is the established format for referencing this article:
Vervoort, J. M., D. H. Keuskamp, K. Kok, R. van Lammeren, T. Stolk, T. (A.) Veldkamp, J. Rekveld, R. Schelfhout, B. Teklenburg, A. Cavalheiro Borges, S. Jánoškóva, W. Wits, N. Assmann, E. Abdi Dezfouli, K. Cunningham, B. Nordeman, and H. Rowlands. 2014. A sense of change: media designers and artists communicating about complexity in social-ecological systems. Ecology and Society 19(3): 10.
http://dx.doi.org/10.5751/ES-06613-190310
Research

A sense of change: media designers and artists communicating about complexity in social-ecological systems

1Environmental Change Institute, University of Oxford, 2Soil Geography and Landscape group, Wageningen University, 3Alterra, 4Institute for Interdisciplinary Studies, University of Amsterdam, 5Laboratory for Geo-information Science and Remote Sensing, Wageningen University, 6ArtScience Interfaculty, Royal Arts Academy, 7Faculty of Geo-Information Science and Earth Observation, University of Twente, 8ArtScience Interfaculty, Royal Academy of Art, 9ArtScience Interfaculty, Royal Academy of Arts, 10Media Technology, Leiden University, 11Oxford Martin School Programme for the Future of Food, University of Oxford

ABSTRACT

To take on the current and future challenges of global environmental change, fostering a widespread societal understanding of and engagement with the complex dynamics that characterize interacting human and natural systems is essential. Current science communication methods struggle with a number of specific challenges associated with communicating about complex systems. In this study we report on two collaborative processes, a short workshop and longer course, that aimed to harness the insights of interactive media designers and artists to overcome these challenges. The two processes resulted in 86 new interactive media concepts which were selected by the participants and organizers using set criteria and then evaluated using the same criteria by a panel of communication and media design experts and a panel of complex systems scientists using the same criteria. The top eight concepts are discussed in this paper. These concepts fell into the categories of serious games, group interaction concepts, and social media storytelling. The serious games focused directly on complex systems characteristics and were evaluated to be intuitive and engaging designs that combined transparency and complexity well. The group interaction concepts focused mostly on feedbacks and nonlinearity but were fully developed and tested in the workshops, and evaluated as engaging, accessible, and easy to implement in workshops and educational settings. The social media storytelling concepts involved less direct interactions with system dynamics but were seen as highly accessible to large scale audiences. The results of this study show the potential of interdisciplinary collaboration between complex systems scientists, designers, and artists. The results and process discussed in this paper show the value of more structural engagement of interactive media designers and artist communities in the development of communication tools about human and natural systems change.
Key words: communication; complexity; participation; scale; serious gaming; social-ecological systems

INTRODUCTION

Accelerating, interconnected changes in all dimensions of life on earth, environmental, cultural, technological, economic, and political, have led to a world of deep complexity facing an uncertain future (Gallopín 2002). The consequences of ignorance of this reality and the lack of willingness and ability to deal with the challenges of global change are far-reaching environmental degradation and growing threats to human well-being (Millennium Ecosystems Assessment 2005).

New scientific paradigms identified under the umbrella of “complex systems science” have been developed to gain a more comprehensive understanding of the structures and dynamics of human and natural systems (Levin 1999). Complex systems science recognizes that traditional mono-disciplinary knowledge development fails to generate the types of understanding needed to govern the interactions of these systems (Holling and Gunderson 2002). Instead, social and environmental systems should be understood as interconnected “social-ecological systems” (Folke 2006).

Complex systems science has proven to be an effective way to organize and clarify information on systems across scales and disciplines (Jacobson and Wilensky 2006, van Bilsen et al. 2010). However, research in education science has shown that many of the basic principles at work in complex systems are counterintuitive. For those not deeply familiar with its concepts, a radical shift in perspective is required for the development of complex systems thinking (Jacobson and Wilensky 2006). Furthermore, developing a shared understanding is not enough to create the impetus for change in the face of future challenges. Actors across the societal spectrum must be engaged with global environmental change and willing to take concerted action to effect structural change (Vervoort et al. 2012a).

The challenge of communicating about complexity in social-ecological systems is exacerbated by the influence of simplistic dominant societal images, mental models, and discourses about environmental governance and sustainable development, such as “the equilibrium of nature” and narratives of endless economic growth and total controllability of natural systems (Richards 2001, Ravetz 2006, Beers et al. 2010).

Interactive media offer ways to overcome the limitations of societal communications on social-ecological change, showing potential for the facilitation of analytic understanding of complex systems (Andrienko and Andrienko 2007, Thomas and Cook 2005) as well as for the generation of affective engagement (Al-Kodmany 2002, Sheppard 2005, Freeman 2010, Vervoort et al. 2012a). These new media also offer ways to make communication about social-ecological complexity more accessible. Serious games, or games produced for learning purposes, form a prominent example, but there are many other virtual and physical approaches (Gooding 2008, Vervoort et al. 2010). In fact, there are social and geographical limitations to who can use digital interactive media as well, particularly in developing world contexts (Fuchs and Horak 2008, Brännström 2012).

More generally, current science communication tools and methods meet with a number of challenges when aiming to communicate about social-ecological complexity. In the research presented in this paper our purpose was to step outside the science communication niche and harness the potential of a wider field of interactive media for communication on complexity in social-ecological systems. To this end, we have organized two collaborations, a workshop and a course, where we brought together complex systems researchers and interactive media designers and artists to develop a range of communication concepts.

The challenges of communicating about social-ecological systems change

Communication about the nature of social-ecological systems change brings up a number of challenges from the perspective of complex systems science. We will refer to these as the “complexity challenges”:
A particular difficulty for science communication is that the above prerequisites for the understanding of systems change are at odds with basic design criteria from interaction design that are crucial for any effective communication (Resnick and Wilensky 1998, Cartwright 1999, Law and Van Schaik 2010). We will refer to these as the “design challenges”:
Because the collaboration between complex systems scientists, interactive media designers, and artists is largely uncharted territory with unmapped potential, we have deliberately not focused on the development of a single tool. Instead, we aimed to capture a wide range of perspectives on complexity communication. The collaborative processes in this paper provided many diverse responses to the challenges listed above.

Objectives

This paper focuses on the following question: How can interactive media designers and artists help science communicators to create communication concepts that take on the challenges of communicating about social-ecological systems change?

To address this question, we use the following objectives: (1) Evaluate a range of communication concepts resulting from the collaborative processes on their ability to deal with communication challenges associated with social-ecological systems; (2) Review the processes (workshop, course) themselves and draw conclusions on how collaborations between complex systems scientists and multimedia designers and artists can be set up effectively.

METHODS

Collaborative design processes and participants

We organized two collaborative processes with multimedia design and art education institutes in the Netherlands, in different formats. The first was a combined course, taking six weeks, for MSc students of the ArtScience program at the Royal Academy for Visual Arts, and students in the MSc program MediaTechnology at Leiden University. Twenty-nine students enrolled. This group was made up of students at the MSc level and consisted of a number of nationalities. Both the ArtScience and MediaTechnology MSc programs focus on a combination of arts, design, and science. With ArtScience, the scientific element serves as inspiration and the emphasis is on the artistic. Conversely, in the MediaTechnology master, the emphasis is on scientific research. Another, single-session workshop taking one afternoon was organized at the Utrecht Arts Academy. Eighty students attended that workshop, a mixture of BSc and MSc students from a wide range of art, design, and multimedia backgrounds. These two different processes helped us to examine how much could be achieved in a short engagement between the complex systems science and art/design disciplines, and what a longer collaborative process with participants who were already introduced to art/science interfaces would yield in comparison.

Process structure

In both the course and the workshop, we gave an opening presentation that introduced complexity in social-ecological systems in a step-wise fashion, discussing complexity in biophysical systems and in human systems. Subsequently we introduced ways in which human and biophysical systems are fundamentally connected (Westley et al. 2002). Following this, we discussed the meaning of this complexity in terms of environmental management and governance and the individual positions of societal actors, considering the present world and the future. We presented the challenges in communicating about social-ecological systems change discussed in this paper, but avoided examples of science communication.

We then set the following, deliberately open-ended, challenge for the students: Create a design that allows for the communication of complexity in interconnected natural and human systems. Design your concept so it does not just convey analytic understanding but provides an experiential sense of this complexity and what it means.

Students were given 30 minutes to come up with a first concept. Then, all students were asked to discuss and evaluate their own concepts and those of others in one-on-one speed-dating sessions, using a 10-point rating system (0 = extremely bad to 10 = extremely good) reflecting how well their or others’ concepts dealt with a number of criteria based on the design and communication challenges. Because of the widely different characters of the design concepts, detailed comparisons would be nonsensical, so the following basic design criteria were used, inspired by interaction design research (Pitt and Nassauer 1992, Norman 2002, Carpendale 2008, van Lammeren et al. 2010, Van Schaik 2010):
  1. The concept is accessible.
  2. The concept has the power to engage users.
  3. The concept conveys the right amount of information.
  4. The concept is transparent while capturing complexity.
  5. The concept is clear.
  6. The concept is designed to be intuitive.
Criteria 3 to 5 were designed to provide different perspectives on the transfer of information.

The Utrecht Arts Academy workshop ended after this first round of concepts and evaluation. The ArtScience/Media Technology course, however, was extended over five remaining weeks through three days of design work per week, an interim workshop to discuss students’ progress, and a final symposium where students’ ideas were presented.

Evaluation of concepts: the roles of participants, organizers, and expert panels

The workshop and course together resulted in a total of 86 concepts, in different stages of development; the ArtScience/MediaTechnology concepts were more developed because of the longer time allotted to them. Most concepts were not in a sufficient state of development that they could be tested on users. Instead, we chose to ask two independent panels of experts, one panel of four multimedia and communication experts (the “media panel”), and one panel of four complex systems scientists (the “science panel”), to judge a selection of concepts.

To limit the amount of time needed for these evaluations, the workshop organizers made a first selection of these concepts, bringing them down to the top 12 concepts best able to deal with the complexity as well as the design challenges. For this, a first selection of 51 concepts was based on a critical review of the participants’ numeric evaluation of preliminary versions of their own concepts and those of their fellow participants. The choice was made to focus on concepts that dealt well with as many challenges as possible to focus on integrated communication.

The influence and bias of the organizing research team has to be noted. The team consisted of three art/science researchers, and two complex systems researchers. Of the art/science researchers, two were artists and lecturers engaging with physical and digital media. The third had a background in literature. The researchers consisted of one researcher on social-ecological systems scenarios. The other researcher focused on work in physiology and ecology. The organizing team was therefore interdisciplinary in the domain of art, though design backgrounds were missing. The researchers lacked a pure social science perspective. These biases matter because for the initial selection of 12 concepts, where comparisons were difficult to make, the team of organizers used its own judgment.

The selection of the expert panels who evaluated 12 concepts from this first selection was done to provide a range of disciplinary perspectives. In addition, the experts where selected on five or more years of high-quality work experience and interdisciplinarity. The science panel consisted of a social scientist working on social-ecological systems complexity at multiple organizational levels, a social-ecological systems researcher focusing on interdisciplinary communication, a researcher with a background in agro-economics, and a geneticist working on micro-organism and cancer. The communication panel consisted of a multimedia/web designer and a software programmer, both experienced in developing software for science communication, an interaction designer working on multiactor communication systems, and a journalist/anthropologist.

Although the science panel reflects a range of disciplines, the communication panel was somewhat biased toward virtual and digital approaches. On reflection this bias reflects the bias of the researchers toward digital communication methods, seeing most potential there for upscaling.

The 12 concepts were evaluated by both panels by their perceived ability to deal with the mentioned design challenges. Additionally, the science panel judged the concepts on their ability to overcome the complexity challenges. The experts also used a 10-point rating system, but on a likert scale (-5 for extremely good to +5 for extremely bad). Experts were also able to comment on the concepts they reviewed. These criteria were used to evaluate the ability of the concepts to overcome the complexity challenges:
  1. The concept captures nonlinear system behavior.
  2. The concept captures feedbacks.
  3. The concept captures cross-scale dynamics.
  4. The concept captures path-dependence.
  5. The concept communicates the value of multiple perspectives on a system.
  6. The concept conveys how complex systems are characterized by deep uncertainty.
  7. The concept facilitates the development of strategic knowledge.
Again, these criteria were not designed to be detailed, because of the different natures of possible concepts. We also asked the panels which audiences and contexts they thought each concept would be most suitable for.

Finally, the roles of those involved in the reporting of the processes in this paper should also be noted. Aiming to make this project a full art/science collaboration, the authors included the organizers of the collaborative processes (artists and scientists) but also the participants who generated the top eight concepts.

RESULTS

Overview

The 86 concepts produced in the workshop and course covered a broad range of media. In the ArtScience/Media Technology workshop, the majority of the ArtScience students focused on dynamic visualizations, social media storytelling concepts, live performances, and interactive group installations. The Media Technology students focused on interactive, digital visualizations and games. The Utrecht Arts Academy workshop produced concepts covering a wider range of media, reflecting their more diverse backgrounds. Both processes, however, produced a mix of physical and virtual concepts.

From both workshops, 51 concepts were seen as having strong responses to one or more of the communication challenges. Appendix 1 summarizes these ideas.

Of the 12 selected concepts, 10 originated in the ArtScience/Media Technology workshop, while 2 of the concepts from Utrecht Arts Academy workshop were included in the final selection. A number of concepts from this shorter workshop consisted of ideas that were strong with regard to one or several communication challenges. In the ArtScience/Media Technology workshop, the concepts had more time to be realized. These concepts were seen as responding to a wider range of challenges. Four concepts were either judged weak or underdeveloped on most points by both panels.

The evaluations of the eight concepts described in the next sections by the participants, the media panel, and science panel on communication criteria are presented in Table 1. The evaluations done by participants were based on a first round of conceptualization where the concepts were still in a preliminary form. Therefore, the comparability of participants’ evaluations with the evaluations of the panels is somewhat limited. Also, a different though comparable 10 point scale was used (1 to 10 versus -5 to +5). The evaluation of the eight concepts based on complexity challenges by the science panel is presented in Table 2. The concept “Time Capsules” was added fairly late in the ArtScience/Media Technology workshop, and was therefore not selected through the above procedure but through a combined preference of the participants and organizers when this concept was discussed.

Game concepts

Organismus

A digital 3D organism building puzzle (Fig. 1) consisting of living cells suspended in space that change shape and properties over time. New cells appear and can be moved together by drag and drop with a controller. The pieces rarely fit as they are, but experimentation helps discover that they can adapt to each other depending on their general shape and properties (different nourishment requirements, adaptations) to temporarily form an organic colony. The maintenance and growth of the colony yields points. However, this colony has to survive in a contextual “soup” that provides limited nourishment and challenges and opportunities (changes in the suitability of the environment for the colony, or shocks) If the context changes too much for the cells to survive together, the organ can fall apart. The player can prevent this by changing and adapting the units.

The player can create multiple units or colonies that remain available and suspended in space and each have their own requirements and adaptations. If the player manages each unit well, he/she can combine these colonies into an organism, where each colony performs the function of a specialized organ, adding different sources of nourishment and adaptation strategies to the organism. On the organism level however, different change dynamics occur both internally and in its environment and the organism has higher requirements, which leads to different requirements on the management of the player. Also, the player will be less likely to have replacement elements at this level. On the upside, maintaining the organism level yields much more points per second. Strategies have to be revised constantly; different strategies apply on different levels. There is no ultimate solution to the game.

Participants rated Organismus very highly in all dimensions of communication, and saw its ability to convey information as exceptional. The media panel found this game lacking in accessibility because of its challenging premise, but solidly engaging, intuitive, and informative and possessing a good balance between transparency and complexity. Science panel members found this game to be informative, clear, and intuitive and found this game to be particularly suited to communicate principles of cross-level interactions and uncertainty. The game scored high in terms of creating a strategic knowledge development experience but low on making multiple perspectives available. Suitable contexts were college-level education and online communication.

Levels of Life

The player plays a series of microgames that take place on different biophysical levels, from the cellular level through individuals, societies and ecosystems to the global level (Figs. 2 and 3). This means managing gene replication, the survival of a cell, the organization of an organ, an individual’s role in a community, the community itself, and so on. The microgame goals and dynamics are particular to the level, but dynamics that are universal to complex adaptive systems should also stand out. This is pointed out at the beginning of each microgame; the player is told how much he/she can rely on experience from previous levels, and how much will be new. On each level, the player has to deal with influences playing out at other levels. The final goal of the game is to collect as much level linkages as possible, which are shown in a metamap between the levels (see Fig. 2). The regular path through the game is to move through the levels via different paths, but there are many places where level jumps can be made. These scale jump “bonus levels” show how cross-level influences are not just between consecutive levels. Think of playing at a single cell stage, and as a bonus getting to play a level where a bacterium becomes a pandemic, seen as a kind of strategic game at the appropriate level. See Figure 3 for examples of two bonus levels that link across scales.

The participants found this concept to be particularly strong on accessibility and a balance of complexity and transparency. The communication panel judged this game to be accessible and engaging with a good approach to combining of complexity and transparency. The complex systems science panel saw Levels of Life as accessible and engaging and very informative, communicating about scale interactions primarily but also on path dependence, perspectives, and uncertainties. Both panels saw the microgame structure as both positive in terms of learning curve and negative in terms of freedom of gameplay. Suitable contexts were secondary school, college, physical installations, and online communication.

ChaosGolf

ChaosGolf is an online game of virtual, 3D mini golf, where the player has to navigate a ball through a dynamic, moving landscape that is symbolic for choices and challenges faced by a manager of nature-society interactions (Fig. 4). The ball represents the state of the system that is managed, the club represents the management tool, putts represent actions, and the landscape represents the evolving field of opportunities and difficulties faced by the manager. This landscape changes dynamically while the game is played, in all dimensions, including the positions of the targets (holes). Club, ball, and landscape all respond to each other’s changes, as management tools, the managed entity and its context would. An idea to link this abstract representation to real issues is for the game to be framed by a context that explains in what ways the hole the player is playing represents a certain case or scenario. A level editor can allow participants to design their own levels to reflect certain cases or scenarios. The ball changes in shape and properties to reflect change of the subject or agent itself.

The preliminary evaluation by the participants found ChaosGolf to be particularly strong on accessibility and intuitive design. Media panel experts found this game to be accessible, intuitive, with a good balance between transparency and complexity. Their criticism was that the gameplay was not directly involved with the management of uncertainty, but rather with navigating it. Science panel members gave this game a good evaluation in terms of clarity and engagement criteria. They found its communication of complex systems characteristics to be mostly focused on nonlinearity, path dependence, and uncertainty. Suitable contexts were high-school education and online.

Spaceship Earth

Spaceship Earth is a scenario storytelling game, using a global perspective where the player “pilots” the planet. A moving globe (Fig. 5) is located within a tension field of axes that represent the different ways in which alternate global futures are polarized in scenario assessments: proactive world vs. reactive world, globalization vs. localization, economy first vs. environment first, and so on. The player starts out with quantities of several different types of capital, financial capital, social capital, knowledge capital, etc., that can be used to “steer” the world toward one variable or another. Different transitions will cost different ratios of these credits (see Fig. 5). The world itself will mirror the state it is in, depending on its position, through a storyline told on its surface. The globe is also the place where (impending) disasters or rapid positive developments can be seen. These developments will be consistent with the kind of world the player has steered toward at that moment: a strongly localized world will have problems with issues on the global level, whereas a strongly globalized world has to deal with the consequences of global governance schemes that are insensitive to the local scale. In this game, the player creates his/her own global future but it is not static, and players have to deal with the consequences of their choices. They also get a sense of the costs and challenges of making global transitions along these dimensions. The game can feature a large collection of different axes or “scenario sets” so that players can experiments with different sets of futures and their consequences. Further expansions can also vary the sets of resources used in the game.

The participants rated Spaceship Earth highly overall, but particularly in its ability to convey information and its intuitive design. The communication panel members appreciated this game for its accessibility, engagement, and clarity. Members of the science panel judged this game concept to be highly informative, clear, engaging, and intuitive. They saw its main communication being on nonlinearity and feedbacks, with cross-level issues and perspectives not being prominent. Suitable contexts were high-school, college, and online.

Group interaction concepts

Ouija drawing

A group of people sit around a table or on the floor, in a circle (Figs. 6 and 7). They are all connected at the wrist by a circular structure of connectors, so the movements of each person affect the movements of the others and vice versa. Each of them is given a pen and paper to draw objects. The individual drawing patterns represent local variability and agency, while the structure and properties of the connectors represent network structure as well as constraints (biophysical, institutional) on the nature of interactions. The group can be given different drawing assignments to reflect different system dynamics, for example:
This concept needed a lot of fleshing out after the first round of conceptualization and participants rated it moderately in accessibility and engagement, though they praised its intuitive design. By contrast, the media panel experts also saw the final version of this concept as very accessible, engaging, and clear but did not believe it was highly informative. The science panel also awarded the final version of this concept with a highly positive evaluation in terms of accessibility, engagement, information, combining transparency, and complexity, and intuitiveness. They saw it as mainly communicating about feedbacks, nonlinearity, and uncertainty. Suitable contexts were secondary school, college, workshops, and physical installations.

Breathing feedbacks

A group of people has to do a short physical exercise, like walking up and down some stairs, to speed up their breathing. Then, they are brought to a room where multiple microphones have been installed (Figs. 8 and 9). Each individual does nothing but breathe into a microphone. The sound of the group’s breathing is amplified throughout the room, providing auditory representation of individual and group feedback on the breathing. In the beginning, the breathing is chaotic and asynchronous, but after some time, people unconsciously sync up. This exercise works as a bodily felt metaphor for a complex adaptive system, like an ecosystem or a society, transitioning from a state of relative chaos to a state of relative equilibrium and synchronicity, because of system agents syncing up. Also, the breath is a bodily function that can switch between unconsciousness and conscious control, which extends the metaphor into the subject of adaptive management. The sound is recorded and analyzed and discussed further.

The participants rated the preliminary version of breathing feedbacks highly on information and clarity, though rather low on engagement. By contrast, the media panel thought the final version of this concept was very engaging, accessible, and clear, but not strong in terms of the amount of information communicated. This concept was judged by the science panel to be very highly accessible, offering a great combination of transparency and maintaining the complexity that makes it very clear and intuitive. A criticism was that it focuses almost exclusively on nonlinearity and feedbacks. Suitable contexts were high school, college, workshops, and possibly online.

Social media storytelling

Time Capsules

A fake set of documents (Fig. 10) is created and promoted as real; these documents appear to be a vision of the future of a seemingly deranged individual, who has a very rigid, one-dimensional perspective of the world, history, and the future. The document makes this person appear to be insane, and engages the reader, who thinks the document is real, to consider his/her own perspectives on the change of societal and natural systems through time. The document was presented in a group context as well as dispersed over the internet, where the illusion that it was real was maintained. There are two principles at the basis of this concept: presenting fabricated information as real, and eliciting the desired response by presenting the opposite perspective. The illusion of reality can make for an interaction with the concept that has very little framing, and can therefore be more direct and outside of a person’s expectations. The presentation of a perspective that is so obviously extreme in its static, peculiar dogmatism can elicit an opposite response. This combination of techniques makes this concept into a “scenario storytelling” exercise in a very different mode than is usual.

The Time Capsule concept was developed near the end of the ArtScience course and was therefore not reviewed by participants in the preliminary evaluation round. Both the media and science panels rated this concept highly on accessibility, very highly on engagement, but neither group found the idea particularly informative or transparent, because of its indirect way of communication. The science panel did not think the concept communicated specific features of complexity particularly well. Suitable contexts were high school, college, workshops, and online.

Indicators

The concept is a series of YouTube videos that shows a person bragging about his capacity to capture the gist of a language in 30 minutes sufficiently to talk and understand native speakers. He then demonstrates his method over a series of videos: a simple, box-relationship model that summarizes a handful of key words and relationships/grammar. This person proceeds to interact with different people using his model of the languages, and the video subtitles show him failing miserably and comically. The video then goes on to show how governments, companies, etc., develop indicators for ecosystem health, etc., building on the metaphor of what is left out in language. Examples from a preliminary version of this concept, in which the participant is shown attempting to learn and use all languages spoken by the diverse group of course participants in a matter of days, have again been made available online at: http://www.youtube.com/user/EstherDavita?feature=watch.

Participants found Indicators to be intuitive and informative as a preliminary concept. The media panel members were divided about the accessibility of this concept, but otherwise rated it positively on all fronts. The science panel was divided but mostly positive about the accessibility of this concept, but all found it to be engaging and informative. They saw it as mostly communicating on the limits of knowledge. Suitable contexts were high school, college, workshops, and online.

DISCUSSION

Overview

Most of the 51 concepts ranked highly by participants and summarized in the overview (Appendix 1) were not selected because they were underdeveloped. However, the quantity and diversity of ideas captured demonstrates the potential of the collaboration between complex systems scientists and interactive media designers and artists. Furthermore, the range of media and genres proposed for these concepts widens the perspectives and options that could be taken into consideration for the communication of complexity in social-ecological systems. Practical feasibility was a criterion that has not been included in the challenges. However, based on conversations it was an implicit consideration in the evaluations of the participants and the expert panels, which will be discussed.

A dominance of virtual rather than physical concepts in the final selection points to the organizers’ and panels’ bias toward virtual concepts noted in the methods section. This moderate preference for virtual concepts was based on the assumption that these provide better potential for up-scaling of the use of such concepts. However, the use of digital interactive media presumes access to such media that may not be available globally (DiMaggio et al. 2001, Fuchs and Horak 2008, Brännström 2012). Easily reproducible physical interaction concepts (Oujia Drawing) might be better to achieve scale in such conditions. The Time Capsules and Indicators concepts are not bound to their formats and can be reproduced as nondigital storytelling on paper or through a play.

Another bias of our selection approach was toward straightforward, explicit communication. Most of those concepts that interpreted systems complexity in an associative way, while perceived by participants and organizers as original and engaging, did not make the final selection. The Time Capsules and Indicators concepts were exceptions that provided associative communication while still fulfilling these criteria.

Game concepts

Organismus received the strongest value of the four concepts for its ability to generate strategic knowledge. The very different “network of microgames” approach of Levels of Life is a real strength in the eyes of both panels and the participants in terms of accessibility and engagement. It allows for the game to deal well with cross-level interactions. ChaosGolf was rated highly on design challenges by participants and panels but because the player navigates, rather than deals with, the dynamic landscape, the game goals are not intricately tied up with the challenges of complex systems. At once problematic and a strong point of Spaceship Earth is its focus on a single (global) level: it largely ignores cross-level interactions, but paradoxically allows for consciousness of the global integration of system changes.

Organismus with its intuitive design and frantic gameplay could be useful as a tool to teach about complex systems in a wide range of educational settings. As an educational tool, Levels of Life also shows much potential because of its piece-by-piece approach, learning curve design, focus on specific cases, and explicit emphasis on cross-scale interactions. ChaosGolf could quickly introduce a sense of dynamic system equilibria and changing contexts in an intuitive and engaging way. Spaceship Earth aims to develop understanding of the challenges of global governance, and of future uncertainties (scenarios).

In terms of feasibility, Organismus does not depend on powerful graphic design; however, the game mechanics are very complex, especially because of the nearly unlimited outcomes, and would require expertise, not so much in terms of manpower but in terms of knowledge and experience, to develop. Levels of Life is less challenging because of its microgame structure. However, there would be a large amount of content and designing to produce. A solution posed by one of the communication experts is to approach this game as an open-source project. ChaosGolf would not need to look spectacular, but instead rely on game-play for its engagement. As a web-based casual game, its game dynamics can be fairly easy to develop. Spaceship Earth could be fairly simple to design if it used relatively few variables and rigid categories of information tables for its scenario combinations.

The four games from the workshops show the potential of serious games to capture systems complexity in intuitive, engaging ways. In these qualities, the games have few precedents. The biggest drawback of all of these games is their lack of focus on capturing the perspectives and contributions of their players. A related gap in these games is the social, multiplayer element. In terms of feasibility they follow the trend of web-based casual games that are often developed by a few experts (Ossmann and Miesenberger 2010).

Previous serious gaming efforts to capture social-ecological complexity include BBC Climate Challenge (http://www.bbc.co.uk/sn/hottopics/climatechange/climate_challenge/ 2007) and Fate of the World (http://fateoftheworld.net/). The successes of these games give additional credibility to the assertion that games such as those proposed in this study, different as their premises might be, can be feasible because they follow similar design principles. Examples outside serious gaming include Spore (http://www.spore.com), which allows players to develop their own organisms, tribes, nations, and interstellar civilizations from the ground up in a way that bears similarities to Levels of Life, From Dust (http://fromdustgame.com/), which features geophysical modeling that can be manipulated by players, or StarCraft 2 (http://eu.blizzard.com/en-gb/games/hots/), which allows for intense, dynamic interactions between multiple players and computer agents. Games such as Spore, From Dust, and StarCraft 2 additionally allow for user-generated and modified content based on physical and geographical models. Table 3 summarizes avenues for future research and development of games that emerged from this research.

Interactive group concepts

The science and media panels disagreed about the amount of information that could be conveyed with Ouija Drawing, though it received high ratings otherwise. We believe this depends on paying attention to what information guides the exercise. In terms of the complex systems criteria, the science panel saw a problem with the communication of scales and multiple perspectives. We as the authors would like to argue that cross-level dynamics can be recognized in the exercise, represented by the constraints of the group structure. Concerning Breathing Feedbacks, the panels disagreed again about the amount of information communicated in contrast to high ratings for other indicators. Again, this could be considered a question of guiding information to explain implications of the interaction. Ouija Drawing can be used in educational or workshop settings as an engaging and direct intervention to quickly give a sense of the role of feedbacks and nonlinearity in complex systems; participants literally “step into the system.” Both physical concepts have the benefit of utilizing a haptic or embodied mode of knowledge generation to explore system dynamics, instead of disembodied, purely cognitive learning (Nash 2000, Picart 2002, Crang 2003). Ouija Drawing could easily be used in a range of settings, as long as these settings are physical. The need for the right electronic equipment makes the Breathing Feedbacks concept more difficult to implement.

The use of interactive group concepts for the communication of complexity has some precedence (Resnick and Wilensky 1998, Cartwright 1999). However, the ways groups were used in the concepts in this workshop bring up new ways to embody complex systems dynamics that warrant future exploration. Their advantages lie in how practically and engagingly they create intuitive understanding of systems dynamics by using the human body and mental processes in group settings. A potential disadvantage lies in the fact that they do not scale up as easily as for instance web-based games do, at least not in the sense of instant playability when computers and internet are available. Indeed, Breathing Feedbacks requires considerable physical technology. Ouija Drawing, however, is easily replicable with very little resources, making it scalable where digital concepts would not be possible (Fuchs and Horak 2008). Table 3 presents the recommendations for physical group interaction concepts.

Social media storytelling

The high values given to Time Capsules by both panels in terms of both accessibility and engagement have a lot to do with its flexible format (online, physical) and its intriguing premise. The strength of the concept is based in two ideas: presenting fictional information as if it is real, and using a reverse message to convey the communication. The drawback of this approach is that it is associative and can be problematic in its clarity. The communication panel commented that much of the impact of this concept relies on later framing of the lessons learned. The complexity panel saw Indicators as being somewhat indirect, but communication panel praised it for its ability to translate the complex concept of uncertainty and lack of knowledge to an understandable metaphor. This reflects the participants’ opinion of the preliminary version. However, in communicating a main idea rather than displaying system dynamics as some concepts in previous categories, Indicators was not evaluated as able to capture many aspects of complex systems well, apart from an understanding of uncertainty. Time Capsules, Indicators, and similar concepts have potential in providing a reframe or a wake-up call in a range of educational and organizational settings. Their viral quality allows it to potentially upscale their impact (Bodin et al. 2006). Because both projects have already been tested online and elicited the responses that were aimed for, feasibility is not an issue. Table 3 summarizes recommendations for social media storytelling.

Untapped potential: collaboration between social-ecological systems scientists and designers and artists

The Utrecht Arts Academy workshop was a single day session with a large number of participants. This workshop generated ideas that showed much potential, but participants lacked the time to fully shape these ideas. In the ArtScience/ Media Technology workshop, there were less participants, but there was much more time to develop the concepts. However, despite this large difference in time, two concepts from the single day workshop still made it to the final selection.

Our conclusion is that although more limited, a shorter workshop can bring up a broad range of ideas. This is useful when the time frame for collaboration is in fact limited, for instance, when a large group of researchers and high-level designers and artists can be brought together for a single occasion.

Both formats asked for communication concepts rather than necessarily finished products. This meant that no marked differences were observed in the types of concepts (such as virtual versus physical) that were proposed; concepts were essentially a result of the backgrounds of participants, which were highly diverse.

We as the authors propose that in addition to having had more time, those involved in the ArtScience/Media Technology program were able to generate the most useful concepts because they were already undergoing a training that included the reflections of art and science on one another. We see the education of hybrid artists/scientists as a crucial requirement for the emergence of communication strategies that create public engagement with and understanding of social-ecological change.

A fundamental design choice was the view that a single concept should be able to deal with most or all of the challenges of communicating about complex systems the researchers posed. An alternative approach could be to focus on single challenges and aim for a toolbox instead, though this does not acknowledge the interconnectedness of such concepts. Also, our use of a set list of criteria can have limited our evaluation of the concepts by using a single frame on a wide variety of ideas rather than grounded theory (Isenberg et al. 2008).

Another design choice was that complex systems science was presented to artists and designers. Other formats can be considered, such as putting complex systems science students together. Another approach would be to have the evaluation be user-driven when testable concepts can be produced online or in physical spaces to interact with potential users in an iterative fashion, broadening concepts of “useful” beyond expert opinion. This could focus on inspiring communities of interactive media designers and artists to join long-term, collaborative projects (e.g., Cape Farewell http://www.capefarewell.com/home.html, or Tipping Point http://www.tippingpoint.org.uk/).

Our results demonstrate that there are many affinities between modes of communication in interactive media and art and in complex systems science. Art is appreciative of complexity and uncertainty and has the ability to convey understanding on an analytic level as well as through experience. Aesthetic appreciation has its own type of rationality that warrants more exploration in the context of complex systems communication (Isenegger et al. 2005, Law and Van Schaik 2010, Ramírez and Ravetz 2011). Criteria such as those in this paper may still be too limited to value concepts that use such aesthetic rationality.

CONCLUSIONS

In this study, we started with the premise that interactive media designers and artists can contribute new perspectives and ideas to generate societal engagement with and understanding of complexity in social-ecological systems. Design concepts developed in collaborative workshops yielded top concepts that were highly diverse but proved engaging, accessible, intuitive, and able to combine challenges of complexity. None of these concepts have much precedence in science communication.

Overall, the concepts had the most difficulty with capturing scale interactions, and they provided little opportunities for the contribution of participants’ perspectives. The digital concepts had a high potential for scaling up learning, experimentation, and interaction, but a high dependency on access to technology. The physical concept Ouija Drawing, by contrast, had high potential for replication where less technology is available, and both group concepts had high potential for haptic learning. The concepts could be used in education, workshops, and online platforms to help facilitate shifts in understanding from linear, central models of understanding to complexity-based understanding.

Though the physical group concepts and social media storytelling concepts were actualized and the games stayed in the conceptual phase, the feasibility of developing these game concepts is fairly high. The evaluation of the games has remained hypothetical, however, which is a limit of this study.

The workshop outcomes in this study demonstrate the value and potential of interdisciplinary exchange and education that combines science, art, and design. It provides one approach for such exchange, but other, more horizontal, iterative, and user-oriented approaches are yet to be explored. Such collaborations could produce communities of interdisciplinary innovators that could help overcome the limits of societal communication on social-ecological systems change.

RESPONSES TO THIS ARTICLE

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.

ACKNOWLEDGMENTS

Our thanks go out to all participants in the Art/Science, Leiden University MediaTechnology and StudiumGenerale workshops. Thanks to Annabeth Simonsz of the MediaTechnology MSc. Programme and Willem Dugardijn of the Utrecht Arts Academy for their support in setting up the workshops. Thanks to the Royal Academy of Arts, Leiden University, and the Utrecht Arts Academy for hosting these workshops. We would like to thank the experts in the design and complex systems science panels for their time and insights. We thank the TransForum research programme for enabling this research. Finally, we would like to thank the anonymous reviewers for their insightful comments.

LITERATURE CITED

Al-Kodmany, K. 2002. Visualization tools and methods in community planning: from freehand sketches to virtual reality. Journal of Planning Literature 17(2):189-211. http://dx.doi.org/10.1177/088541202762475946

Andrienko, N., and G. Andrienko. 2007. Designing visual analytics methods for massive collections of movement data. Cartographica: The International Journal for Geographic Information and Geovisualization 42(2):117-138. http://dx.doi.org/10.3138/carto.42.2.117

Beers, P. J., A. Veldkamp, F. Hermans, D. van Apeldoorn, J. M. Vervoort, and K. Kok. 2010. Future sustainability and images. Futures 42(7):723-732. http://dx.doi.org/10.1016/j.futures.2010.04.017

Bishop, I. D. 2011. Landscape planning is not a game: Should it be? Landscape and Urban Planning 100(4):390-392. http://dx.doi.org/10.1016/j.landurbplan.2011.01.003

Bodin, Ö., B. Crona, and H. Ernstson. 2006. Social networks in natural resource management: What is there to learn from a structural perspective? Ecology and Society 11(2): r2. [online] URL: http://www.ecologyandsociety.org/vol11/iss2/resp2/

Brännström, I. 2012. Gender and digital divide 2000-2008 in two low-income economies in Sub-Saharan Africa: Kenya and Somalia in official statistics. Government Information Quarterly 29(1):60-67. http://dx.doi.org/10.1016/j.giq.2011.03.004

Carpendale, S. 2008. Evaluating information visualizations. Information Visualization: Lecture Notes in Computer Science 4950:19-45. http://dx.doi.org/10.1007/978-3-540-70956-5_2

Cartwright, W. 1999. Extending the map metaphor using web delivered multimedia. International Journal of Geographical Information Science 13(4):335-353. http://dx.doi.org/10.1080/136588199241238

Cowley, B., D. Charles, M. Black, and R. Hickey. 2008. Toward an understanding of flow in video games. Computers in Entertainment 6(2). http://dx.doi.org/10.1145/1371216.1371223

Crang, M. 2003. Qualitative methods: touchy, feely, look-see? Progress in Human Geography 27:494-504. http://dx.doi.org/10.1191/0309132503ph445pr

DiMaggio, P., E. Hargittai, W. Russell Neuman, and J. P. Robinson. 2001. Social implications of the internet. Annual Review of Sociology 27:307-336. http://dx.doi.org/10.1146/annurev.soc.27.1.307

Dörner, D. 1996. The logic of failure: why things go wrong and what we can do to make them right. Metropolitan Books, New York, New York, USA.

Edge Staff. 2007. 50 Greatest Game Design Innovations. 1 November. [online] URL: http://www.next-gen.biz/features/50-greatest-game-design-innovations?page=0%2C3

Folke, C. 2006. Resilience: the emergence of a perspective for social-ecological systems analyses. Global Environmental Change 16(3):253-267. http://dx.doi.org/10.1016/j.gloenvcha.2006.04.002

Folke, C., S. R. Carpenter, B. Walker, M. Scheffer, T. Chapin, and J. Rockström. 2010. Resilience thinking: integrating resilience, adaptability and transformability. Ecology and Society 15(4): 20. [online] URL: http://www.ecologyandsociety.org/vol15/iss4/art20/

Freeman, W. 2010. Red redemption: socially aware games can be commercial. Develop, 13 July. [online] URL: http://www.develop-online.net/news/35345/Red-Redemption-Socially-aware-games-can-be-commercial

Fuchs, C., and E. Horak. 2008. Africa and the digital divide. Telematics and Informatics 25(2):99-116. http://dx.doi.org/10.1016/j.tele.2006.06.004

Gallopín, G. C. 2002. Planning for resilience: scenarios, surprises, and branch points. Pages 361-394 in L. H. Gunderson and C. S. Holling, editors. Panarchy: understanding transformations in systems of humans and nature. Island, Washington, D.C., USA.

Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, and M. Trow. 1994. The new production of knowledge: the dynamics of science and research in contemporary societies. Sage, London, UK.

Gooding, J. 2008. Web 2.0: a vehicle for transforming education. International Journal of Information and Communication Technology Education 4(2):44-53. http://dx.doi.org/10.4018/jicte.2008040104

Hmelo-Silver, C. E., and R. Azevedo. 2006. Understanding complex systems: some core challenges. Journal of the Learning Sciences 15(1):53-61. http://dx.doi.org/10.1207/s15327809jls1501_7

Holling, C. S., and L. H. Gunderson. 2002. In quest of a theory of adaptive change. Pages 3-24 in L. H. Gunderson and C. S. Holling, editors. Panarchy: understanding transformations in systems of humans and nature. Island, Washington, D.C., USA.

Isenberg, P., T. Zuk, C. Collins, and S. Carpendale. 2008. Grounded evaluation of information visualizations. Proceedings of the 2008 Conference on BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization. ACM, New York, New York, USA. http://dx.doi.org/10.1145/1377966.1377974

Isenegger, D., B. Price, Y. Wu, A. Fischlin, U. Frei, R. Weibel, and B. Allgöwer. 2005. IPODLAS - A software architecture for coupling temporal simulation systems, VR, and GIS. ISPRS Journal of Photogrammetry and Remote Sensing 60(1):34-47. http://dx.doi.org/10.1016/j.isprsjprs.2005.10.003

Jacobson, M. J., and U. Wilensky. 2006. Complex systems in education: scientific and educational importance and implications for the learning sciences. Journal of the Learning Sciences 15(1):11-34. http://dx.doi.org/10.1207/s15327809jls1501_4

Kim, J. Y., J. P. Allen, and E. Lee. 2008. Alternate reality gaming. Communications of the ACM 51(2):36-42. http://dx.doi.org/10.1145/1314215.1340912

Kok, K., and T. (A.) Veldkamp. 2011. Scale and governance: conceptual considerations and practical implications. Ecology and Society 16(1): 23. [online] URL: http://www.ecologyandsociety.org/vol16/iss2/art23/main.html

Law, E. L. C., and P. Van Schaik. 2010. Modelling user experience - an agenda for research and practice. Interacting with Computers 22(5):313-322. http://dx.doi.org/10.1016/j.intcom.2010.04.006

Levin, S. 1999. Fragile dominion: complexity and the commons. Perseus, Cambridge, Massachusetts, USA.

Mennin, S. 2007. Small-group problem-based learning as a complex adaptive system. Teaching and Teacher Education 23(3):303-313. http://dx.doi.org/10.1016/j.tate.2006.12.016

Millennium Ecosystems Assessment. 2005. Ecosystems and human well-being: synthesis. Island, Washington, D.C., USA.

Nash, C. 2000. Performativity in practice: some recent work in cultural geography. Progress in Human Geography 24(4):658-664. http://dx.doi.org/10.1191/030913200701540654

Norman, D. A. 2002. Emotion & design: attractive things work better. Interactions 9(4):36-42. http://dx.doi.org/10.1145/543434.543435

Ossmann, R., and K. Miesenberger. 2010. Accessibility of a social network game. Computers Helping People with Special Needs: Lecture Notes in Computer Science 6179:243-246. http://dx.doi.org/10.1007/978-3-642-14097-6_39

Parker, L. 2010. The Future of AI in Games. GameSpot, 10 November. [online] URL: http://www.gamespot.com/features/6283722/index.html

Picart, C. J. 2002. Dancing through different worlds: an autoethnography of the interactive body and virtual emotions in ballroom dance. Qualitative Inquiry 8(3):348-361. http://dx.doi.org/10.1177/10778004008003009

Pitt, D. G., and J. I. Nassauer. 1992. Virtual reality systems and research on the perception, simulation and presentation of environmental change. Landscape and Urban Planning 21(4):269-271. http://dx.doi.org/10.1016/0169-2046(92)90038-2

Ramírez, R., and J. Ravetz. 2011. Feral futures: Zen and aesthetics. Futures 43(4):478-487. http://dx.doi.org/10.1016/j.futures.2010.12.005

Ravetz, J. R. 2006. Post-normal science and the complexity of transitions towards sustainability. Ecological Complexity 3(4):275-284. http://dx.doi.org/10.1016/j.ecocom.2007.02.001

Red Redemption. 2007. Climate Challenge. [online] URL: http://makesyouthink.net/games/operation-climate-control/

Red Redemption. 2011. Fate of the world. [online] URL: http://www.red-redemption.com/

Resnick, M., and U. Wilensky. 1998. Diving into complexity: developing probabilistic decentralized thinking through role-playing activities. Journal of the Learning Sciences 7(2):153-172. http://dx.doi.org/10.1207/s15327809jls0702_1

Richards, R. 2001. A new aesthetic for environmental awareness: chaos theory, the beauty of nature, and our broader humanistic identity. Journal of Humanistic Psychology 41(2):59-95. http://dx.doi.org/10.1177/0022167801412006

Rieber, L. P. 1996. Seriously considering play: designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development 44(2):43-58. http://dx.doi.org/10.1007/BF02300540

Sefton, J. 2008. The roots of open-world games. GamesRadar, 11 July. [online] URL: http://www.gamesradar.com/f/the-roots-of-open-world-games/a-200807111086555044

Sheppard, S. R. J. 2005. Landscape visualisation and climate change: the potential for influencing perceptions and behaviour. Environmental Science & Policy 8(6):637-654. http://dx.doi.org/10.1016/j.envsci.2005.08.002

Squire, K. D. 2008. Video game–based learning: an emerging paradigm for instruction. Performance Improvement Quarterly 21:7-36. http://dx.doi.org/10.1002/piq.20020

Sweeney, L. B., and J. D. Sterman. 2007. Thinking about systems: student and teacher conceptions of natural and social systems. System Dynamics Review 23(2-3):285-311. http://dx.doi.org/10.1002/sdr.366

Thomas, J. J., and K. Cook. 2005. Illuminating the path: the research and development agenda for visual analytics. National Visualization and Analytics Center, Richland, Washington, USA.

van Bilsen, A., G. Bekebrede, and I. Mayer. 2010. Understanding complex adaptive systems by playing games. Informatics in Education 9(1):1-18.

van der Sluijs, J. 2005. Uncertainty as a monster in the science-policy interface: four coping strategies. Water Science and Technology 52(6):87-92.

van Lammeren, R., J. Houtkamp, S. Colijn, M. Hilferink, and A. Bouwman. 2010. Affective appraisal of 3D land use visualization. Computers, Environment and Urban Systems 34(6):465-475. http://dx.doi.org/10.1016/j.compenvurbsys.2010.07.001

Van Schaik, P. 2010. Using interactive 3-D visualization for public consultation. Interacting with Computers 22(6):556-568. http://dx.doi.org/10.1016/j.intcom.2010.06.002

Vervoort, J. M., K. Kok, P. J. Beers, R. Van Lammeren, and R. Janssen. 2012a. Combining analytic and experiential communication in participatory scenario development. Landscape and Urban Planning 107(3):203-213. http://dx.doi.org/10.1016/j.landurbplan.2012.06.011

Vervoort, J. M., K. Kok, R. van Lammeren, and T. Veldkamp. 2010. Stepping into futures: exploring the potential of interactive media for participatory scenarios on social-ecological systems. Futures 42(6):604-616. http://dx.doi.org/10.1016/j.futures.2010.04.031

Vervoort, J. M., L. Rutting, K. Kok, F. L. P. Hermans, T. Veldkamp, A. K. Bregt, and R. van Lammeren. 2012b. Exploring dimensions, scales, and cross-scale dynamics from the perspectives of change agents in social-ecological systems. Ecology and Society 17(4): 24. [online] URL: http://www.ecologyandsociety.org/vol17/iss4/art24/

Westley, F., S. R. Carpenter, W. A. Brock, C. S. Holling, and L. H. Gunderson. 2002. Why systems of people and nature are not just social and ecological systems. Pages 103-120 in L. H. Gunderson and C. S. Holling, editors. Panarchy: understanding transformations in systems of humans and nature. Island, Washington, D.C., USA.

Wilensky, U., and M. Resnick. 1999. Thinking in levels: a dynamic systems approach to making sense of the world. Journal of Science Education and Technology 8(1):3-19. http://dx.doi.org/10.1023/A:1009421303064

Wilkinson, A. 2009. Scenarios practices: in search of theory. Journal of Futures Studies 13(3):107-114.

Wilkinson, A., and E. Eidinow. 2008. Evolving practices in environmental scenarios: a new scenario typology. Environmental Research Letters 3:045017. http://dx.doi.org/10.1088/1748-9326/3/4/045017

Wilkinson, A., and R. Ramirez. 2010. Canaries in the mind: exploring how the financial crisis impacts 21st century future-mindfulness. Journal of Futures Studies 14(3):45-60.

Erratum: The order in which the figures appear was changed on 1 August 2014.

Address of Correspondent:
Joost M. Vervoort
Environmental Change Institute
Oxford University Centre for the Environment, South Parks Road
Oxford, Oxfordshire
United Kingdom
OX1 3QY
joost.vervoort@eci.ox.ac.uk
Jump to top
Table1  | Table2  | Table3  | Figure1  | Figure2  | Figure3  | Figure4  | Figure5  | Figure6  | Figure7  | Figure8  | Figure9  | Figure10  | Appendix1