Ecology and SocietyEcology and Society
 E&S Home > Vol. 19, No. 4 > Art. 20
The following is the established format for referencing this article:
Janssen, M. A., A. Lee, and T. M. Waring. 2014. Experimental platforms for behavioral experiments on social-ecological systems. Ecology and Society 19(4): 20.
http://dx.doi.org/10.5751/ES-06895-190420
Synthesis, part of a special feature on Advancing the Understanding of Behavior in Social-Ecological Systems: Results from Lab and Field Experiments

Experimental platforms for behavioral experiments on social-ecological systems

1Arizona State University, 2University of Maine

ABSTRACT

Recently, there has been an increased interest in using behavioral experiments to study hypotheses on the governance of social-ecological systems. A diversity of software tools are used to implement such experiments. We evaluated various publicly available platforms that could be used in research and education on the governance of social-ecological systems. The aims of the various platforms are distinct, and this is noticeable in the differences in their user-friendliness and their adaptability to novel research questions. The more easily accessible platforms are useful for prototyping experiments and for educational purposes to illustrate theoretical concepts. To advance novel research aims, more elaborate programming experience is required to either implement an experiment from scratch or adjust existing experimental software. There is no ideal platform best suited for all possible use cases, but we have provided a menu of options and their associated trade-offs.
Key words: education; lab experiments; research; software

INTRODUCTION

The quest for sustainability is fraught with social dilemmas regarding the use of shared resources. Social dilemmas occur when individual incentives conflict with collective interests. This is especially apparent with shared resources such as forests, fish, clean and fresh water, and fertile land. Garrett Hardin (1968) argued that resource users cannot govern their own resources and that a tragedy of the commons is unavoidable without the establishment of state control or privatization of the shared resource. To the contrary, Elinor Ostrom and her colleagues showed that such tragedy is avoidable (Ostrom 1990, Dietz et al. 2003) by gathering many empirical examples of common resources that are successfully managed by local communities that create innovative institutional arrangements.

An important contribution to the success of the study of governance of common pool resources is the theoretical framework called the institutional analysis and development framework, which is designed to help understand the complexity of empirical cases of institutional design (Ostrom 2005). A recent extension of this work is the emphasis of the social and ecological contextual variables in the social-ecological systems framework (Ostrom 2009). The study of governance of social-ecological systems cannot be performed by one methodological approach but requires investigation using different approaches such as case study analysis, formal modeling, and behavioral experiments (Poteete et al. 2010). We focus on experimental methods.

Behavioral experiments in the laboratory and field are increasingly used to test hypotheses emerging from theory and case study research (Ostrom et al. 1994, Poteete et al. 2010). Experiments have been performed with university student populations in various countries, as well as nonstudent populations, including farmers in rural India or fishers from the coast of Colombia (Anderies et al. 2011, Waring and Bell 2013). These experiments show that overharvesting of common resources happens as theory predicts if participants cannot communicate and cannot use costly punishment. In contrast to theory, and in line with case study research, experiments show that communication and costly punishment reduce the overharvesting of common resources (Poteete et al. 2010). Communication without the ability to enforce promises, called cheap talk, is not expected to have an effect on rational selfish actors, yet leads to a significant increase in group cooperation (Balliet 2010). The option to reduce the earnings of others at a cost to oneself, i.e., costly punishment, is not expected to be used. However, both cheap talk and costly punishment are found to have a significant effect and confirm observations in case study analysis (Ostrom et al. 1992).

One limitation of early public goods and common pool resource experiments for the use of studying governance of social-ecological systems was the lack of relevant ecological complexity. Scholars have begun to explore the consequences of more specific and relevant ecological complexity in the study of social dilemmas in social-ecological systems (Janssen et al. 2010, McAllister et al. 2011, Prediger et al. 2011, García-Gallego et al. 2012, Anderies et al. 2013, Cardenas et al. 2013, Kimbrough and Wilson 2013). A result of increased scientific interest in social-ecological experiments has been the parallel proliferation of experimental software designed to implement them. Experimental software platforms have different aims, code bases, features, and user communities. This diversity in software platforms is valuable but also creates difficult choices for researchers to determine which platform is best suited for their needs and presents challenges for standardization and interoperability, reproducibility, extensibility, and reuse.

One of our purposes is to address common inquiries by colleagues and students on how to start conducting social-ecological behavioral experiments. Beyond understanding the methodologies for designing and running experiments (Friedman and Sunder 1994, Roth and Kagel 1995, Camerer 2003), it is important to know what kinds of software tools are available and appropriate. Like most software applications, software experiments are typically developed for a specific set of research questions and assumptions, and advanced programming experience is needed for major adjustments. However, most frameworks provide “extension points” that enable experimenters to modify features of an experiment via configuration parameters or even custom programming logic. Some experiments can also be implemented via standard tools or platforms used for other purposes. We review the various software tools and platforms available for conducting behavioral social-ecological experiments with an eye toward clarifying the options. We restrict experimental platforms to those that are publicly available and include examples of public good and/or common pool resource experiments. We evaluate these platforms on their goals, design, abilities, and usage. We do not intend to provide an overview of experimental platforms in experimental economics but purposely focus on a specific problem domain for scholars from interdisciplinary backgrounds who are interested in using experimental approaches for their studies of the interactions between humans and their environment.

EXPERIMENTAL APPROACHES TO GOVERNANCE OF SOCIAL-ECOLOGICAL SYSTEMS

We discuss some common variations of standard experiments. This will provide a perspective on the scope of possible experiments and the justification for certain choices of platforms. All relevant experiments are designed to test specific hypotheses. To do so, we define different treatments. For example, to test the effect of communication, we will need a number of groups with and without communication allowed. A sufficient number of groups need to be formed such that nonparametric tests can be used to test significance of the differences attributable to the treatment effect.

Experiments are often designed intentionally to be simplistic and narrow in scope to focus on specific research questions and exclude sources of noise present in the real world. Results from experiments by themselves do not prove the effect of an intervention and are part of a larger body of experimental results and findings from other studies that accumulatively lead to improved insights (Poteete et al. 2010).

Because social-ecological systems often contain social dilemmas with environmental implications, we limit the scope of our examination to experimental platforms used to examine public good and common pool resource scenarios. Experiments reported in other papers in this special issue on behavioral experiments for social-ecological systems are variations of the public good and common pool resources.

In a standard linear public good experiment, a group of N participants receives an endowment (w) every round. In each round, participants decide independently how much (X) of the endowment to invest in a public fund and how much to invest in a private fund. The total investment in the public fund is multiplied by (m) and shared equally among the participants. A participant’s earnings in a given round are the remainder of the private fund (wX) plus an even share of the public fund (m/N) × sum X. A selfish rational actor will not invest anything in the public fund. If all participants are selfish and rational, the earnings will be equal to the initial endowment. The social optimum solution is obtained, by contrast, if every participant invests his or her entire endowment in the public good, increasing the earnings by m.

Common pool resource experiments are similar to public good experiments except that the decision to be made is how much to extract from a common pool resource in contrast to how much to invest in a private fund. A selfish rational actor will harvest as much as possible and will expect others to do the same. The social optimum resource consumption is a lower consumption that provides social benefits such as resource longevity, or greater harvest per unit effort. In both public good and common pool resource scenarios, N participants make numerical investments or harvesting decisions in parallel, every round.

We use the institutional analysis and development framework (Ostrom 2005) to discuss what kinds of variations of so-called action situations we can explore in experiments (Fig. 1). Participants can fill positions that allow certain actions. Individuals have control over the actions that lead to outcomes and have information about actions, outcomes, and their linkages, as well as their costs and benefits. We can vary who participates, e.g., students and farmers; in which kind of ecology, varying in spatial and temporal dynamics; and from which “culture,” e.g., rural/urban populations, hunter-gatherers, or small-scale rice farmers in Nepal. In the experiments, we can vary the production functions, i.e., net cost and benefits, which may reflect different types of ecologies, i.e., how the actions of individuals affect the payoff of other participants. The ecological context can also affect the positions participants have, e.g., connected in networks or via asymmetric interactions; when decisions can be made; and who has information about the actions and earnings of others. The variations can be endless yet are guided by observations in case studies and a theoretical framework that captures the interactions of people.

Modern computer software and programming languages now make it possible for experimenters to create complex, interactive, multiparticipant experiments with highly specific social roles and varying environmental resource conditions. Computer systems have expanded the realm of testable ecological and social conditions, as well as the depth of data that can be collected from such experiments, i.e., chat data and real-time attention tracking. However, the methodological challenges in this experimental research do not stem primarily from a need for greater complexity. Instead, the biggest constraints for social-ecological experimentation are (1) the difficulty of capturing the complexity of social-ecological systems in tractable behavioral experiments and (2) the challenge of creating robust and reusable experimental software with limited time and funding.

We focus on laboratory experiments. There is also an increasing use of field experiments to study social-ecological systems. In field experiments, participants are recruited beyond the traditional student population, e.g., farmers and fishers, and these kinds of experiments have specific challenges and opportunities (Harrison and List 2004, Anderies et al. 2011). Social-ecological field experiments are typically deployed to study behavior in a particular social-ecological system, such as cooperative irrigation in a multicaste village (Waring and Bell 2013), forest use (Cardenas 2000), artisanal fisheries (Castillo et al. 2011), and pastoralism (Prediger et al. 2011). Field experiments on social-ecological systems often attempt to mimic the local social and ecological contexts in a stylized way to collect insights from local resource users on governance of common resources. Although we focus on laboratory experiments, software systems that are used for laboratory experiments can be the starting point for field experiments, and the two domains have increasingly similar technical needs.

The subsequent discussion of the platforms is aimed at scholars from diverse disciplines who study social-ecological systems and are interested in exploring options for their research and teaching. A good way to start is to try out existing experiments in class as an educational experience on collective action. When one desires to continue to use this for research, the best platforms to use depend on programming skills and specific research questions.

Experimental domains

Social-ecological experiments are used in a growing number of domains, including the laboratory, the field, the classroom, and online. Current tools allow experimenters to vary participants, i.e., numbers and types; production functions, i.e., net costs and benefits; environmental details, i.e., resource distribution, timing, and type; experimental framing, i.e., how the experiment is introduced; rules, i.e., rules of the game and payoff structure; positions, i.e., network position, asymmetric commons, group structure, and differential abilities; and information conditions, i.e., communication, information displayed, and resource visibility. However, different domains of use have different goals and entail particular logistical constraints, influencing the design of experiments and experimental platforms created for them. We provide archetypal descriptions of use domains that use experimental software to help clarify their influence on methodological choices.

Laboratory research experiments

Social-ecological experiments conducted in university laboratories are typically not focused on particular contextual factors but often strive to ask basic scientific questions and seek results that are generalizable across social and ecological contexts. Laboratory experiments are also not as constrained by particular contextual factors, or by logistical issues, and may be substantially more complex (Janssen et al. 2010, Kimbrough and Wilson 2013). In comparison to field experiments, laboratory experiments are often employed to examine more theoretical research questions, such as, “is social learning conformist in a variable environment?” (McElreath et al. 2005).

Open online experiments

Behavioral experiments are also conducted online, e.g., using Amazon’s distributed platform for computer-based labor, Mechanical Turk (Horton et al. 2011, Rand and Nowak 2011, Rand 2012). Open online experiments of this type partially address the criticism that traditional laboratory experiments often attempt to draw general conclusions about human behavior from an unrepresentative student population (Henrich et al. 2010). Open online experiments for social-ecological systems research promise to increase participation and cultural and ethnic diversity in experimental results, but they also come with significant challenges associated with identity and behavioral verification and validation. Experiments with Mechanical Turk show similar results as found in traditional experiments (Amir et al. 2012).

Educational experiments

Social-ecological simulations are increasingly used in education, where their form and function overlap significantly with research experiments. However, research experiments often entail additional social science measures such as surveys and follow-up questionnaires and are tightly controlled to maintain conditions such as anonymity to ensure high data quality. By contrast, in an educational context, an experimental game can be used in isolation but in combination with a lesson plan, thematic exposition, and a group debriefing. The use of experiments has been shown to increase comprehension of teaching material in economics (Cartwright and Stepanova 2012) and therefore might be explored as a teaching tool to teach governance of social-ecological systems.

Description of platforms

We describe the publicly available software platforms for performing behavioral experiments of relevance to social-ecological systems. The results are compiled in a summary table (Table 1). We provide for each software platform some illustrative screenshots to show the variety of graphical user interfaces employed.

Paper and pencil

The simplest experimental method is the “paper-and-pencil” option. Its simplicity makes the paper-and-pencil approach a natural alternative where software or technological options are not appropriate. Even some elaborate experiments can be conducted with paper and pencil (e.g., Cardenas et al. 2013). Paper-and-pencil experiments are often used for field experiments because of their logistical simplicity, physical reliability, and rapid implementation. With clear forms and protocols, a well-trained group of experimenters can run very successful experiments. In each round, experimenters collect data from the participants, perform the relevant calculations, and provide feedback to the participants.

Although a paper-and-pencil implementation can be cumbersome, the simplicity of this approach has one important benefit that technologically intensive solutions have yet to match. Importantly, paper-and-pencil experiments are easy to prototype, test, and modify. They effectively require zero investment in equipment or software development, and as a result, paper-and-pencil experiments can rapidly iterate in their design and quickly move to data collection. Paper-and-pencil experiments suffer two major drawbacks, however. First, they are limited in terms of the attainable environmental and social complexity for any given experiment. For example, including spatial and temporal dynamics in an experiment quickly produces practical challenges in implementation. Second, although calculations by experimenters can be done more efficiently with spreadsheets, data and forms must be processed manually, which introduces the potential for errors. As a consequence, there is a limited complexity that can be managed through paper-and-pencil experiments.

z-Tree

The Zurich Toolbox for Readymade Economic Experiments, or z-Tree (http://www.iew.uzh.ch/ztree), was developed in the 1990s by economist Urs Fischbacher (2007) and has become a standard tool among experimental economists. z-Tree is a client-server Windows application that comes with many standard economics experiments (http://www.iew.uzh.ch/ztree/treatments.php), including social dilemmas and auction markets. It also provides a programming environment in which new experiments that conform to its data and interaction model (http://www.iew.uzh.ch/ztree/ztree21tutorial.pdf) can be rapidly developed via a programmable interface. The software allows one to create and parameterize experimental treatments with support for custom programming logic and common parameters such as number of subjects, groups, group size, number of rounds, and show-up fees. z-Tree uses a round-based experimental structure to organize interactions as do many such software solutions. Although the latest version offers graphical capabilities, the software is not designed to provide engaging visualizations and interactions and typically relies on participants making text input decisions based on textual feedback (Fig. 2). The software is closed source and freely available and must be installed on participants’ computers or virtual machines running Microsoft Windows with a shared network drive.

GameWeb

GameWeb was developed in 2004 by anthropologist Richard McElreath (http://gameweb.sourceforge.net) and is an open-source, web-based platform implemented with PHP, MySQL, and JavaScript for round-based, multiperson behavioral experiments (Fig. 3; McElreath et al. 2005, 2008). GameWeb was designed to overcome some of the limitations of z-Tree. It features separate web consoles for participants and experimenters and can be integrated with cursor-tracking software (http://mouselabweb.org) to provide millisecond cursor-tracking data and reveal changes in participants’ attention and information access, i.e., what information participants have viewed and for how long. The use of standard languages also makes the modification of the GameWeb engine possible. GameWeb can be deployed on any computer device with a web browser and has been used on handheld devices in areas with no electricity or Internet connection (Waring and Bell 2013) and in laboratory settings (McElreath et al. 2008). There is no ready-made menu of experimental designs to select from, so experiments in GameWeb must be implemented by the research team by extending the GameWeb framework.

Bonn Experiment System

The Bonn Experiment System (BoXS) was developed by Mirko Seithe (2012) and is a Java-based software system designed to create and conduct experiments for laboratory environments as well as Internet experiments (Fig. 4; http://boxs.uni-bonn.de/). The experimental environment is relatively new and contains examples for public good experiments and Dutch auctions. Experimenters can create new experiments online using a Java applet experiment builder. The participant interfaces are written in HTML, and the examples are primarily text-based with minimal graphical user interface support.

Social-Ecological Systems Experimental Framework

The Social-Ecological Systems Experimental Framework (SESEF) was developed by Allen Lee in 2006 at the Center for the Study of Institutional Diversity and is an open-source, client-server framework implemented in Java (https://bitbucket.org/virtualcommons/sesef). It provides support for experiment configuration via XML or properties files, data persistence, networking, and graphics for group experiments. Two types of experiments have been built on SESEF, a foraging game and an irrigation game.

The foraging game began development in the early 2000s to investigate spatial foraging strategies (Goldstone and Ashpole 2004) and the influence of real-time changes in other participants’ foraging patterns (Goldstone et al. 2005). It evolved to examine how groups of participants interact with a renewable resource under different conditions (Janssen et al. 2008, 2010) and is freely available as open-source experimental software (http://bitbucket.org/virtualcommons/foraging). Generated data files can be exported to CSV for data analysis, and participants’ foraging patterns can be converted into QuickTime movies so experimenters can replay experiments.

A key difference between this software and a package like z-Tree is the way decisions are made. Instead of discrete rounds in which participants make independent decisions in each round with information from previous rounds, participants have a configurable period of a specific number of minutes in which they make real-time decisions on where to move in the spatially explicit renewable resource and when to collect resources (see Fig. 5). Participants will often click several times a second to maneuver their avatar and collect resources. A software platform designed for real-time continuous interactivity with graphics and network synchronization capabilities is needed to support this kind of dynamic interaction because the experiment’s game state must be updated and transmitted efficiently to each member of a group multiple times per second. Installation of the software requires Java to be installed on the experimenter and client computers. Experiment clients are installed and run from a web browser using Java Web Start.

The irrigation game (https://bitbucket.org/virtualcommons/irrigation) was developed using the same core engine as the foraging game described previously but with game dynamics addressing coordination and information asymmetry issues in irrigation systems (Anderies et al. 2013). The game itself consists of multiple rounds where participants make infrastructure investment decisions, have the option to communicate via text chat, and engage in real-time decision making, i.e., controlling an irrigation gate, during a specific amount of time (Fig. 6). The experimental environment also includes configurable parameters that introduce shocks to the water supply and infrastructure and can limit the information participants can see about each other.

VCWeb

The Virtual Commons Web Environment (VCWeb; http://vcweb.asu.edu) was developed in 2011 by Allen Lee and is an open-source, web-based framework implemented in Python, Django, and Bootstrap for developing and running collective action experiments (Fig. 7). There is no need to install specialized software beyond a modern web browser such as Chrome, Safari, or Firefox, and experiments can be performed in a controlled setting, e.g., computer lab, where the transition between rounds in the experiment is managed by an experiment facilitator or in a solely online setting where participants access the experiment via a desktop or mobile browser. It offers real-time functionality including text chat between groups or individuals, experiment configuration and parameterization, and the ability for experiment facilitators to synchronously advance all participants from one round to the next. The software originated from a desire to investigate issues of scale in collective action theory and to conduct experiments with large groups of participants (Janssen et al. 2013). Creation of new experiments requires experience with Python and web development.

ConG

ConG (http://leeps.ucsc.edu/cong) is a specialized experimental environment for continuous-time games for experimental economics developed in 2009 and implemented in Java (Pettit et al. 2012). Like GameWeb, experiment configuration is implemented with CSV files and offers real-time interactivity and visualization for several standard economics experiments including the prisoner’s dilemma, hawk dove, and a voluntary contribution mechanism public goods game. The existing visualizations are similar to real-time graphs and implemented in the PROCESSING language (Fig. 8; http://processing.org). They offer PDF documents describing the experiments, but there are no in-game instructional screens to explain the variables or interaction model. The environment can be extended with customized payoff functions and visualizations.

VeconLab

VeconLab (http://veconlab.econ.virginia.edu/admin.htm) was developed in 2005, offers 44 discrete-round, web-based economics experiments, and is implemented in PHP. Instructors or researchers can register for an account on their website and run experiments. A sample experiment begins with a set of instructions, a quiz to test for understanding, and then a number of repeated discrete rounds of numerical decision making similar to z-Tree or MobLab. The visual interface is primarily textual with on-screen instructions, input validation, and contextual descriptions of the experiments and their goals (Fig. 9). Customization of the software would require contacting the author because the software appears to be closed source and does not document any known extension points.

MobLab

MobLab was developed in 2012 and is a start-up founded in 2011 developing proprietary interactive economics games that can be run in the classroom using an HTML5-compatible web browser or on Android and iOS mobile devices (http://www.moblab.com/). They currently offer 22 standard games from experimental economics including public good, prisoner’s dilemma, and bargaining games with a focus on providing engaging user interfaces. Experimenters can adjust basic parameters of the experiments, i.e., payoff and number of participants, and decision making occurs in discrete rounds (Fig. 10). Although this experimental environment does not yet include more comprehensive games with ecological dynamics, it is a viable example of a tool for the classroom and is currently free for academic use.

NetLogo/HubNet

NetLogo (http://ccl.northwestern.edu/netlogo) is a software platform for developing agent-based models. NetLogo’s HubNet participatory simulation system merges built-in local networking abilities with an agent-based simulation engine commonly used in education and research. HubNet allows complex spatial simulations to be shared and allows users to manipulate them in real time. Because of the NetLogo foundation, participatory simulations can include much more complex and dynamic and spatial environments than other experimental systems we have mentioned (see, e.g., Frey and Goldstone 2013). HubNet uses NetLogo’s built-in user interface features, i.e., sliders, buttons, spatial display, plots, and charts, to convey a rich, but constrained participant experience (Fig. 11). HubNet uses NetLogo’s simplified agent-oriented programming language. HubNet is well documented, and experiments may be developed relatively quickly. Although NetLogo’s HubNet architecture is capable of real-time participatory simulations, it carries performance and programmatic constraints as compared to a pure Java framework like SESEF.

Comparing platforms

We evaluated all the computer-based platforms on a number of basic criteria for usability and performance, and we compared these with our extensive experience with pencil-and-paper experiments. Table 1 contains the results based on available documentation and expertise with the various platforms. As a group, we have a diverse but incomplete expertise with the platforms presented, and therefore, we may not fully represent the possibilities of all platforms. Most of our expertise is related to SESEF, VCWeb, z-Tree, and GameWeb. However, our comparison has revealed a few key features that distinguish experimental platforms and tend to be associated with different domains of use.

Perhaps the most important consideration is whether one needs ready-made experiments for quick deployment, as in an educational setting, or a system on which to design a customized experiment, as is often needed in research. For ready-made experiments, z-Tree, MobLab, and VeconLab are good options. If a customized experiment is needed, selecting an open-source framework such as ConG, BoXS, SESEF, GameWeb, or VCWeb is recommended because the source code for the framework is freely available and can be customized at will. Open-source software offers the greatest potential for extension and conveys the added benefits of advancing and contributing to a community good. However, most open-source research frameworks, including those the authors have developed, lack up-to-date and thorough documentation. To be successful, these open-source research platforms must learn from z-Tree and other successful open-source projects, where clear, cogent, and well-structured documentation is essential for fostering community adoption and contribution. Currently, research groups work independently and often reimplement slight variants of common problems: organizing participants into groups, managing experiment parameterizations and experiment data organized into multiple treatments, and providing useful scaffolding for developing server-side and client-side components tailored to the specific research questions under investigation. Closed-source platforms may offer extension points where custom logic can be placed, but these are dependent on the platform authors. With platforms like z-Tree, a large community has created and shared scripts of variations of experimental treatments within the programming logic supported by z-Tree and facilitated by its dedication to creating useful documentation and other user support mechanisms including active mailing lists and forums.

Another important consideration is whether the platform requires the installation of custom software on each computer running the experiment or whether the experiment can be accessed via a standard web browser. Web-based experiments offer a number of advantages over traditional client-server software: they are always up-to-date, can support larger group sizes, require no local software installation, and are easily designed to be robust to interruptions in network connectivity and other concurrency issues. Real-time user interactivity is now possible with the advent of web-sockets and HTML5-enabled web browsers such as Mozilla Firefox, Google Chrome, or Apple Safari, though the supporting technology is still emerging and in flux.

The software used to generate published experimental results should be archived and preserved as an important artifact in the research process because it is essential to the reproducibility of the experimental results. Open-source software is typically stored on a web-hosted version control system such as SourceForge (http://sourceforge.net), Bitbucket (https://bitbucket.org), or GitHub (https://github.com). All modern version control systems offer the ability to “tag” a software release, taking a snapshot of the exact state of the code base and assigning it a symbolic name. Proper usage of version control tagging and maintenance branches should be used when generating publishable experiment data using open-source software in active development. Care must also be taken to include versions of all dependent libraries. An added benefit of open-source software stored in version control systems is transparency because the commit history, a record of every change that occurred in the code base, is always available. Commits to version control systems are also typically e-mailed to an archived mailing list, adding another layer of auditability and redundancy. Closed-source experimental software by definition lacks this type of transparency and furthermore cannot be scrutinized for bugs that can manifest themselves in the generated data. If closed-source software is used, an archive of the specific version of the software and any associated parameterizations, scripts, or other customizations and dependencies should be preserved along with the data.

To create new experimental designs in any of the platforms, programming expertise is needed to implement additional experimental logic and the desired behavior. Table 1 lists the types of programming expertise required. Platforms like z-Tree and NetLogo are well-documented, have active communities providing knowledgeable support, and are more accessible to nonprogrammers. However, those platforms have a limited capacity for real-time interaction and graphical visualizations desirable for social-ecological experiments.

To conclude, we believe there is no ideal platform that is free, user-friendly, and makes it effortless to develop novel experiments to study governance of detailed, complex social-ecological systems. Instead, we identify three categories of experimental platforms. We group the available systems into categories based on how well each platform handles the concerns discussed previously.

Prototyping

Tools like paper-and-pencil games and NetLogo/HubNet are excellent ways to develop prototype experiments as well as actual experiments for research. If experimental designs are simple, these tools can also be used for actual experiments. The benefits will be the rapid development of the specific software without specialized programming experience. A drawback is the lack of customization for experimentation, meaning that running the experiments might be ad hoc using manual calculations and interventions.

Educational games

Two platforms, MobLab and VeconLab, are excellent resources for running experiments in classrooms. They have been used in many classes and have proved to be robust. However, because the games offered in these platforms are standard games, and there is no ability to create new games, the platforms are not useful for use in research that requires novel game designs.

Research platforms

Research platforms such as z-Tree, ConG, BoXS, SESEF, GameWeb, and VCWeb can be used to develop different types of specialized round-based experiments. They differ in the expertise needed and the kind of experiments for which they are used. z-Tree was the first platform of its kind and is common in experimental economics in general; ConG focuses on experiments with continuous, instead of discrete, decision making; GameWeb is mainly used for experiments on cultural learning; BoXS is a new web-based platform; SESEF focuses on controlled social-ecological system experiments in a lab; and VCWeb is used for online experiments that can also be conducted via mobile devices outside the lab. All these platforms can be used to develop new experiments if you know the right programming languages.

Discussion

The previous discussion on platforms provides an incomplete overview of the tools available. We focused on experimental platforms that can be used for the study of the governance of social-ecological systems and that are easily available. Some scholars have developed relevant platforms for their own research but have not made them generally available (Hey et al. 2009, Kimbrough and Wilson 2013).

For novices to experimental methods, good starting points are the use of paper-and-pencil games, as well as platforms like VeconLab, MobLab, and z-Tree. For those options, no specific programming skills are needed to make use of a library of experimental treatments. For more advanced research that includes specific social-ecological features, platforms like SESEF and VCWeb would be useful. SESEF offers the greatest possibility for dynamic environments and user interfaces but also requires the highest level of programming skills. As an alternative, NetLogo/HubNet would be very appropriate to use for experiments with complex dynamic environments. The main drawback of using NetLogo/HubNet is the limited control over how participants see the information.

Good starting points for novices do not require programming experience, but the platforms lack specific focus on social-ecological systems and have limited control over the information participants see during communication. Better quality software is available, though these options require extensive programming experience and, often, significant time investments. Experimental software is complex because it must maintain data integrity across concurrent access to experimental data, be robust to crashes and heterogeneous network conditions, and handle all the special case needs for running multiple parameterized treatments. Furthermore, implementing novel experiments on top of an existing framework often requires changes to the framework that were not in the original design and require deep knowledge to implement properly.

The development of robust platforms for more advanced experiments that involve spatial and temporal dynamics of social-ecological systems, including social-ecological networks, will require advanced programming expertise to implement. Software development has substantial investment costs and suffers from a very high failure rate. In many cases, the software developed while research funding is available is abandoned and falls into disuse after the funding cycle is over. We believe that the research community should be building on each other’s work in an open and collaborative manner instead of continually reinventing the wheel. We hope that the current trend of open-source development of such experimental platforms will make it more likely that such software platforms can continue to provide value and be extended and maintained by the community.

RESPONSES TO THIS ARTICLE

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.

ACKNOWLEDGMENTS

The authors acknowledge financial support from Arizona State University and from the National Science Foundation via grant numbers 0748632 and 1210856.

LITERATURE CITED

Amir, O., D. G. Rand, and Y. K. Gal. 2012. Economic games on the Internet: the effect of $1 stakes. PLoS ONE 7(2):e31461. http://dx.doi.org/10.1371/journal.pone.0031461

Anderies, J. M., M. A. Janssen, F. Bousquet, J.-C. Cardenas, D. Castillo, M.-C. Lopez, R. Tobias, B. Vollan, and A. Wutich. 2011. The challenge of understanding decisions in experimental studies of common pool resource governance. Ecological Economics 70(9):1571-1579. http://dx.doi.org/10.1016/j.ecolecon.2011.01.011

Anderies, J. M., M. A. Janssen, A. Lee, and H. Wasserman. 2013. Environmental variability and collective action: experimental insights from an irrigation game. Ecological Economics 93:166-176. http://dx.doi.org/10.1016/j.ecolecon.2013.04.010

Balliet, D. 2010. Communication and cooperation in social dilemmas: a meta-analytic review. Journal of Conflict Resolution 54(1):39-57. http://dx.doi.org/10.1177/0022002709352443

Camerer, C. F. 2003. Behavioral game theory. Princeton University Press, Princeton, New Jersey, USA.

Cardenas, J.-C. 2000. How do groups solve local commons dilemmas? Lessons from experimental economics in the field. Environment, Development and Sustainability 2(3-4):305-322. http://dx.doi.org/10.1023/A:1011422313042

Cardenas, J.-C., M. Janssen, and F. Bousquet. 2013. Dynamics of rules and resources: three new field experiments on water, forests and fisheries. Pages 319-345 in J. A. List and M. K. Price, editors. Handbook on experimental economics and the environment. Edward Elgar, Cheltenham, UK. http://dx.doi.org/10.4337/9781781009079.00020

Cartwright, E., and A. Stepanova. 2012. What do students learn from a classroom experiment: not much, unless they write a report on it. Journal of Economic Education 43(1):48-57. http://dx.doi.org/10.1080/00220485.2012.636710

Castillo, D., F. Bousquet, M. A. Janssen, K. Worrapimphong, and J. C. Cardenas. 2011. Context matters to explain field experiments: results from Colombian and Thai fishing villages. Ecological Economics 70(9):1609-1620. http://dx.doi.org/10.1016/j.ecolecon.2011.05.011

Dietz, T., E. Ostrom, and P. C. Stern. 2003. The struggle to govern the commons. Science 302(5652):1907-1912. http://dx.doi.org/10.1126/science.1091015

Fischbacher, U. 2007. z-Tree: Zurich toolbox for ready-made economic experiments. Experimental Economics 10(2):171-178. http://dx.doi.org/10.1007/s10683-006-9159-4

Frey, S., and Goldstone R. L. 2013. Cyclic game dynamics driven by iterated reasoning. PLoS ONE 8(2):e56416. http://dx.doi.org/10.1371/journal.pone.0056416

Friedman, D., and S. Sunder. 1994. Experimental methods: a primer for economists. Cambridge University Press, New York, New York, USA. http://dx.doi.org/10.1017/CBO9781139174176

García-Gallego, A., N. Georgantzís, R. Hernán-González, and P. Kujal. 2012. How do markets manage water resources? An experiment. Environmental and Resource Economics 53:1-23. http://dx.doi.org/10.1007/s10640-012-9545-7

Goldstone, R. L., and B. C. Ashpole. 2004. Human foraging behavior in a virtual environment. Psychonomic Bulletin & Review 11:508-514. http://dx.doi.org/10.3758/BF03196603

Goldstone, R. L., B. C. Ashpole, and M. E. Roberts. 2005. Knowledge of resources and competitors in human foraging. Psychonomic Bulletin & Review 12:81-87. http://dx.doi.org/10.3758/BF03196350

Hardin, G. 1968. The tragedy of the commons. Science 162:1243-1248. http://dx.doi.org/10.1126/science.162.3859.1243

Harrison, G. W., and J. A. List. 2004. Field experiments. Journal of Economic Literature 42(4):1009-1055. http://dx.doi.org/10.1257/0022051043004577

Henrich, J., S. J. Heine, and A. Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences 33(2-3):61-83. http://dx.doi.org/10.1017/S0140525X0999152X

Hey, J. D., T. Neugebauer, and A. Sadrieh. 2009. An experimental analysis of optimal renewable resource management: the fishery. Environmental and Resource Economics 44: 263-285. http://dx.doi.org/10.1007/s10640-009-9285-5

Horton, J. J., D. G. Rand, and R. J. Zeckhauser. 2011. The online laboratory: conducting experiments in a real labor market. Experimental Economics 14(3):399-425. http://dx.doi.org/10.1007/s10683-011-9273-9

Janssen, M. A., R. L. Goldstone, F. Menczer, and E. Ostrom. 2008. Effect of rule choice in dynamic interactive spatial commons. International Journal of the Commons 2(2):288-312.

Janssen, M. A., R. Holahan, A. Lee, and E. Ostrom 2010. Lab experiments for the study of social-ecological systems. Science 328:613-617. http://dx.doi.org/10.1126/science.1183532

Janssen, M. A., A. Lee, and H. Sundaram. 2013. Investing time in the public good: a web-based threshold public good experiment. Center for Studies of Institutional Diversity (CSID) Working Paper Series, #CSID-2013-011. CSID, Arizona State University, Tempe, Arizona, USA. [online] URL: https://csid.asu.edu/sites/csid.asu.edu/files/csid_wp_2013-011.pdf

Kimbrough, E. O., and B. J. Wilson. 2013. Insiders, outsiders, and the adaptability of informal rules to ecological shocks. Ecological Economics 90:29-40. http://dx.doi.org/10.1016/j.ecolecon.2013.02.008

McAllister, R. R. J., J. G. Tisdell, A. F. Reeson, and I. J. Gordon. 2011. Economic behavior in the face of resource variability and uncertainty. Ecology and Society 16(3): 6. http://dx.doi.org/10.5751/ES-04075-160306

McElreath, R., A. V. Bell, C. Efferson, M. Lubell, P. J. Richerson, and T. Waring. 2008. Beyond existence and aiming outside the laboratory: estimating frequency-dependent and pay-off-biased social learning strategies. Philosophical Transactions of the Royal Society B 363:3515-3528. http://dx.doi.org/10.1098/rstb.2008.0131

McElreath, R., M. Lubell, P. J. Richerson, T. M. Waring, W. Baum, E. Edsten, C. Efferson, and B. Paciotti. 2005. Applying evolutionary models to the laboratory study of social learning. Evolution & Human Behavior 26(6):483-508. http://dx.doi.org/10.1016/j.evolhumbehav.2005.04.003

Ostrom, E. 1990. Governing the commons: the evolution of institutions for collective action. Cambridge University Press, Cambridge, UK. http://dx.doi.org/10.1017/CBO9780511807763

Ostrom, E. 2005. Understanding institutional diversity. Princeton University Press, Princeton, New Jersey, USA.

Ostrom, E. 2009. A general framework for analyzing the sustainability of social-ecological systems. Science 325(5939):419-422. http://dx.doi.org/10.1126/science.1172133

Ostrom, E., R. Gardner, and J. Walker. 1994. Rules, games, and common-pool resources. University of Michigan Press, Ann Arbor, Michigan, USA.

Ostrom, E., J. Walker, and R. Gardner. 1992. Covenants with and without a sword: self-governance is possible. American Political Science Review 86(2):404-417.

Pettit, J., D. Friedman, C. Kephart, and R. Oprea. 2012. Continuous game experiments. Economics Department, University of California, Santa Barbara, Santa Barbara, California, USA.

Poteete, A. M., M. A. Janssen, and E. Ostrom. 2010. Working together: collective action, the commons and multiple methods in practice. Princeton University Press, Princeton, New Jersey, USA.

Prediger, S., B. Vollan, and M. Frölich. 2011. The impact of culture and ecology on cooperation in a common-pool resource experiment. Ecological Economics 70(9):1599-1608. http://dx.doi.org/10.1016/j.ecolecon.2010.08.017

Rand, D. G. 2012. The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology 299:172-179. http://dx.doi.org/10.1016/j.jtbi.2011.03.004

Rand, D. G., and M. A. Nowak. 2011. The evolution of antisocial punishment in optional public goods games. Nature Communications 2:434. http://dx.doi.org/10.1038/ncomms1442

Roth, A. E., and J. Kagel, editors. 1995. Handbook of experimental economics. Princeton University Press, Princeton, New Jersey, USA.

Seithe, M. 2012. Introducing the Bonn Experiment System (BoXS). Bonn Econ Discussion Papers, No. 01/2012. University of Bonn, Bonn, Germany.

Waring, T. M., and A. V. Bell. 2013. Ethnic dominance damages cooperation more than ethnic diversity: results from multi-ethnic field experiments in India. Evolution & Human Behavior 34(6):398-404. http://dx.doi.org/10.1016/j.evolhumbehav.2013.07.003

Address of Correspondent:
Marco A. Janssen
School of Human Evolution and Social Change
Arizona State University
Tempe, AZ 85287-2402
Marco.Janssen@asu.edu
Jump to top
Table1  | Figure1  | Figure2  | Figure3  | Figure4  | Figure5  | Figure6  | Figure7  | Figure8  | Figure9  | Figure10  | Figure11