Experimental platforms for behavioral experiments on social-ecological systems

Recently, there has been an increased interest in using behavioral experiments to study hypotheses on the governance of social-ecological systems. A diversity of software tools are used to implement such experiments. We evaluated various publicly available platforms that could be used in research and education on the governance of social-ecological systems. The aims of the various platforms are distinct, and this is noticeable in the differences in their user-friendliness and their adaptability to novel research questions. The more easily accessible platforms are useful for prototyping experiments and for educational purposes to illustrate theoretical concepts. To advance novel research aims, more elaborate programming experience is required to either implement an experiment from scratch or adjust existing experimental software. There is no ideal platform best suited for all possible use cases, but we have provided a menu of options and their associated trade-offs.


INTRODUCTION
The quest for sustainability is fraught with social dilemmas regarding the use of shared resources. Social dilemmas occur when individual incentives conflict with collective interests. This is especially apparent for shared resources such as forests, fish, clean and fresh water, and fertile land. Garrett Hardin (1968) argued that resource users cannot govern their own resources and that a tragedy of the commons is unavoidable without the establishment of state control or privatization of the shared resource. To the contrary, Elinor Ostrom and her colleagues showed that such tragedy is avoidable (Ostrom 1990, Dietz et al. 2003, by gathering many empirical examples of common resources that are successfully managed by local communities who create innovative institutional arrangements. An important contribution to the success of the study of governance of common pool resources is the theoretical framework called the Institutional Analysis and Development framework, which is designed to help understand the complexity of empirical cases of institutional design (Ostrom 2005). A recent extension of this work is the emphasis of the social and ecological contextual variables in the Social-Ecological Systems framework (Ostrom, 2009).
The study of governance of social-ecological systems cannot be performed by one methodological approach, but requires investigation using different approaches such as case study analysis, formal modeling, and behavioral experiments (Poteete et al. 2010). This paper focuses on experimental methods, but taking into account the findings derived by other methods.
Behavioral experiments in the laboratory and field are increasingly used to test hypotheses emerging from theory and case study research (Ostrom et al. 1994, Poteete et al. 2010. Experiments have been performed with university student populations in various countries, as well as in diverse other populations including farmers in rural India or fishers at the coast of Colombia (Anderies et al. 2011, Waring andBell 2013). These experiments show that overharvesting of common resources happens -as theory predicts -if participants cannot communicate and cannot use costly punishment. In contrast to theory, and in line with case study research, experiments show that communication and costly punishment reduce the overharvesting of the common resources (Poteete et al. 2010). Communication without the ability to enforce promises, called cheap talk, is not expected to have an effect to rational selfish actors, yet leads to a significant increase in group cooperation (Balliet 2010). The option to reduce the earnings of others at a cost to oneself (i.e., costly punishment) is not expected to be used. However, both cheap talk and costly punishment are found to have a significant effect and confirm observations in case study analysis (Ostrom et al. 1992).
1 Arizona State University 2 University of Maine 3 Stockholm University A limitation of early public goods and common pool resource experiments for the use of studying governance of social-ecological systems was the lack of relevant ecological complexity. Scholars have begun to explore the consequences of more specific and relevant ecological complexity in the study of social dilemmas in social-ecological systems , McAllister et al. 2011, Prediger et al. 2011, García-Gallego et al. 2012, Kimbrough and Wilson 2013, Cardenas et al. 2013, Anderies et al. 2013. A result of increased scientific interest in social-ecological experiments has been the parallel proliferation of experiment software to implement them. Experiment software platforms have different aims, code bases, features and user communities. This diversity in software platforms is valuable but also creates difficult choices for researchers and presents challenges for standardization and interoperability.
One of the purposes of this paper is to address common inquiries by colleagues and students on how to start conducting social-ecological behavioral experiments. Beyond understanding the methodologies for designing and running experiments (Friedman and Sunder 1994, Roth and Kagel 1995, Camerer 2003, it is important to know what kinds of software tools are available and appropriate. Like most software applications, software experiments are typically developed for a specific set of research questions and assumptions and advanced programming experience is needed for major adjustments. However, most frameworks provide extension points that enable experimenters to modify features of an experiment via configuration parameters or even custom programming logic. Some experiments can also be implemented via off-the-shelf tools such as NetLogo, Excel, or Google Documents. In this article we review the various software tools and platforms available for conducting behavioral social-ecological experiments with an eye towards clarifying the options. We evaluate these platforms on their goals, design, abilities and usage.

EXPERIMENTAL APPROACHES TO GOVERNANCE OF SOCIAL-ECOLOGICAL SYSTEMS
In this section we discuss some common variations of standard experiments. This will provide a perspective on the scope of possible experiments and the justification for certain choices of platforms. All relevant experiments are designed to test specific hypotheses. In order to do so we will have to define different treatments. For example, to test the effect of communication we will need a number of groups with and without communication allowed. Sufficient groups need to be performed such that non-parametric tests can be used to test significance of the differences due to the treatment effect.
Experiments are often simplistic and narrow in scope by design in order to focus on specific research questions and exclude noise from the complex real world. Results from experiments by themselves do not prove the effect of an intervention, and are part of a larger body of experimental results and findings from other studies that accumulatively lead to improved insights (Poteete et al. 2010).
In a standard linear public good experiment a group of N participants receives an endowment (w) every round. In each round participants decide independently how much (X) of the endowment to invest in a public fund and how much to invest in a private fund. The total investment in the public fund is multiplied by (m) and shared equally among the participants. A participant's earnings in a given round are the remainder of the private fund (w-X) plus an even share of the public fund (m/N)*sum X. A selfish rational actor will not invest anything in the public fund, and the earnings will be equal to the initial endowment. The social optimum solution is obtained, by contrast, if every participant invests their entire endowment in the public good, increasing the earnings by m.
Common pool resource experiments are similar to public good experiments except that the decision to be made is how much to extract from a common pool resource in contrast with how much to invest in a private fund. A selfish rational actor will harvest as much as possible and will expect others to do the same. The social optimum resource consumption is a lower consumption, which provides social benefits such as resource longevity, or greater harvest per unit effort. In both public goods and common pool resource scenarios, N participants make numerical investment or harvesting decisions in parallel, every round.
We use the Institutional Analysis and Development framework (Ostrom 2005) to discuss what kinds of variations of so-called action situations we can explore in experiments ( Figure 1). Participants can fill positions that allow certain actions. Individuals have control over the actions that lead to outcomes and have information about action, outcomes and their linkages as well as the costs and benefits. We can vary who participates (e.g., students, farmers), in which kind of environment (computer-based, paper and pencil, mobile apps) and with which "culture" (rural/urban populations, hunter-gatherers, small-scale rice farmers in Nepal). In the experiments we can vary the production functions (net cost and benefits), which may reflect different types of ecologies (how do actions of individuals affect the payoff of other participants). The ecological context can also affect the positions participants have (e.g., connected in networks or via asymmetric interactions), when decisions can be made, and who has information about the actions and earnings of others. The variations can be endless yet are guided by observations in case studies and a theoretical framework that capture the interactions of people. Modern computer software and programming languages now make it possible for experimenters to create complex, interactive, multi-participant experiments with highly specific social roles and varying environmental resource conditions. Computer systems have expanded the realm of testable ecological and social conditions, as well as the depth of data that can be collected from such experiments (i.e., chat data, real time attention tracking). However, the methodological challenges in this experimental research do not stem primarily from a need for greater complexity. Instead, the biggest constraints for social-ecological experimentation are (a) the difficulty of capturing the complexity of social-ecological systems in tractable behavioral experiments, and (b) the challenge of creating robust and reusable experiment software that is flexible enough to allow incremental and iterative modification until the experiment design best addresses the research question(s) at hand. In addition, social-ecological experiments and simulations are increasingly used in a variety of domains and modes (i.e., educational, laboratory, online), across which there are divergences of practice.
In this paper we focus on laboratory experiments, and will not discuss in-depth the challenges and opportunities for field and natural experiments (Harrison andList 2004, Anderies et al. 2011). The discussion of the platforms below is aimed at scholars interested in exploring options for their research and teaching. A good way to start is to try out existing experiments in class as an educational experience on collective action.

Experimental domains
Social-ecological experiments are used in a growing number of domains, including the laboratory, the field, the classroom, and online. Current tools allow experiments to be designed in a large variety of combinations, varying participants (numbers, types), production functions (net costs and benefits), environmental details (resource distribution, timing, type), experimental framing, rules, positions (network position, asymmetric commons, group structure, differential abilities), and information conditions (communication, information displayed, resource visibility). However, different domains of use have different goals and entail particular logistical constraints, influencing the design of experiments and experimental platforms created for them. For instance field experiments are often tailored to fit real-world scenarios (with resource details, societal roles and rules, and information conditions matching a specific user group and resource) while laboratory experiments are often designed to simulate stylized social-ecological systems for greatest generalizability . We provide archetypal descriptions of these use-domains to help clarify their influence on methodological choices.

Field experiments
Social-ecological field experiments are typically deployed to study behavior in a particular social-ecological system, such as cooperative irrigation in a multi-caste village (Waring and Bell 2013), forest use (Cardenas 2000), artesian fisheries , and pastoralism (Prediger et al. 2011). Field experiments tend to be focused on determining the behavioral responses of local people to a local environment, and are not usually intended to be directly generalizable to all human groups. As such, field experiments often attempt to mimic the local social and ecological contexts in a stylized way.

Laboratory experiments
Social-ecological experiments conducted in university laboratories are typically not focused on particular contextual factors, but often strive to ask basic scientific questions and seek results that are generalizable across social and ecological contexts. Laboratory experiments are also not as constrained by particular contextual factors, or by logistical issues, and may be substantially more complex (Janssen et al. 2010, Kimbrough andWilson 2013). In comparison to field experiments, laboratory experiments are often employed to examine more theoretical research questions, such as "does conformity influence resource conservation in a variable environment?" (McElreath et al. 2005).

Open online experiments
Behavioral experiments are also conducted on open networks across large spans of space and time (Rand 2012, Horton et al. 2011, Rand and Nowak 2011. Open online experiments of this type partially address the criticism that traditional laboratory experiments often attempt to draw general conclusions about human behavior from an unrepresentative student population (Henrich et al. 2010). Open online experiments for social-ecological systems research promises to increase participation and cultural and ethnic diversity in experimental results, but also comes with significant challenges associated with identity and behavioral verification and validation.

Educational experiments
Social-ecological simulations are increasingly used in education, where their form and function overlap significantly with research experiments. However, research experiments often entail additional social science measures such as surveys and follow-up questionnaires and are tightly controlled to maintain conditions such as anonymity in order to ensure high data quality. By contrast, in an educational context an experimental game can be used in isolation but completed by a lesson plan, thematic exposition and a group debriefing. The use of experiments has shown to increase comprehension of teaching material in economics (Cartwright and Stepanova 2012) and therefore might be explored as a teaching tool to teach governance of social-ecological systems.

Description of platforms
In this section we describe the publicly available software platforms available for performing behavioral experiments of relevance to social-ecological systems. The results are compiled in a summary table (Table 1).

Paper and pencil
The simplest experimental method is the "paper and pencil" option. Its simplicity makes the paper and pencil approach a natural alternative where software or technological options are not appropriate. Elaborate experiments can even be performed with paper and pencil only (e.g., Cardenas et al. 2013). Paper and pencil experiments are often used for field experiments due to their logistical simplicity, physical reliability and rapid implementation. With clear forms and protocols, a well-trained group of experimenters can run very successful experiments. In each round experimenters collect data from the participants, perform the relevant calculations and provide feedback to the participants.
While the implementation of a paper and pencil experiment can be cumbersome, it has a number of important benefits, which more technologically intensive solutions have yet to match. First, paper and pencil experiments are easy to prototype, test and modify. They require effectively zero investment in equipment or software development, and as a result paper and pencil experiments can rapidly iterate in their design and quickly move to data collection. Paper and pencil experiments suffer two major drawbacks, however. First, they are limited in terms of the attainable environmental and social complexity for any given experiment. Secondly, although calculations by experimenters can be done more efficiently with spreadsheets, data and forms must be processed manually which introduces the potential for errors. As a consequence there is a limited complexity that can be managed through paper and pencil experiments.

Z-Tree
The Zurich Toolbox for Readymade Economic Experiments, or z-Tree (http://www.iew.uzh.ch/ztree) has become a standard tool among experimental economists (Fischbacher, 2007). Z-Tree is a client-server Windows application that comes with many standard economics experiments (http://www.iew.uzh.ch/ztree/treatments.php) including social dilemmas and auction markets. It also provides a programming environment in which new experiments that conform to its data and interaction model (http://www.iew.uzh.ch/ztree/ztree21tutorial.pdf) can be rapidly developed via a wizard-driven interface. The software allows one to create and parameterize experimental treatments with support for custom programming logic and common parameters such as number of subjects, groups, group size, number of rounds, and show-up fees. Although the latest version offers graphical capabilities, the software is not designed to provide engaging visualizations and interactions and typically relies on participants making text input decisions based on textual feedback. Z-Tree uses a round-based experimental structure to organize interactions as do many such software solutions. The software is closed source, freely available and must be installed on computers or virtual machines running Microsoft Windows with a shared network drive. GameWeb was designed to overcome some of the limitations of z-Tree. It features separate web-consoles for participants and experimenters and can be integrated with cursor tracking software (http://mouselabweb.org) to provide millisecond cursor tracking data and reveal changes in participants' attention and information access (i.e., what information participants have viewed, and for how long). The use of standard languages also makes the modification of the GameWeb engine possible. GameWeb can be deployed on any computer device with a web browser and has been used on handheld devices in areas with no electricity or internet connection (Waring and Bell 2013) and in laboratory settings (McElreath et al. 2008). There is no readymade menu of experimental designs to select from, so experiments in GameWeb must be implemented by the research team by extending the GameWeb framework.

CSID experiment framework
The CSID experiment framework is an open-source, client-server framework implemented in Java and developed by the Center for the Study of Institutional Diversity (CSID) (https://bitbucket.org/virtualcommons/csidex). It provides support for experiment configuration via XML or properties files, data persistence, networking, and graphics for group experiments.
Two types of experiments have been built on the CSID framework, a foraging game and an irrigation game.
The foraging game began development in the early 2000s to investigate spatial foraging strategies (Goldstone and Ashpole 2004) and the influence of real-time changes in other participants' foraging patterns (Goldstone et al. 2005). It evolved to examine how groups of participants interact with a renewable resource under different conditions (Janssen et al. 2008 and is freely available as an open source (http://commons.asu.edu/software/foraging). Generated data files can be exported to CSV for data analysis and converted into QuickTime movies so experimenters can replay experiments.
A key difference between this software and a package like z-Tree is the way decisions are made. Instead of discrete rounds where participants make independent decisions in each round with information from previous rounds, participants have a period of four minutes in which they make real-time decisions on where to move in the spatially explicit renewable resource and when to collect resources (see Figure 2). Participants will often click several times a second to maneuver their avatar and collect resources. A software platform designed for real-time continuous interactivity with graphics and network synchronization capabilities is needed to support this kind of dynamic interaction as the experiment's game state must be updated and transmitted efficiently to each member of a group multiple times per second. Installation of the software requires Java to be installed on the experimenter and client computers. Experiment clients are installed and run from a web browser using Java Web Start. The irrigation game (https://bitbucket.org/virtualcommons/irrigation) was developed using the same core engine as the foraging game described above but with game dynamics addressing coordination and information asymmetry issues in irrigation systems (Anderies et al. 2013). The game itself consists of multiple rounds where participants make infrastructure investment decisions, have the option to communicate via text chat, and engage in real time decision making (controlling an irrigation gate) during a specific amount of time. The experimental environment also includes configurable parameters that introduce shocks to the water supply and infrastructure and can limit the information participants can see about each other.

VCweb
The Virtual Commons Web Environment (http://vcweb.asu.edu) is an open-source, web-based framework implemented in Python, Django, and Bootstrap for developing and running collective action experiments. There is no need to install specialized software beyond a modern web browser such as Chrome, Safari, or Firefox, and experiments can be performed in a controlled setting (e.g., computer lab) where the transition between rounds in the experiment is managed by an experiment facilitator or in a solely online setting where participants access the experiment via a desktop or smartphone browser. It offers real-time functionality including text chat between groups or individuals, experiment configuration and parameterization, and the ability for experiment facilitators to synchronously advance all participants from one round to the next. The software originated from a desire to investigate issues of scale in collective action theory and to conduct experiments with large groups of participants . Creation of new experiments requires experience with Python and web development.

MobLab
MobLab is a startup developing proprietary interactive economics games that can be run in the class room using an HTML5 compatible web browser or on Android and iOS mobile devices (http://www.moblab.com/). They currently offer twenty-two standard games from experimental economics including public good, prisoner's dilemma, and bargaining games. Experimenters can adjust basic parameters of the experiments (payoff, number of participants) and decision making occurs in discrete rounds. Although this experimental environment does not yet include more comprehensive games with ecological dynamics, it is a viable example of a tool for the classroom.

ConG
ConG (http://leeps.ucsc.edu/cong) is a specialized experimental environment for continuous-time games for experimental economics implemented in Java (Pettit et al. 2012). Like GameWeb, experiment configuration is implemented with CSV files and offers real-time interactivity and visualization for several standard economics experiments including the prisoner's dilemma, hawk dove, and a voluntary contribution mechanism public goods game. The existing visualizations are similar to real-time graphs and implemented in the processing language (http://processing.org). They offer PDF documents describing the experiments but there are no in-game instructional screens to explain the variables or interaction model. The environment can be extended with customized payoff functions and visualizations.

VeconLab
VeconLab (http://veconlab.econ.virginia.edu/admin.htm) offers 44 discrete round web-based economics experiments and is implemented in PHP. Instructors or researchers can register for an account on their website and run experiments. A sample experiment begins with a set of instructions, a quiz to test for understanding, and then a number of repeated discrete rounds of numerical decision making similar to z-Tree or MobLab. The visual interface is primarily textual with on-screen instructions, input validation, and contextual descriptions of the experiments and their goals. Customization of the software would require contacting the author as the software appears to be closed source and does not document any known extension points.

NetLogo:HubNet
NetLogo (http://ccl.northwestern.edu/netlogo) is a software platform for developing agent-based models. NetLogo's HubNet participatory simulation system merges built-in local networking abilities with an agent-based simulation engine commonly used in education and research. HubNet allows complex spatial simulations to be shared and allows users to manipulate them in real-time. Because of the NetLogo foundation, participatory simulations can include much more complex and dynamic and spatial environments than other experiment systems mentioned here (see, for example, Frey and Goldstone 2013). HubNet uses NetLogo's built in user interface features (sliders, buttons, spatial display, plots and charts) to convey a rich, but constrained participant experience. HubNet uses NetLogo's simplified language agent-oriented programming language. HubNet is well documented, and experiments may be developed relatively quickly. Although NetLogo's HubNet architecture is capable of real-time participatory simulations it carries performance and programmatic constraints as compared to a pure Java framework.

Collaborative documents
Currently available online platforms for collaborative document editing can be harnessed for prototyping of simple experimental games. In these types of platforms files are stored in a file hosting service and accessed by users through a web-browser connected to the Internet. Experiments are built using one or several interconnected document files. Google Drive (http://drive.google.com/) is an example of an online file hosting and synchronization service that allows multiple users to access and edit documents in real-time. A game design can be implemented as online editable documents and made available to experiment participants on a computer with Internet access. An experiment built in this type of platform enrolls the real-time updating capability to allow players to take decisions by editing individual documents. Through mechanisms of document interlinkage, these decisions are reported to an experimenter and are used to calculate earnings and, depending on the game design, to produce the new state of the system. We are not aware of published experiments using this method of executing experiments.

Comparing platforms
We evaluated all the computer based platforms on a number of basic criteria for usability and performance, and compared these with our extensive experience with pencil and paper experiments. Table 1 contains the results based on available documentation and expertise with the various platforms. As a group we have a diverse but incomplete expertise with the platforms presented here and therefore we may not fully represent the possibilities of all platforms. Most of our expertise is related to the CSID framework, VCweb, z-Tree, GameWeb and collaborative documents. However, our comparison has revealed a few key features that distinguish experimental platforms and tend to be associated with different domains of use.
Perhaps the most important consideration is whether one needs readymade experiments for quick deployment (as in an educational setting), or a system on which to design a customized experiment (as is often needed in research). If a customized experiment is needed, selecting an open source framework is recommended as the source code for the framework is freely available and can be customized at will. Closed source platforms may offer extension points where custom logic can be placed, but these are dependent on the platform authors.
Another important consideration is whether the platform requires the installation of custom software on each computer running the experiment or if the experiment can be accessed via a standard web browser. Web-based experiments offer a number of advantages over traditional client server software -they are always up-to-date, can support larger group sizes, require no custom software installation, and are easily designed to be robust to interruptions in network connectivity and other concurrency issues. Real-time user interactivity is now possible with the advent of HTML5-enabled web browsers such as Mozilla Firefox, Google Chrome, or Apple Safari though the technology is still emerging and in flux.
MobLab, Veconlab, and z-Tree have many standard economics experiments in their library. However, only z-Tree provides extensibility in experiment design as both MobLab and Veconlab are closed source.
Open source research platforms such as ConG, the CSID experiment framework, GameWeb, and VCweb offer the greatest potential for extension if programming expertise is available. Furthermore, we believe that the software used to generate published experimental results should be archived and preserved as an important artifact in the research process as it is essential to the reproducibility of the experimental results. All software carries the potential for error, and closed source experimental software cannot be scrutinized for potential errors that may or may not manifest themselves in the generated data.
In order to create new experimental designs in any of the platforms, programming expertise is needed to implement additional experiment logic and the desired behavior. Table 1 lists the types of programming expertise required. Platforms like z-Tree, Google Docs and NetLogo are welldocumented, have active communities providing knowledgeable support, and are more accessible to non-programmers but are also more limited in the range of possible experimental designs, e.g., real-time interaction and graphical visualizations.
To conclude, there is no ideal platform that is free, user-friendly, and makes it effortless to develop novel experiments to study governance of detailed, complex social-ecological systems. Instead, we identify four categories of experimental platforms.

Proto-typing
Tools like paper and pencil games, Collaborative Documents, and Netlogo HubNet are excellent ways to develop prototype experiments. If experimental designs are simple, those tools can also be used for actual experiments. The benefits will be the rapid development of the specific software without specialized programming experience. A drawback is the lack of customization for experimentation, meaning that running the experiments might be ad hoc using manual calculations and interventions.

Educational games
Two platforms, Moblab and Veconlab, are excellent resources to use experiments in classrooms.
They have been used in many classes and have proven to be robust. However, since the games offered in these platforms are standard games, and there is no ability to create new games, the platforms are not useful for use in research that requires novel game designs.

Accessible platforms
Research platforms such as z-Tree, ConG, GameWeb, and VCWeb can be used to develop different types of specialized round based experiments. They differ in the expertise needed and the kind of experiments for which they are used as well. Z-Tree was the first platform of its kind and is common among experimental economics in general, ConG focuses on experiments with continuous (instead of discrete) decision making, GameWeb is mainly used for experiments on cultural learning, and VCWeb is used for experiments conducted via mobile devices outside the lab. All platforms can be used to develop new experiments if you know the right programming languages.

Specialized platforms
The foraging game and irrigation experiments are specialized platforms targeting specific types of social-ecological systems. Although many variations of the existing set of experiments performed with these platforms are possible, anyone wishing to use it must have some technical expertise to install, configure, and deploy the experiments, as well as advanced expertise in java programming to develop new variations.

Discussion
The discussion on platforms above provides an (incomplete) overview of tools available. We focused on experimental platforms that can be used for the study of governance of socialecological systems and that are easily available. Some scholars have developed relevant platforms for their own research but have not made them generally available (Hey et al. 2009, Kimbrough andWilson 2013).
The experimental economics platforms (Veconlab, Moblab, ConG) have limited options to include ecological complexity within the experimental design. In fact, the prototyping tools are excellent starting points in developing experiments with ecological complexity. Despite being constrained with the scheduling of rounds in a smooth way, one can even implement paper and pencil games with ecological dynamics (Cardenas et al. 2013). Similar experiments can be implemented in the Collaborative Documents, which might have the advantage of a smoother process of running the experiments without manual calculations of the progress in the experiment. Netlogo/Hubnet would be very appropriate to use for experiments with complex dynamic environments. One can use agent-based models and other dynamic systems in which participants can affect decisions via one of the agents in the computational environment. The main drawback of using Hubnet/Netlogo is the limited control on how participants see the information.
Prototype tools are a good way to begin exploring possibilities, but they lack a smooth progression and have limited control over the information participants see during communication and often require manual adjustments. Better quality software is available, but this also requires extensive programming experience and often significant time investments. Experiment software is complex as it must maintain data integrity across concurrent access of experiment data, be robust to crashes and heterogeneous network conditions, and handle all the special case needs for running multiple parameterized treatments. Furthermore, implementing novel experiments on top of an existing framework often requires changes to the framework that were not in the original design and require deep knowledge to implement properly.
The development of robust platforms for more advanced experiments that involve spatial and temporal dynamics of social-ecological systems, including social-ecological networks, will require advanced programming expertise to implement. We hope that the current trend of open source development of such experimental platforms will make it more likely that such software platforms can continue to provide value and be extended and maintained by the community.