Lessons learned from synthetic research projects based on the Ostrom Workshop frameworks

. A generalized knowledge of social-ecological relationships is needed to address current environmental challenges. Broadly comparative and synthetic research is a key method for establishing this type of knowledge. To date, however, most work on social-ecological systems has applied idiosyncratic methods to specific systems. Several projects, each based on the frameworks developed by Elinor Ostrom and colleagues, stand out for their application of consistent methods across a broad range of cases. In this paper we compare seven of these projects and draw conclusions regarding their potential benefits and the challenges that scholars can expect in conducting this type of research. The two main challenges that we identified are (1) the collective-action dilemmas that collaborators face in producing and maintaining the social and technical infrastructure that is needed for such projects; and (2) balancing complexity and comparability in the structure of the databases used and the associated methods for characterizing complex social-ecological cases. We discuss approaches for meeting these challenges, and present a guiding checklist of questions for project design and implementation to provide guidance for future broadly comparative research.


INTRODUCTION
Addressing sustainability challenges presented by current social and environmental change requires governance of environmental commons built on a clear understanding of social-ecological system (SES) relationships. A large body of literature on SES governance has identified a number of influential social and ecological attributes and the relationships among them (Partelow 2018). This literature has largely consisted of case studies using inconsistent measures for SES attributes, which has hindered comparative analyses and thus, generalizations about SES relationships.
Developing comparable case descriptions requires the use of a common framing to ensure that standard research protocols are used. In support of this common framing, Elinor Ostrom and colleagues at the Workshop in Political Theory and Policy Analysis at Indiana University developed several analytical frameworks. These "workshop frameworks" are the Institutional Analysis and Development (IAD) Framework (Ostrom 2005), the Social-Ecological Systems (SES) Framework (Ostrom 2007), and the Robustness Framework (Anderies et al. 2004). These frameworks are designed to enable analyses of systems in which human actors, frequently resource users, interact with each other and their social and biophysical environments. The IAD framework centers around the idea of an action situation, or a situation in which actors make decisions that jointly affect each other's welfare. How decision makers interact depends in turn on biophysical, social, and institutional conditions. The SES framework directly expands on the IAD, although it incorporates the language of variable tiers and more extensively unpacks the social and ecological variables that affect an action situation. The Robustness Framework similarly considers social-ecological systems and the resource users within them that interact to use natural resources; but it focuses less on focal action situations and more on adaptability as an outcome and the public infrastructure that affects resource user interactions.
Although these frameworks provide a common perspective for scholars, the literature on commons governance that employs them has mostly remained idiosyncratic in terms of the choice, definitions, and measures of variables (Thiel et al. 2015, Partelow 2018, Schlager and Cox 2018. This, together with a tendency to employ a case study approach (Poteete and Ostrom 2008), hinders synthetic and comparative research. Nevertheless, there have been several collaborative, data-driven projects based on the workshop frameworks that have stood out in utilizing them to produce consistent descriptions of a set of commons-oriented cases. In this paper, we summarize these projects (Table 1) and the lessons learned from them. Strengths and weaknesses are discussed in order to derive guidance for future broadly comparative socialecological research.

COMPARATIVE AND COLLABORATIVE SOCIAL-ECOLOGICAL RESEARCH
In this paper we focus on broadly comparative research projects that use the workshop frameworks to make comparisons among a relatively large number of commons-oriented cases. What we Table 1. Summary of the projects arising from the workshop frameworks. File refers to the data format used to store the data (RDB = relational database, Flat = spreadsheet). Data collection refers to whether the data is collected in the field vs. collected from published sources. The databases are based on different frameworks, although because of commonalities among the workshop frameworks, this does not lead to as much divergence in data structures as one might expect. 125 Primary Fisheries Individual to regional SES https://www.wcs.org/ mean by "broadly" here is domain specific. Much of the truly large-n analysis found in social science research occurs at the individual level or the national level (Poteete et al. 2010). Within our context, the unit of analysis ranges from small, e.g., local communities of natural resource users, to large governance systems, e.g., regional marine protected area systems, conservation agreements on the open seas, or international outcomes of conservation interventions. With these larger units of analysis, "broadly" does not mean that thousands of observations are analyzed; rather we are referring to databases of several dozen to several hundred observations, and often comprising different social and ecological contexts (see Table 1).
Arguably, this shortage of comparative data at critical scales reflects a failure of collaboration and coordination within the commons community. The commons community itself has developed the analytical tools to understand this as well. As Poteete et al. (2010:262) describe, collaborative research partnerships, like the ones we are considering, face collectiveaction problems, or a divergence between the interests of the group as a whole, and its individual members: Most scholars are individually better off if they invest in specialized skills and publish primarily single-method research in disciplinary journals; they are more likely to get a good job, get their research published quickly, receive tenure, and win grants and awards. The scholarly and policy communities, however, would be collectively better off if more research made use of multiple methods and integrated insights from a variety of disciplines.
This challenge is ubiquitous and derives from the public good nature of science and collaborative enterprises, and is compounded for data-intensive projects because of the infrastructure involved, itself a public good. Moreover, the number of participants involved in many of these projects raises coordination costs and makes it more difficult to prevent freeriding (Casari and Tagliapietra 2018). The diversity of perspectives that are involved can also make coordination more difficult. Representational diversity, for example, can help groups to a point, but, after that it becomes increasingly difficult for people to begin to understand each other, especially if the diversity in question is related to core values beliefs, and objectives of individuals (Cronin and Weingart 2019). As Poteete et al. (2010) also describe, the level of fragmentation within academia stymies efforts to address these challenges. Within academia there exists a high level of individualism at the level of the researcher, the research group, and the discipline. For each unit of social organization, collaboration is threatened by competition.
These challenges have arguably been more effectively met within the life sciences than in the social sciences, or within the socialecological space between them (Parker et al. 2010). The most notable examples of research networks within ecology include the Long-Term Ecological Research (LTER) and National Ecological Observatory Network (NEON) programs. Also of note is the National Center for Ecological Analysis and Synthesis (NCEAS), which has supported the synthesis of ecological data and research outputs for some time (Baron et al. 2017). Based on the NCEAS model, the National Socio-Environmental Synthesis Center (SESYNC) has stood since its inception as the pre-eminent center for the synthesis of social-ecological information, based on a rigorous approach to team science (Palmer et al. 2016). Indeed, this paper was conceived at a SESYNC working group meeting.
Although the findings associated with the research on working groups at these synthesis centers are important (Hampton and Parker 2011), these groups operate in a specific context, during a relatively short period of time and are supported by the center's infrastructure with the goal of aggregating existing data. Less is known about projects in "the wild," particularly interdisciplinary and transdisciplinary data-driven social-ecological projects of the kind we focus on here.
In addition to the challenge of collaboration, Poteete and Ostrom (2008) find that a critical challenge facing broadly comparative research is the tension between complexity and comparability (see also Cox 2008). With a limited amount of resources, each project must incur some opportunity cost by focusing more on individual cases, or more on representing them in a way that enables a comparison across them. Poteete and Ostrom (2008:189) describe a way to meet this challenge: Ecology and Society 26(1): 17 https://www.ecologyandsociety.org/vol26/iss1/art17/ Guidelines and checklists facilitate comparative research while leaving researchers with considerable flexibility to adapt their data-collection strategies to local conditions, or to supplement the basic set of issues to pursue local concerns.
We believe that these challenges are intertwined, and that a guiding checklist can help with each of them. The use of the word "checklist" here might surprise some readers. In a popular book on the topic, Gawande (2014) describes how checklists can be used to increase the consistency of a set of practices among practitioners. From this perspective, checklists are the enemy of inconsistency (also known as flexibility), by design. This lens has been used to criticize supposedly "blueprint" approaches to management as being overly formulaic (Cox et al. 2010). Here we reinterpret this concept to be at the center of the just discussed tension between flexibility and uniformity. Checklists can help researchers maintain a balance between flexibility (complexity) and uniformity (comparability) if they are presented not as simple boxes to be checked based on highly legible criteria, but more as guiding questions to be used as the basis for discussion among participants. In this way, our view of checklists is closer to the idea of a diagnosis rather than that of a blueprint. In what follows we reflect on each of the projects, their benefits, and the two fundamental issues addressed herein. We conclude by offering a version of the checklist that Poteete and Ostrom (2008) had called for. In so doing we hope to contribute to the ability of future scholars to address the key challenges discussed.

SUMMARY OF THE PROJECTS
See Appendix 1 for full descriptions of each of the seven projects. Each project is summarized by the following categories: (1) origins and purpose, (2) description and process, and (3) innovations, outcomes, and lessons learned. The Common-Pool Resource (CPR), Nepal Irrigation Institutions and Systems (NIIS), and the International Forestry Resources and Institutions (IFRI) projects were all developed at the Ostrom Workshop in the 1990s, while the others discussed here were developed in the following decades. These first three projects were designed to analyze the institutional dynamics of community-based natural resource management. The databases for these projects have several similarities owing to the overlapping network of participants and a common underlying framework (the IAD), variables, and definitions, which were established through the development of the first CPR database. The NIIS database further developed and refined the CPR database to specifically address the irrigation sector, and later to incorporate primary data. According to Elinor Ostrom (verbal communication), NIIS and IFRI are each a further development of the CPR adapted to the respective resource. Of the 593 variables that comprise the CPR database, 319 are also found in NIIS (54%), accounting for approximately 58% of its 554 variables. For IFRI, it has been estimated that the overlap is roughly 20%, although direct comparison is not feasible because of the use of an alternative system of naming conventions. The final four projects all drew on the original three and their use of the frameworks in different ways. Interestingly, none of them used the IAD framework on its own, although this still played a central conceptual role in each project.

Benefits of synthetic projects
The primary benefit of datasets containing a large number of cases is that they help address the "many variables, few cases" problem that characterizes smaller-n research, and facilitate comparative analyses that allow us to assess the generalizability of important relationships. Each project examined here has been used to develop and test hypotheses concerning the social, institutional, and biophysical drivers of environmental sustainability. The databases have thus enabled the study of SES governance to evolve from single-case based research to multiplecase comparisons enabling higher generalization to theory. Perhaps counterintuitively, the supporting infrastructure developed by several projects enables users to unpack the complexity of specific cases and objects, partly through visualization (SES Library, SESMAD, MACMON). In the SESMAD database, for example, the complexity of case representation facilitated a novel technique for case study analysis.
Some benefits are based on the type of data collected, and are not limited to broadly comparative research. For the projects that involve the coding of secondary data, this process serves as a type of structured literature review, and participants can gather a synthetic understanding of a literature. For projects that involve primary data there is the opportunity to develop collaborative relationships with local partners. This creates the potential for the knowledge produced by these projects to fill knowledge gaps of local partners in particular times and places.
For the broader research community, comparative databases provide scholars with a broad menu of potentially relevant factors, and an approach for consistently measuring them. Although lists of variables may seem a rather limited contribution, they encapsulate the evolution of the sustainability sciences from a field emphasizing simple panacea solutions to one involving complex interactions among a wide range of social, biophysical, and institutional variables (Agrawal 2003, Ostrom 2007, Liu et al. 2007). Students and researchers can draw from these lists to develop comparable research protocols and findings, and avoid reinventing the wheel when it comes to operationalizing and measuring key concepts and variables. The SESMAD website and the practitioner's manual for the MACMON project (Gurney and Darling 2017), for example, provide guidance for how future scholars and practitioners can implement the SES framework. Through such learning we may be able to produce more consistent projects and analyses in the future, even if these are not fully integrated into a common project.

Challenges and issues
Collaboration Producing the benefits just described requires collaboration among a set of academics and often nonacademic partners. As described above, the challenge of scientific collaboration can be straightforwardly interpreted as a public goods dilemma: it can be difficult to exclude members of a collaboration from the benefits it produces, leading to the free-rider problem. As seen in the previous section, there are multiple benefit streams that the projects produce. For the research group, a direct benefit comes from the publications through which the contributions of the https://www.ecologyandsociety.org/vol26/iss1/art17/ project become legible to the broader community, enabling participants to benefit reputationally (see Fig. 1).
In order to produce these publications, several inputs are needed. Underlying all other aspects of a project are the resources needed to carry out activities. These include social capital, by which we mean the mix of formal rules and informal norms that are required in order to provide incentives for researchers to contribute and a predictable social structure within which to operate: ideally a wellordered group has positions that confer well-understood rights and responsibilities, and these are at least minimally codified. In each of the projects discussed here, a subset of actors acted as leaders, lowering the cost of participation for other participants, in part by establishing what the rights and responsibilities for the group members are. A project's social structure should guide the process of developing the needed technical infrastructure, which includes guidance materials that provide project partners with a common definition of cases, components, and variables, as well as the procedures to use for collecting and entering data into the database. The construction of the databases involves the creation of at least two public goods. First, there is the data storage and management infrastructure. This includes the development of at least a coding manual and data storage platform. Providing all this infrastructure requires reaching a minimum of collaboration (threshold); however, once the infrastructure is provided, its maintenance is comparatively lower. Then, there is the data itself, which is a good that grows with the number of contributors. Thus, while infrastructure can be provided by a small group of collaborators or even a single leader, the population of the database requires the collaboration of larger numbers of contributors. The quality of the data is dependent upon the skills and fidelity of those who collect and enter them, and this issue becomes more challenging over time as different researchers (and other contributors) enter and exit the team.
A significant barrier to infrastructure development is that researchers often lack the knowledge required to design, develop, and maintain databases. In multiple projects this gap has been filled by IT professionals, but longevity of these contractual relationships can be difficult to maintain. Open-source software products, i.e., Kobo Toolbox, are a potential solution to sustainability and customization. Furthermore, technical difficulties, such as formulating queries in relational databases, can persist beyond the data collection stage, requiring an ongoing emphasis on strengthening data science skills with project teams, or collaborations with data scientists. In the past, this led to the replacement of complex analyses by simple ones, limited to a few key variables, with which simpler analyses such as linear regressions are then performed. This means that the division of the data into many tables (users, resources, rules) because of good technical practice de facto leads to simpler scientific analyses, since the lack of IT knowledge means that information can no longer be brought together in analyses. Here it can help to use graphical user interfaces to increase accessibility even for users who are not so technically proficient and for whom a collaborative project is one among many.
As discussed in Poteete et al. (2010), there are aspects of the research ecosystem that can make these dilemmas particularly challenging. Academic organizations tend to be individualistic, with individual-level metrics that lead to rewards and prestige. Collaborations such as those discussed here most often involve partners from different institutions who voluntarily join temporary collaborative groups, rather than belonging to a relatively fixed team within an organization, unlike groups in national science labs in the United States or others within the CGIAR system. A further problem is that grants are usually short-term, i.e., 2-3 years. Developing and maintaining a database, e.g., updating the technical infrastructure or entering new cases, however, is a longterm project.
As a result of these collective action challenges, progress in maintaining and expanding the databases has been intermittent for most of these projects as scholars struggle to maintain incentives for detailed data collection and coding, and to collect longitudinal data, with the notable exception of the IFRI database, where for some locations up to five measurements are available at intervals of a few years, and the NIIS database, where an extremely high coverage and completeness of the data was achieved through targeted visits to irrigation systems where the data situation was inadequate.
Further, those broadly comparative projects that employ an interdisciplinary approach or involve collaborations with nonacademic actors face additional collaboration challenges. Although most of the projects focus primarily on social and institutional information, in others, substantial biophysical (e.g., MACMON, IFRI, NIIS) data are collected and represented, thus involving engagement with natural scientists. Further, some of the projects including primary data collection also involved https://www.ecologyandsociety.org/vol26/iss1/art17/ engagement with resource management practitioners, from consultative to transdisciplinary approaches that entail knowledge coproduction to meet real-world information needs (Mauser et al. 2013). In the case of MACMON, the need for the project was identified by conservation practitioners and required subsequent collaboration with academics to develop and ground the project in theory. The subsequent diversity of perspectives involved in these cross-boundary approaches results in heterogeneous interests, priorities, project histories, epistemologies, and potentially tension over what constitutes salient, credible, and legitimate knowledge (Cash et al. 2003), particularly with respect to which SES attributes to include and how to operationalize them. Overcoming this tension and integrating diverse perspectives has been aided by employing the SES and IAD frameworks as basis for the projects because they effectively act as "boundary objects," being both adaptable to different viewpoints and robust enough to maintain identity across them (Star and Griesemer 1989). Indeed, the frameworks provide a shared language, a point of reference, and a means to facilitate discussion and learning across boundaries .
Collaborations with large conservation and development government and nongovernment organizations may provide a way to generate large-n longitudinal primary datasets. For example, given that the MACMON project was designed to meet the information and SES monitoring needs of the Wildlife Conservation Society's global coral reef program, it now forms the basis of much of their monitoring and its future sustainability is being built on integration across multiple projects and country programs. Indeed, many similar organizations have ongoing commons management projects in multiple sites for which they are required to undertake social and ecological monitoring, thereby circumnavigating the challenges associated with largely academic initiatives.
In the face of these challenges, we believe that researchers must ask themselves if it is worthwhile to engage in a broadly comparative research project. If a group decides to engage in such a project, it will be helpful to tightly integrate the ongoing maintenance of the database with teaching practice, and/or funding and collaborative agreements. In general, the more synergies that can be built between the larger project and related side-projects and activities, the more sustainable it will be.

Balancing complexity vs. comparability
A central challenge inherent to developing broadly comparable data projects is navigating the trade-off between case-based relevancy and generalizability. This challenge stems from tensions between the goal of accumulating datasets of comparable cases and the underlying theoretical framework that suggests that SES outcomes depend upon interactions among a wide range of factors. The former encourages scholars to adopt a reductionist approach by applying rigid definitions of the types of cases analyzed, the number of components included, and rely heavily upon a small number of variables and associated questions with fixed-item responses. It can privilege SES attributes that are more easily quantified than those that are not, e.g., power, values, and cultural context. A standardized approach increases the possibility of oversimplification and omission of SES attributes that are important in a certain place, including local expressions of those attributes . In contrast, the latter encourages scholars to adopt a flexible approach to provide a detailed representation of the components and the relationships among them as they occur in the world, and code them using large numbers of fixed and open-ended questions. The approach taken to reducing the trade-off between case-based relevancy and generalizability in the MACMON project was to design the standardized SES "core" framework with the intention that it be supplemented with context-specific indicators. A similar division between a "core" set of variables and more peripheral, contextspecific variables has been discussed for both the IFRI and SESMAD projects as well.
Selection of variables is a core challenge that involves important questions about their theoretical relevance, the availability of indicators, and the relative costs and benefits of including them. In theory, we would expect the number of variables to increase with the number of research questions and diversity of cases that a project aims to address. In practice, however, database projects tend to err on the side of inclusion, reflecting the generally recognized importance of many factors (e.g., see Agrawal 2003). Large numbers of variables have the benefit of facilitating numerous publications from a single project, as most notably seen in the IFRI program. It can also help avoid the trap of "tunnel vision" where a very narrow portion of a system is examined to the exclusion of others, and analytical pluralism in general. But this also results in a laundry list of variables, most of which never feature in an empirical analysis. The IFRI data protocols, for example, contain a variable about improved bee-keeping techniques, which to our knowledge has never been used. Additionally, this situation can encourage theory-free data mining and related analytical issues, e.g., p-hacking, which can undermine the replicability of comparative work. Although these behaviors are exhibited at the analysis stage, we believe that they are facilitated by the presence of many measured variables without an immediate analytical "home." It is easier to go on a "fishing expedition" if you have many fishing grounds. At the same time, large datasets can be explored to generate new hypotheses, but in this case, scholars need to be clear that such hypotheses are being generated and not tested.
There is similarly a tendency to develop complex database structures, e.g., with a relational database, that reflect the interactions of actors, resources, and governance systems that collectively comprise SESs. The SESMAD database, for instance, is highly flexible in its ability to capture interactions among system components, but this creates challenges for comparing these heterogeneously characterized cases. Such data structures can pose challenges for aggregating measures across multiple components, i.e., 2 or more actor groups, requiring analysts to make nontrivial decisions about whether to use sums, means, medians, or other measures. For instance, empirical studies of forest outcomes using the IFRI database have tended to aggregate multiple user groups by taking the sum of features such as group size, the mean of others such as fuelwood dependence, and the maximum value for institutional arrangements such as local monitoring.
This example reflects a broader challenge that commons research has faced for some time, this being the consistent and valid measurement of (often biophysical) outcomes (Young et al. 2006). Investigators often have to decide how to pool nonstandardized Ecology and Society 26(1): 17 https://www.ecologyandsociety.org/vol26/iss1/art17/ indicators from different methodologies, or change their existing methods to a standardized and more comparable approach. For example, biodiversity is often a desired outcome for SES systems, which can include measurements of species richness, ecosystem integrity and functioning, or other ecological proxies for productive and sustainable ecosystems. Ecological monitoring can identify key indicators for comparative studies, however measuring biophysical variables using standardized and comparable methods can remain challenging and requires insight and forward-thinking strategies within an SES research team.
An underlying driver of some of these challenges is the lack of a core set of well-defined theories to drive data collection and analysis. Although there has been some work done to encourage midrange theorization (Young 2002, projects tend to be designed to test a multiplicity of theories. The projects included here share a broad framing (social dilemmas in environmental management and governance), but this is inclusive of a great many theoretically important factors. NIIS is a mild exception here, in that it was designed to test the effects of decentralization in Nepal and compare top-down vs. more bottom-up irrigation systems, and yet still includes over 500 variables. Likewise, the MACMON project was designed to understand the social and ecological outcomes of comanagement and the institutional and contextual characteristics associated with different sets of outcomes.

A GUIDE FOR FUTURE PROJECTS
We have suggested that scholars ask themselves whether engaging in broadly comparative work is worth it, given the challenges that we have just described. These are significant challenges, but we believe that they must be met in order to advance a critical line of social-ecological research. Simply put, we believe there is too much reinventing of the wheel in terms of frameworks and empirical work. There is deep value in having a diversity of methods, but much of this comes from the ability to learn from and consolidate the practice of successful approaches. We do not mean to diminish the importance of more case-oriented research in this endeavor. In the end, good research is good research, whatever its framing and perspective, and our goal here is to support good comparative research.
With all of this in mind, we have developed a checklist of questions for future groups to consider as they develop their own broadly comparative data projects ( Table 2). "Checking" an item here does not mean that a particular task is finished, but rather that a question has been asked and various answers considered. It can be helpful for any group launching a collaboratively comparative project to brainstorm their responses to these questions as they proceed. Some of the social content in this checklist represents the application of collective-action theory with which we are expertly familiar, such as  design principles, and the importance of boundary objects and actors in facilitating cooperation within diverse groups . We believe we ought to view research groups as communities, and that members, and particularly leaders of such groups, should maintain a "reflexive" frame of mind, by turning their analytical lenses inward to understand the factors that affect their own behavior and the behavior of other group members.
Beyond these considerations for individual projects and groups, it would help if the broader research community valued less traditional outputs more. We need to move away from a productivist culture in social-ecological science (Paasche and Österblom 2019), and toward one that creates space and time to build understandings across intellectual and professional divides. We also need to be cautious that the drive for large-n research does not ignore relevant theory and squeeze out the detailed smalln case studies (e.g., Morrison 2017) that are a critical means for generating new hypotheses (Yin 2017, Cairney et al. 2019. We also need to consider how large datasets and single case studies can more usefully intersect.
Ultimately, we need to do a better job at sharing the public goods that over time make everything else possible. Making research protocols publicly available in a single platform could help. Open data platforms are also a promising option. Other scientific fields have already demonstrated that this is feasible and offers enormous benefits. For renewable energies, there are various open data platforms, e.g., https://www.renewables.ninja/ or https:// open-power-system-data.org/. Such a platform could also contribute to setting standards in data quality, variables measured, and so greatly enhance comparability. Finally, a more wiki-type database with broad community development could help support the needed social capital for future collaborative projects. These platforms could also be promoted via linkages with open access journals like Ecology and Society and the International Journal of the Commons. As a decentralized research community, we should leverage our diversity by experimenting with these approaches, and over time, select and maintain those that support methodological and theoretical synthesis. Aside from peer-reviewed articles, other important and sometimes underemphasized outputs could include shared infrastructure, guides for practitioners, and learning resources. Project framing

Outcomes
Consider a range of outcomes, including theoretical and methodological development, uptake in real-world policy and practice, establishment of a community of practice, colearning. Crosscutting Synergies and cobenefits Take advantage of path dependence and interdependencies. Consider at each step how the project can leverage existing resources (databases, coding forms) and synergize with other activities (teaching and course development) to lower costs and maximize multiple benefits. Crosscutting

Boundary spanners
Consider whether boundary actors and/or boundary objects can serve to establish a common understanding amongst a diverse set of project participants. Social Group boundaries Criteria for membership in a research project are made clear to all members. Social Group size Larger groups may enable interdisciplinarity and division of tasks but may also increase temptations to free-ride and undermine social capital. Social Group heterogeneity More diverse groups offer more perspectives, but increase the costs of establishing a common understanding of protocols.

Social
Social capital Social capital facilitates dialogue among group members, and motivates members to participate. Social Proportionality If group members who contribute more also perceive greater benefits, this can facilitate continued participation and prevent free-riding.

Social Leadership and followership
Leaders establish important public goods (infrastructure norms) for a project. It also helps to have flexible leaders who can be good followers for different parts of a project. Social Nested enterprises A core group of contributors can effectively move a project forward with more intermittent contributions from others. This extends the leadership principle to a small group. Social Incentives A balanced mix of short-term and long-term incentives may be required to motivate members to participate. Social Adaptive management Individual and social learning of the group should be considered as an output of the project and inform project developments.

Social Monitoring and sanctioning
Authorship and contributions should be traceable to some degree, and graduated sanctions should be established to ensure that members are contributing, but without alienating members for extreme sanctions early on. Social Intergroup collaboration Working with external actors such as IT professionals and NGOs can fill knowledge gaps and satisfy multiple values, but increases the costs of collaboration. Technical Database/case complexity More complex, disaggregated case descriptions in a DB increases representational validity and averaging over difference, but decreases comparability across cases. Technical Variable count Large numbers of variables facilitate empirical analysis of multiple questions but poses challenges for coding and may lead to unused data. Technical Core/periphery Identifying a core set of essential variables to be explored in all cases and reserving others to be explored in subsets can help with the variable count challenge. Technical Database format Flat files are simpler, but relational databases (RDB) enable more complexity. RDB that are too complex may be difficult to query and be underutilized. Technical Database location A server increases consistency through the one version. A local solution (multiple copies) enables flexible handling, modifications and a simpler implementation, but also fragmentation. Technical Database and project hosting Hosting at an institution can help provide a home for a project, but then institutional rules may pose barriers to collaboration.

Technical Usability
Providing graphical user interfaces for data entry and data extraction can help make data available to group members. Methods Primary vs secondary data Primary data is more costly but may offer higher measurement validity, while secondary could also be used to check inferences.

Methods Sampling
Random sampling is seldom feasible in these types of projects, leaving purposive sampling as the main alternative.

Methods Data collection platforms
Digital platforms such as Qualtrics and Kobo toolbox may improve data collection and management options.

Methods Levels of measurement
Interval-level measurement provides the greatest flexibility for analysis, but is often unavailable or inconsistent across cases.

Methods Reliability of measurement
Measurement reliability is made more difficult as group size and group heterogeneity increases, and as database complexity and variable count increases.

Methods Language of measurement
English facilitates setting up of platform but may lead to exclusivity, for example, by excluding local actors and partners.

Methods Analysis and theory
It is helpful to consider what theories are to be tested and which analyses will test them early on, to constrain data collection and avoid p-hacking-type behaviors that lower reproducibility. https://www.ecologyandsociety.org/vol26/iss1/art17/

Origins, actors and purpose
The seed for the CPR database project emerged from a National Research Council Panel on the Study of Common Property Resource Management formed in 1983. Based on discussion at this meeting Oakerson (1986) developed a framework that captured key concepts and variables drawn from multiple disciplines to allow scholars to consistently analyze and compare how resource users interact with respect to their use and governance of different types of commons. Oakerson's framework provided the foundation for developing coding forms to capture key features of common pool resources and resource users. In the mid-1980s, Elinor Ostrom and colleagues completed these forms as part of a larger commons project. In addition to the development of the CPR database, the larger project included fieldwork, game theory, and laboratory experiments (Ostrom et al. 1994).

Description and process
The CPR database was innovative for its time because it was relational, which reflected the importance given in the IAD framework to the joint impact of biophysical, social and institutional conditions on resource use decisions. Thus, each coding form has a relation to other coding forms. For instance, a single location coding form would be linked to multiple appropriation resource forms. The appropriation resource form represents the "Biophysical conditions" in the IAD framework. It consists of a series of questions about the location, size, stock, and flows as well as questions about physical infrastructure related to an appropriation resource. Adequately identifying key biophysical features applicable across many resources was a key challenge for the project.
The operational level coding form capture how resource users interacted with one another and with the resource. In other words, it captured the focal action situations. When it became clear that resource users differed from one another in ways important to the IAD framework, a subgroup form was introduced. Related to the subgroup form was the operational rules form, which was organized around the IAD framework's rule typology (Kiser and Ostrom 2000). The rules form is arguably the most complex of the forms, in order to capture rules that would be applicable across different types of resources, and at the same time capture key dimensions of the rules. In sum, the CPR coding forms attempted to develop a set of questions, based on the IAD framework, which could be applied to any common pool resource, and to store that information in a way that would be efficient and accessible.

Innovations, outcomes and lessons learned
The CPR coding forms were the first comprehensive effort to operationalize the IAD framework in relation to a type of good -common pool resources-and to operationalize variables from an emerging theory of common pool resources not explicitly represented in the framework. The CPR database has been used to test hypotheses concerning the relationship between user group characteristics, collective action and resource conditions (Ruttan 2006;Frey and Rusch 2014). Throughout the initial process, the framework and theory evolved together. For instance, measures of some, but not all, of  design principles are captured in the coding forms. And some parts, but not all, of the syntax of the grammar of institutions (Crawford and Ostrom 1995) are captured in the rules coding form.
However, even though frameworks and theory work hand-in-hand, it is important to recognize the distinction between them. The coding forms consist of several hundred variables. Not surprisingly, there is a lot of missing data, which points to the limitations of exclusive reliance on secondary data. In addition, the abundance of missing data also suggests the need for further prioritization of included variables through a process of theory development and testing. Ultimately the CPR database should be viewed as a part of a larger research project: the CPR database provided the foundation for the NIIS and IFRI projects. (Ostrom et al. 1993). In Nepal, decentralization translated into a process of increased involvement of communities in the irrigation sector, following a period of partially donor-driven centralization. Some scholarly works from the Ostrom workshop are based on these data and describe them in more detail (Lam 1998;Tang 1992;Ostrom 1992).

Description and process
Throughout its implementation, the NIIS database was based on the framing and infrastructure of the CPR database. Its focus was still very much on the social dilemmas that local resource users faced, and the scale of analysis was still highly local. To develop the project, Ostrom visited Nepal and began to engage with colleagues there, obtaining written reports of local irrigation systems produced from rapid rural appraisal, student theses, field notes, and standard publications. The Workshop team and colleagues began to develop a set of coding forms based directly on the CPR coding forms, but increasingly adapted to the irrigation sector. Over time it was determined that these secondary materials left too many gaps in the data for the studies they reported on. Thus, a protocol was developed to quickly collect primary data on existing sites to fill these data gaps.
Core to the Workshop frameworks is the distinction between resource systems/infrastructure and resource units, and the understanding that management requires separate attention to each (Hinkel et al. 2013). In terms of identifying these components, the irrigation system has served as a paradigmatic "simplest case." The irrigation system itself is seen as the resource system associated with public good provision problems, while the water obtained via this public good is seen as a resource unit (see Ostrom et al. 1994). Moreover, because of their connection to a physical subcomponent of the irrigation system, identifying groups within this project proved to be easier than it has for the CPR or IFRI programs.

Innovations, outcomes and lessons learned
The NIIS represents a step forward by incorporating secondary and primary data collection, and multiple visits over time, whereas the CPR only consisted of one-shot secondary data. This introduced the new challenge of obtaining such data consistently, which was partially addressed by the use of the existing coding forms originally developed for the CPR database. A second innovation in the NIIS database was the theoretical focus on decentralization as a process, and an explicit comparison between relatively centralized and decentralized systems in Nepal. This focus helped constrain the research and provide a theoretical basis for the ensuing work, thereby addressing a key challenge faced by the CPR project, which was the sheer number of variables involved compared to the number of cases described in the database. 21 cases (8%) described in the database are managed by the Nepalese government (AMIS, agency-managed irrigation systems), 208 cases (79%) are managed by users (FMIS, farmer-managed irrigation systems), and 28 cases (11%) are jointly managed. A third innovation was the explicit analysis within resource user groups of heterogeneities in resource dependence and associated vulnerabilities (Ostrom 1992). Finally, the NIIS added sector-specific biophysical indicators of the irrigation systems to enable analyses tying institutional arrangements to these outcomes. A key finding that resulted from this work is that farmer-managed systems often outperformed agency managed systems, even with the superior technology of the latter, in part because this technology did not conform well with the traditional institutions of the farmers in those systems (Lam 1998). More recently, the database has been used to the variety of factors that make farmer and agency managed systems more sustainable (Lam and Ostrom 2010), and the different ways that local communities and government divide labor in co-managed irrigation systems (Frey et al. 2016).

Origins, actors and purpose
The IFRI project started in 1992 with funding from the United Nations Food and Agriculture Organization (UNFAO). The IFRI database is relational, separately identifying community forests, forest user groups, nearby communities, and on-the-ground biophysical measures of forest conditions such as species, tree density, and biomass. While building on the CPR and NIIS databases and containing some overlapping variables, it is also larger and more complex than either previous database. Later projects have used IFRI protocols as a baseline and collected additional data such as remote sensing information and household surveys. Furthermore, a major vision for IFRI was to make a longitudinal database where sites could be visited at regular intervals in order to assess social, institutional, and ecological variation both crosssectionally and over time. From the beginning, the project was an interdisciplinary collaboration meant to link institutional variation with ecological outcomes.

Description and process
The Workshop required a training seminar on IFRI data collection methods for every primary investigator for an IFRI case. Research sites around Bloomington, Indiana served as pedagogical laboratories for quantitative and qualitative training. Around 2003 the IFRI project moved to the University of Michigan under the direction of Arun Agrawal. The IFRI program has moved once again under the direction of Ashwini Chhatre at the Indian School of Business in Hyderabad, India.
Research teams for an IFRI team are composed of at least one person to head social science data gathering (social and institutional data) and one person to lead efforts at biophysical data (forest measurement). Research teams have a binder of information with each variable (question) that they answer and then upload into site-specific spreadsheets. The research subjects (users of community forests) are not surveyed to answer a specific set of questions; rather, IFRI researchers talk with key informants, gathering information that they then later enter as data. To complement these, there are standard protocols to collect biophysical data through well-established forest mensuration techniques.

Innovations, outcomes and lessons learned
In Ostrom's inaugural publication when elected to the National Academy of Science (Ostrom and Nagendra, 2006), she relied on IFRI data to provide some of the first quantitative comparative analysis for her main thesis in Governing the Commons ) -that formal designations such as private, community, or government management of forests were not significant predictors of good biophysical outcomes. Instead, she found that rule enforcement was a significant and meaningful predictor. Coleman (2009) also used comparative IFRI data to show that this result holds when looking at trends in forest change (thus controlling for place-specific differences between forests). In a series of papers, Chhatre and Agarwal (2009) and Persha, Agrawal, and Chhatre (2011) found that local rulemaking autonomy is important both for carbon storage (as proxied by above-ground biomass), biodiversity preservation, as well as forest livelihood benefits. IFRI data have also been used to examine the ability of local actors to evaluate forest condition. Salk et al. (2014) showed that foresters' qualitative assessments of forest condition are broadly correlated with extensive and highly time-consuming quantitative measurements using hundreds of plots. Salk et al. (2020) later examined the feasibility of using IFRI data to capture biodiversity outcomes in forests, concluding that the plot-based tree-centric IFRI protocol should be complemented by additional methods at little additional cost.
A challenge for the IFRI program has been the collection of data for hundreds of variables at each research site. The tradeoff of a program seeking to collect such fine-grained, comparable data across sites is that some variables in some sites were less relevant than others. It also proved to be quite burdensome for someone wanting to be involved in collecting the entire inventory of data if resources were limited in doing so. Recently, alternative, short-form IFRI protocols were adopted to lessen the burden of researchers unable to collect data on variables within the full protocol. Furthermore, it has proven difficult to convince funding agencies to continue funding the main project and recurring visits to the same field sites over time. Instead, the program has come to rely more and more on individual researchers finding funding to collect data at a particular site. Thus, some sites in IFRI have been studied multiple times across semi-regular intervals over time, while for many sites data collection has only occurred once.

Origins, actors and purpose
The SES Library (seslibrary.asu.edu) project emerged from two NSF-funded projects focused on characterizing robust institutions. Both projects involved collaborations with Elinor Ostrom to use data from the CPR database and other case studies to extend Ostrom's focus on institutions and collective action in an attempt to understand the capacity of institutional arrangements to cope with change and persist over time, i.e. their robustness. The study of the dynamics of institutions and institutional change required the development of formal mathematical models connected to data. The SES Library is an effort to develop and curate qualitative case study data, coded data (CPR, NIIS) and mathematical models.
Understanding robustness either requires a mathematical model or a number of time series data sets (one related to exogenous drivers and another to system change related to the external drivers of sufficient duration). Such time series are exceedingly rare for SES. Thus, emerges the importance of dynamic models for studying robustness. Understanding the robustness of institutional forms was the impetus to develop the Robustness of SES Framework. This is the reason why the SES Library uses the Robustness of SES Framework to organize conceptual models and emphasizes the curation of mathematical models.

Description and process
Unlike the other databases described here, the SES Library is not organized around the coding of institutional arrangements, social, and biophysical contexts. Rather, it is organized around the underlying structure of SESs. That is, the SES Library is designed to develop conceptual models of classes of related SESs. For this, the SES Library uses the Robustness Framework which characterizes SES as composed of 4 key elements: Resource Users, Resource System (the natural resource system from which resource users withdraw resource units, Public Infrastructure (institutions + monitoring and sanctioning capacity to operationalize institutions as rules-in-use), and Public Infrastructure Providers (collective choice). The nature of these elements and how they are linked up provides a way to classify the structure of feedbacks in SES.
The SES Library is organized around 'cases'. A 'case' is a 'unit of scholarship' around an SES or collection of SESs. Thus, a case can be place based -i.e. a study of a particular SES. It can also be theme-basedi.e. a comparative analysis of a specific type of SESs, e.g. of coastal systems. A key element of the SES Library is that it is an "ecosystem" of cases where place-based cases can be linked to theme-based cases and models. This is facilitated by the structure of a 'case' data type which includes 1) spatial information, 2) summary of resource type and social dilemma, 3) brief "institutional analysis" that follows a fixed template that employs the IAD, rule types, etc. and is linked to coded institutional data if available, 4) a system representation, 5) sources -i.e. case contributors and source documents, and 6) related studies and models. The last element allows users to follow links to studies in nearby areas (e.g. with similar problems and issues), to models either of the case itself, or to models that address similar issues and problems.

Innovations, outcomes and lessons learned
The main technical innovation is the inclusion of a real-time dynamic model visualization tool. This tool allows users to explore how key parameters affect model behavior without downloading any software, etc. Also, the SES Library should not be seen as a "database". It is both a database and a meta database. It has its own data and connects to other databases -as part of cases. That is, if a place-based case has multiple related models in different databases, those databases are linked. Further, the SES Library works on a "wikipedia" model. Anyone can request access, add cases or edit cases. The SES Library is focused on developing a community of practice rather than providing data. Given the costs involved, the fact that the SES Library is used as part of the developer's research program helps considerably in maintaining, improving, and expanding the database. Perhaps the key take home lesson is such databases should be tightly integrated with teaching practice, and/or funding and collaborative agreements.

Origins, actors and purpose
The SESMAD project was initially conceived to enable comparative analysis across a range of cases of environmental governance, particularly complex, large-scale cases (see Cox 2014). While it is not limited to large-scale cases, the database was developed with the idea that multiple interactions among social and ecological components in large systems would be represented for each case. The project was conceived at a meeting of Resilience Alliance Young Scholars (RAYS), prior to a meeting of the Resilience Alliance in 2011. Scholars associated with this group were joined by several others associated with the Workshop in Political Theory and Policy Analysis at Indiana University, USA.

Description and process
Physically, at the core of the SESMAD approach is a relational database and a website-based front-end (sesmad.dartmouth.edu). The set of interconnected tables in the database store information on socialecological systems, as well as the variables used to measure relevant aspects of these systems and the theories that subsequent analyses could test. This database and a series of data-entry forms enables a user to consistently enter data and describe their system(s).
Over a series of in-person meetings, SESMAD group members developed a common understanding of the variables required to describe the subsequent cases. Computing support staff were heavily involved in the design of the final version of the database and providing much of its functionality. As such the SESMAD project has benefited from a type of transdisciplinarity between scientists and software engineers. Once the database was near completion, an initial set of five large-scale cases was coded (Fleischman et at. 2014a). Following this, a set of marine cases were analyzed and a theories table was added to leverage the existing variables table.

Innovations, outcomes and lessons learned
The two primary innovations associated with the SESMAD database are based on the structure of the database itself. First, the SESMAD database represents the first repository of variables and theories relevant to natural resource management and governance. These aspects have enabled a major synthesis of the variables and theories that have been developed to study natural resource governance . Second, the SESMAD database directly implements the "action situation" concept, initially associated with the IAD framework and implicitly included in the SES framework via the concept of "interactions". When cases are coded in the SESMAD database, users code the interactions of multiple system components, following the framing that the database inherited from the initial frameworks (see Epstein et al. 2014, Evans et al. 2014, Fleischman et al. 2014b. With this fidelity the SESMAD database answers an important question: how might we store data that reflects the complexity described by the most common representations of these frameworks? The subsequent analyses of these complex cases have led to several publications documenting the complex trade-offs among social and ecological outcomes that large-scale MPA governance experiences (Davies et al. 2018), and the extent to which design principles for successful community-based resource management can explain patterns in these outcomes (Ban et al. 2017).
As early-career researchers, the SESMAD team faced the traditional incentives and constraints (publish or perish) that could have discouraged them from contributing to a longer-term project that would not yield immediately legible results. To counteract this, much of the success of the SESMAD project depended on the social capital that has existed among a relatively small group of around 14 members. This social capital benefited from and enabled the multiple in-person meetings that were held during the early years of the project.
When designing the database that would be used to represent cases, the SESMAD project has had to contend with the trade-off between complexity and generalizability. Due to its fidelity to the SES framework and the concept of an action situation involving multiple actors and biophysical components inherited from the IAD framework, cases in SESMAD are costly to code and difficult to directly compare with each other. If we envision a comparison as occurring between units at the same scale, it is not obvious, for example, how we can compare large complex cases with multiple interacting components. The framework and database also do not provide guidance on what the boundaries of the system should be set at. Finally, the project has suffered from the "many variables, few cases" problem, due to the large number of variables the database contains and the costliness of coding the cases. The literature that describes complex frameworks generally does not address these critical issues.

Origins, actors and purpose
The small-scale Fisheries Institutions and Governance (ssfIG) framework originated as an intent of a group of academics and practitioners to systematize existing knowledge on the social-ecological dimensions of marine systems in northwest Mexico and to coordinate future work in the region through the development of a common framework. The initiation of the National Diagnostics of Fisheries Organizations project, funded by the Walton Family Foundation and the Resilience and Adaptive Capacity of Small-Scale Fishing Communities and Coastal Marine Ecosystems to Environmental and Economic Variability, funded by the NSF (CNH-L: 1632648), served as a catalyst for the development of the framework.

Description and process
Conceptually, the ssfIG system framing is rooted in the structure of the IFRI database. Each fishing community studied has one or more user groups that interacts with one or more resources. Each system component is characterized via a set of systematically measured variables. Theoretically, the ssFIG framing draws on the common pool resources literature (e.g. Agrawal 2003). The primary goal behind this initiative is to evaluate institutional performance by examining variation of a range of ecological and/or social outcomes.
The ssfIG framework consists of 147 variables that are arranged around the SES framework. Each variable is operationalized through one or more questions, and is linked to a particular data collection instrument. Such instruments range from surveys and interviews to focus groups and participant observations. Social science data are stored in a proprietary data repository, ssfIG database (http://ssfdatabase.vm.duke.edu). Ecological data is currently not unified, although there is an ongoing effort to consolidate it within a data storage and visualization platform, dataMares (http://datamares.ucsd.edu/).
The variable identification process started with a review of the common pool resource, resilience, and organizational literatures. It was further refined through workshops and focus groups that included practitioners, managers, and resource users from Mexico. Knowledge and expertise of the participants in these venues was a critical step towards generating a list of variables that were fully relevant to the small-scale fisheries context. Variable operationalization followed a similar path where the initial set of questions developed for each of the variables was discussed and modified with input from all participants.
The ssfIG framework and its methodological approach have been implemented in Mexico and are currently ongoing in Trinidad and Tobago and Suriname in collaboration with Food and Agriculture Organization of the United Nations. There is an ongoing effort to expand it to other regions of the world.

Innovations, outcomes and lessons learned
The ssfIG framework has been implemented to systematically collect data on small-scale fishery governance at multiple points in time and across different spatial contexts. The goal of the project has been the co-production and co-ownership of knowledge by all partners (Shackeroff and Campbell 2007). This requires active engagement of all the research partners in all phases of research.
During the initial phase partners evaluated the framework and ensured that the identified variables and their operationalization were adapted to local social-ecological realities. The continuous and sustained interaction among the partners created a space that allows for honest and respectful discussions and debates to emerge, which, over time, led to strengthening of social capital among the participants. The result was a generation of more reliable data and more robust interpretations of the results based on diverse expertises and experiences. Introduction of the ssfIG framework as a boundary object at the inception allowed participants to clarify and agree on a common set of terms and their meanings. Within the transdisciplinary setting, this substantially reduces transaction costs associated with the development of research questions and interpretation of results that can emerge from having multiple, and sometimes conflicting, interpretations of the same concepts.
Active participation of a diverse set of actors is a fundamental requirement for the successful implementation of the ssfIG framework as their respective expertises and experiences are necessary for a firm grounding of the framework within the particular social-ecological context. However, initiating and maintaining active participation within the interdisciplinary setting is largely constrained due to differences in political, ethnic, cultural, or religious values that may exist among the participants, limited financial and/or personnel resources, and changes in the broader political and environmental setting (e.g. government elections, civil unrests, droughts, and cyclones). This inevitably results in longer timeframes when compared to conventional research and needs to be explicitly considered in planning activities.

Origins, actors and purpose
The Marine and Coastal Monitoring (MACMON) framework (Gurney andDarling 2017, Gurney et al. 2019) was developed through a two-year transdisciplinary process with the aim of supporting the Wildlife Conservation Society (WCS), an international non-profit, in identifying the social and ecological outcomes of their coral reef management programs. Since 2016, it has been implemented in six countries (Indonesia, Fiji, Kenya, Madagascar, Solomon Islands, Papua New Guinea), yielding data on over 125 communities and 5000 households, and is intended to form the basis of WCS's long-term coral reef monitoring.

Description and process
The MACMON framework comprises ~90 variables that are organized under Ostrom's SES framework. Data are elicited using underwater ecological surveys, household and key-informant surveys, and publicly available secondary data. Importantly, the social surveys are designed to be supplemented with project-and context-specific variables. Data are housed in open-source data platforms, with social data managed in Kobo Toolbox (www.kobotoolbox.org) and ecological data in the built-for-purpose MERMAID tool (www.datamermaid.org).

Innovations, outcomes and lessons learned
The transdisciplinary MACMON project seeks to contribute to advancing the science and practice of SES and commons governance, thereby contributing to bridging the research-implementation gap. Indeed, MACMON data are currently informing decision-making at multiple levels, from individual project assessment to global program-level prioritization, and are being used for multi-country comparative analyses addressing some critical gaps in the literature related to multiple outcomes. A key innovation of the MACMON framework is the transdisciplinary development of Ostrom's SES framework for application to real-world management across multiple countries. The MACMON approach of standardized SES variables supplemented with context-specific indicators helps ensure the framework is relevant to a range of projects and contexts, providing a means to address the tension between casebased relevancy and data comparability (Partelow 2016).