Ecology and SocietyEcology and Society
 E&S Home > Vol. 26, No. 2 > Art. 2
The following is the established format for referencing this article:
Paas, W., I. Coopmans, S. Severini, M. K. Van Ittersum, M. P. M. Meuwissen, and P. Reidsma. 2021. Participatory assessment of sustainability and resilience of three specialized farming systems. Ecology and Society 26(2):2.

Participatory assessment of sustainability and resilience of three specialized farming systems

1Plant Production Systems, Wageningen University & Research, The Netherlands, 2Business Economics, Wageningen University & Research, The Netherlands, 3Social Sciences Unit, Flanders Research Institute for Agriculture, Fisheries and Food (ILVO), Merelbeke, Belgium, 4Division of Bioeconomics, University of Leuven (KU Leuven), Belgium, 5Department of Agricultural and Forestry Sciences, Università degli Studi della Tuscia, Italy


There is a need for participatory methods that simultaneously assess agricultural sustainability and resilience at farming system level, as resilience is needed to deal with shocks and stresses on the pathways to more sustainable systems. We present the Framework of Participatory Impact Assessment for Sustainable and Resilient FARMing systems (FoPIA-SURE-Farm). FoPIA-SURE-Farm investigates farming system functioning, dynamics of main indicators, and specifies resilience for different resilience capacities, i.e., robustness, adaptability, and transformability. Three case studies with specialized farming systems serve as an example for the used methodology: starch potato production in VeenkoloniŽn, The Netherlands; dairy production in Flanders, Belgium; and hazelnut production in Lazio, Italy. In all three farming systems, functions that related to food production, economic viability, and maintaining natural resources were perceived as most important. Perceived overall performance of system functions suggest moderate sustainability of the studied farming systems. In the studied systems, robustness was perceived to be stronger than adaptability and transformability. This indicates that finding pathways to higher sustainability, which requires adaptability and transformability, will be a challenging process. General characteristics of farming systems that supposedly convey general resilience, the so-called resilience attributes, were indeed perceived to contribute positively to resilience. Profitability, having production coupled with local and natural resources, heterogeneity of farm types, social self-organization, and infrastructure for innovation were assessed as being important resilience attributes. The relative importance of some resilience attributes in the studied systems differed from case to case, e.g., heterogeneity of farm types. This indicates that the local context in general, and stakeholder perspectives in particular, are important when evaluating general resilience and policy options based on resilience attributes. Overall, FoPIA-SURE-Farm results seem a good starting point for raising awareness, further assessments, and eventually for developing a shared vision and action plan for improving sustainability and resilience of farming systems.
Key words: adaptability; agriculture; robustness; social-ecological systems; transformability


Assessing sustainability and resilience of farming systems

Sustainability and sustainable intensification of agriculture is well studied (Pretty 2008, Godfray 2015) and multiple frameworks and tools for sustainability assessments are available (Alkan Olsson et al. 2009, Arodudu et al. 2017, Sieber et al. 2018). Several approaches are specifically designed for assessing multi-dimensional sustainability and sustainable development with participatory approaches (Morris et al. 2011, Delmotte et al. 2013, König et al. 2013, Vaidya and Mayer 2014). Yet, these have not been designed to study resilience of agriculture, i.e., the robustness, adaptability, and transformability of agricultural systems in the context of shocks and challenges (Meuwissen et al. 2020). Many studies have contributed to the search for operationalization of resilience research in agricultural and food systems (e.g., Callo-Concha and Ewert 2014, Ge et al. 2016, Prosperi et al. 2016, Peterson et al. 2018). Only few have reflected on addressing agricultural sustainability and resilience simultaneously (e.g., Tendall et al. 2015, Meuwissen et al. 2019). We see sustainability and resilience as two separate, but often complementary concepts that may, or may not influence each other, depending on the situation (Marchese et al. 2018). Assessing both sustainability and resilience is necessary to move beyond static sustainability assessments and to explain dynamics. Addressing both is also needed to identify unsustainable but resilient systems or vice versa. Resilience is needed to deal with known and novel shocks and stresses to keep track of pathways to more sustainable systems. Other researchers have studied resilience and sustainability of farms (e.g., Darnhofer 2010, 2014), but did not address the farming system as a whole in which multiple actors beyond the farm also play a prominent role. Ashkenazy et al. (2018) address resilience at farm and regional level, but lack the perspective of a well-defined resilience assessment framework and the attention for the role of different actors. The latter can be evaluated with a participatory assessment that is necessary to adequately address perspectives and issues of multiple actors and issues. Currently, multiple resilience assessment frameworks are available (Quinlan et al. 2016, Douxchamps et al. 2017). The “wayfinder guide” of the Stockholm Resilience Centre ( provides an extensive framework for the integration of multiple, iterative sustainability and resilience assessments for social-ecological systems, including notions on persistence, adaptability, and transformability. However, to the best of our knowledge, there is no single framework, designed for a one-day workshop, that combined the assessment of multi-dimensional sustainability and resilience (specified and general) of farming systems that include notions of all three resilience capacities, i.e., robustness, adaptability, and transformability.

In this paper we aim to present and test a framework for using a participatory integrated assessment (PIA; Toth 2001, Ridder and Pahl-Wostl 2005) to address perceived sustainability and resilience of farming systems. PIA, in combination with other methods, can contribute to a project cycle where the following steps can be distinguished: data gathering and analysis, planning, implementation, and monitoring and evaluation (Ridder and Pahl-Wostl 2005). In the PIA presented in this paper, the emphasis is on data gathering and analysis at the start of a project cycle. Participatory input allows to identify the most important indicators of the system from the stakeholders’ point of view. The identification of these indicators is a first step toward determining the identity of the system, i.e., mapping its most important structure and feedback mechanisms (Cumming and Peterson 2017). Participatory input also allows to assess variables that are not readily measurable, e.g., social variables such as satisfaction or pride of being a farmer. Variables from the social domain are potentially important for system functioning, but often neglected in studies for instance related to land use (Gliessman 2015, Winkler et al. 2018). Participatory research also reveals differences in perceptions of goals and functioning of a system, which is important to take into account when assessing sustainability and resilience (Robards et al. 2011).

Intensive specialized agriculture in the EU

The framework was applied to three intensive, specialized farming systems in the EU through three workshops. Having three farming systems as case studies in different regions allows to demonstrate how the framework works in different conditions. A common denominator, which in our case studies is the intensive and specialized nature of the farming systems, also allows for evaluating the usage of the framework for comparison between farming systems. Applying the proposed framework to intensive, specialized systems in this study is relevant and interesting from different perspectives. Regarding representativeness, the bulk of the food produced within the European Union is provided by medium to highly intensive, specialized farming systems (Andersen et al. 2007). Regarding sustainability, more intensive agricultural systems are associated with higher yields and revenues through economy of scales (Peterson et al. 2018), but also with higher pressure on the environment, certainly per unit of area (Tilman et al. 2002, Pretty and Bharucha 2014). As to resilience, specialized and intensive agricultural systems are often optimized for production under stable socioeconomic and biophysical environments (Urruty et al. 2016). However, many external influences, such as increased occurrence of weather extremes and volatile markets, create a more variable production environment than most specialized agricultural systems are designed for. Farms in such systems often have a relatively high share of financial capital invested in production equipment that cannot be reinvested without making substantial losses (sunk costs). Sunk costs create path-dependency and lock-in of individual farmers, which makes it difficult for the whole agricultural system to adapt and transform if new challenges arrive (Balmann et al. 2006). In addition, the professional network required for providing inputs to and processing outputs from intensive, specialized systems can create a lock-in of stakeholders’ interests inside and outside the agricultural system. Examples may refer to businesses that need to fulfill shareholder expectations regarding economic profit (Westley et al. 2011), or intellectual property rights in technology intensive agriculture (Plumecocq et al. 2018).

Concepts used in this study

In this study we define a farming system as a geographical region with relatively homogenous agro-ecological and social conditions. In the farming system, we start with one farming sector and its farms as the focal point of attention and then include all actors in the farming system who influence the focal farmers and who are themselves influenced by focal farmers (Meuwissen et al. 2019). Our working definition for sustainability of farming systems is an adequate performance of all system functions across the environmental, economic, and social domain (see, e.g., Morris et al. 2011, König et al. 2013). Obviously “adequate” is normative and depends on environmental thresholds and economic and societal constraints and objectives. For resilience, we distinguish three resilience capacities: robustness, adaptability, and transformability. Robustness is the capacity to resist and endure shocks and stresses; adaptability is the capacity to actively respond to shocks and stresses without changing farming system structures and feedback mechanisms; and transformability is the capacity of a system to reorganize its structure and feedback mechanisms (Meuwissen et al. 2019). We consider along with Walker and Salt (2012) that to assess resilience, we need to distinguish between specified and general resilience. Assessing specified resilience relates to the questions of resilience “of what?,” “to what?,” and “for what purpose?” (Carpenter et al. 2001, Quinlan et al. 2016). General resilience relates to a system’s robustness, adaptability, and transformability, regardless the type of challenge or shock. General resilience is mainly assessed by looking at system principles that are presumably conveying resilience. The five principles for general resilience that we use in this paper are tightness of feedbacks, modularity, diversity, openness, and system reserves (Resilience Alliance 2010). Based on these principles, more concrete resilience attributes or indicators are needed to assess the general resilience of a specific system. Cabell and Oelofse (2012), for instance, present 13 resilience indicators for agro-ecosystems that are considered as conveying resilience to the system. Some of these indicators are somewhat linked to either robustness, adaptability, or transformability, although not in a consistent manner, as we will elaborate further in this study by addressing potential contributions of resilience attributes to the three defined resilience capacities. Appendix 1 provides an overview of all important concepts used in this study.


To study the perceptions of farming systems’ current sustainability and resilience, we designed the Framework of Participatory Impact Assessment for Sustainable and Resilient EU Farming systems (FoPIA-SURE-Farm; Paas et al. 2019, Reidsma et al. 2019). FoPIA-SURE-Farm includes elements from five different existing sources. First, for assessing perceived sustainability, it is inspired by the semi-quantitative approach of the Framework for Participatory Impact Assessment (FoPIA; Morris et al. 2011, König et al. 2013). Second, for assessing perceived resilience, elements from the Resilience Assessment Framework are implemented (RAF; Resilience Alliance 2010). Third, for analyzing dynamics of sustainability indicators, we included participatory techniques used for system dynamics modeling by Herrera (2017). Fourth, general resilience was assessed based on a list of resilience attributes as proposed by Cabell and Oelofse (2012), which was tailored and complemented for the assessment of farming systems. Last, FoPIA-SURE-Farm builds on the framework developed by Meuwissen et al. (2019) in the context of the SURE-Farm project ( Meuwissen et al. (2019) propose to investigate farming system resilience by answering the following questions: (a) resilience of what? (defining the farming system); (b) resilience to what? (identifying challenges), (c) resilience for what purpose? (identifying main goods and services delivered by farming systems to society), (d) what resilience capacities? (assessing robustness, adaptability, as well as transformability), (e) what resilience attributes? (identifying system characteristics that convey resilience to the system). These questions facilitate the framing of general and specific research topics related to resilience for which qualitative as well as quantitative research methods can be applied.

Farming systems

Compared to studies at farm level, a study at the farming system level allows to take into account challenges that operate at similar levels of integration (Peterson et al. 2018), such as the decrease of farm numbers and specific climate changes in a region. Also, at the farming system level, processes and actors that can influence system dynamics in the face of challenges, such as stakeholder interaction with the environment (Urruty et al. 2016) and self-organization (Cabell and Oelofse 2012), can be included. Consequently, at this level the system dynamics are not entirely due to external challenges, but include strategies by different actors to adapt and transform (Cumming et al. 2017). At the same time, the farming system is a level above the farm (Giller 2013) at which individual stakeholders can still be heard (Cabell and Oelofse 2012), allowing for multiple stakeholder input in participatory settings.

Case studies

Workshops were held in three specialized farming system case studies (CS): starch potato production in Veenkoloniën, the Netherlands (NL-Starch potato), dairy production in Flanders, Belgium (BE-Dairy), and hazelnut production in Lazio, Italy (IT-Hazelnut). NL-Starch potato is a capital and input-intensive system with relatively low economic productivity per unit of input (Table 1). Predominant soils are sandy with a high amount of inactive organic matter. The most economically productive crop, starch potato, is typically grown with cereals and sugar beets in a narrow 1:2 or 1:3 rotation. On some farms also onions, carrots, or tulips are cultivated. Relative to cereals and sugar beets, most crop protection products are applied to starch potatoes. Farmers are organized in a cooperative that processes the starch potatoes, which are often grown on a contract basis. Sugar beets are also grown on contract and sold to a cooperative. Main challenges in NL-Starch potato are low economic productivity, plant parasitic nematodes in the soil, and changing policies and legislation. In NL-Starch potato, the number of farmers is decreasing and the prices of agricultural land are increasing.

BE-Dairy is also a capital intensive system (Table 1). Livestock diets contain mainly grass (silage), supplemented with maize silage and feed concentrates. Farmers are organized in cooperatives that collect, process, and market the milk and its derived products. Important challenges are competition on export markets and fluctuating prices of milk and feed. Nitrogen surpluses put pressure on the environment. Other debated subjects in this system are the production of greenhouse gasses and the use of antibiotics. Future farm succession is a concern in this farming system because of the capital intensity (Table 1), a decreased interest into farming of the younger generation (relating to a competition with other occupations), and the challenge of administrative and legislative demands.

IT-Hazelnut is the least capital intensive system (Table 1). Use of crop protection products and abstraction of ground-water for irrigation have been claimed to put pressure on the environment, especially on surface waters causing public concern (Liberti 2019). Most of the production is sold to processing facilities outside the farming system. In the region some cooperatives collect the raw product and perform only the first processing step, i.e., shelling, and provide storage services. Main challenges for this system are price instability and competition on the world market, mainly with Turkey. Recent modernization of harvesting through auto-propelled machines has increased labor productivity and instigated a demand for more land for hazelnut cultivation. As a result, land that is less suitable for hazelnut cultivation will be taken into production in the coming years. Results from FoPIA-SURE-Farm in IT-Hazelnut are presented in detail in Nera et al. (2020). Compared to Nera et al. (2020), this study puts more emphasis on presenting and evaluating the methodology when applied to three different farming systems. Nera et al. (2020) evaluate sustainability and resilience in the case study in more detail.

Stakeholder participation

Multiple actors influence the dynamics of a farming system. The heterogeneity of actors included in the systems has hence been accounted for in the workshops. Participants were invited via existing stakeholder networks in the case study areas. Attendance to the workshop was based on participants’ own initiative and therefore not necessarily balanced across stakeholder groups. Participants mainly consisted of farmers and representatives from the government, NGOs, research institutes and the processing industry (Table 2). Participant numbers varied across case studies (from one to eight per stakeholder group, and 12 to 21 in total).

To bring all participants to the same level of analysis, the research team, which differed in each case study, started the workshop by presenting the social delineation of the farming system (Table 3), showing farming system actors and (in)direct influencers of the farming system. Participants were given the opportunity to react to the farming system representation. Updates were made if necessary.

Assessing sustainability of farming system functions

Eight functions of the farming system, along with their representative indicators, all identified by the research team, were presented to the participants and discussed in a plenary session (Table 3). If necessary, changes were made to the list of representative indicators (Table 4). In BE-Dairy, the function “Bio-based resources” was interpreted broadly as all edible products from the system other than milk. In IT-Hazelnut, “Animal health and welfare” was not assessed because animals are not part of this farming system. Participants were invited to individually assess the importance of the different functions (see Table 3 for details). Similarly, they were asked to assess all indicators regarding their degree of representativeness for the function they were to represent. The outcomes for the degree of representativeness were transformed to relative importance in order to compare importance of indicators across functions (Equation A2.1). Furthermore, the performance of each indicator was assessed by each participant. Function performance was calculated per participant as the sum of scores of indicators per function times the average indicator representativeness according to the stakeholder group to which the participant belonged to (Equation A2.2). Importance and performance of functions and indicators was directly fed back and discussed with the participants during the workshop. Perceived indicator and function performance was interpreted as being indicative for perceived sustainability levels of the system (see, e.g., Morris et al. 2011, König et al. 2013 who applied this method for assessing policy impact on sustainable development). Perceived importance and performance levels of functions between farmers and non-farmers were tested for significant differences, using a Kruskal Wallis test in R (R Core Team 2015). Perceived function importance and performance across functions were tested for significant differences using a Kruskal Wallis test and a post-hoc Conover Iman test with Bonferroni correction using the R-package “conover.test” (Dinno 2017).

Assessing resilience

Based on indicator importance and performance, participants decided in a plenary session which 3 to 4 indicators were most interesting to assess in further detail with regard to farming system resilience. Participants were invited to sketch the yearly dynamics of the selected indicators over the time span 2000–2018, and to identify challenges that induced the sketched dynamics. Also they were invited to identify strategies that have been applied by farmers and other farming system actors to deal with the identified challenges. Identified strategies were assessed for their level of implementation and for their contribution to resilience capacities. Results were directly fed back to participants in a plenary setting. In the evaluation phase, strategies were linked to resilience attributes in order to visualize the connection between specified and general resilience and to allow for comparability of strategies between case studies.

To assess general resilience, a list with resilience attributes was constructed (Table 5) based on Cabell and Oelofse (2012) and Meuwissen et al. (2019) to serve the purpose of this study within the context of the SURE-Farm project. This implied that details on the farming system were added in the resilience attribute description. We also split up certain attributes as provided by Cabell and Oelofse (2012). Finally, we arrived at a list with 22 resilience attributes from which we selected 13 to study in the workshop to not overtask the participants. This selection was based on the (1) SURE-Farm research focus, (2) paying equal attention to the different resilience principles, and (3) avoiding overlap between attributes. For instance, “Reflective and shared learning” is partly dependent on “Social self-organization,” which is why we selected the latter as the overarching attribute. We use the term “resilience attribute,” which refers to a higher hierarchical level compared to the term “resilience indicator” as was originally used in Cabell and Oelofse (2012). This distinction also helps to avoid confusion with the concept of “function indicators” as used in this study. Resilience attributes were assessed for their level of presence and for their contribution to resilience capacities. Because of time limitations, results could only be fed back into a limited extent during the workshop. Perceived contribution of resilience attributes to resilience capacities across attributes was tested for significant differences using a Kruskal Wallis test and a post-hoc Conover Iman test with Bonferroni correction in R using the “conover.test” package (Dinno 2017).


Farming system actors

Participants provided feedback on the social delineation of all three farming systems (Fig. 1). For BE-Dairy and NL-Starch potato, many farming system actors and influencers were identified, while these were much fewer in IT-Hazelnut. In NL-Starch potato, the cooperative for processing starch potatoes was seen as part of the farming system. In BE-Dairy, the cooperative for processing and distributing milk was moved inside the farming system, after feedback from participants. In IT-Hazelnut, cooperatives existed within the farming system, but main processors of hazelnut were considered to be outside the farming system, because they operate on the international market and are not directly affected by changes within the considered farming system. Local NGOs were mentioned in BE-Dairy and NL-Starch potato, but not in IT-Hazelnut.

System sustainability

In all three case studies, “Food production” and “Economic viability” were considered to be among the most important functions. In NL-Starch potato and BE-Dairy, “Maintaining natural resources” was also considered as important (Fig. 2). In NL-Starch potato, the function “Food production” was evenly represented by the three main crops: starch potato, sugar beet, and wheat (Table 4). In IT-Hazelnut, this function was represented by hazelnut quantity and quality. In BE-Dairy, this function was represented by milk production and the price consumers pay for milk in the supermarket. Representative indicators for “Economic viability” related to farm income and profit per hectare. Representative indicators for “Maintaining natural resources” related to soil and water quality.

Performance of “Food production” and “Economic viability” was considered high in IT-Hazelnut and moderate in BE-Dairy (Fig. 1). For “Natural resources,” participants in BE-Dairy perceived a moderate to high performance, while participants in IT-Hazelnut and NL-Starch potato perceived a moderate performance. Participants in NL starch potato indicated that they found it challenging to assess individual functions, because they perceived functions as interacting with each other. Participants in IT-Hazelnut indicated that they perceived that the recent modernization and enlargement of the sector increased pressure on the environment, while neglecting the importance of the function of “Natural resources.”

In all case studies, it seemed that farmers perceived “Economic viability” as more important and “Natural resources” as less important compared to other stakeholders. However, only for “Natural resources” in BE-Dairy, a significant difference was detected. Farmers generally allocated less importance to functions that deliver public goods. At the same time, in BE-Dairy and NL-Starch potato, farmers tended to assess performance of “Natural resources” on average higher than other stakeholders. In BE-Dairy, participants indicated that these results would have been different if more people from nature organizations had participated in the workshop. In NL-Starch potato, farmers also assessed the performance of “Biodiversity & habitat” higher than other stakeholders (Appendix 3).

Dynamics of sustainability indicators

In all case studies, participants indicated that they had little knowledge on year-to-year fluctuations of selected indicators. However, they were able to indicate trends and important years with regard to changes in trends, lows, and peaks. For each selected indicator, participants identified main challenges and strategies applied to maintain or improve the indicator performance (Table 4). According to participants, important underlying causes for dynamics in NL-Starch potato were nematode pressure and increased costs per hectare that were amongst others counterbalanced by improved potato varieties that resist nematodes, reducing costs and increasing efficiency. The change from production-based to area-based subsidies in 2013 was seen as a big challenge in NL-Starch potato. As strategy to deal with this challenge, the cooperative decided to abandon less lucrative lower quality starch markets and invest in product innovation. In BE-Dairy, participants indicated that main indicators were particularly affected by the abolishment of the milk quota and events that affected the international market. Identified strategies to cope with these challenges were mainly related to improved efficiency, but also included strategies related to risk management and diversification at farm and farming system level. In IT-Hazelnut, dynamics of indicators were perceived to be mainly influenced by the development of new machinery that enhanced labor productivity. Apart from competition at the international market with Turkey, no major challenges were reported. Strategies to maintain or improve main indicators in IT-Hazelnut were related to mechanization, cooperatives, producer groups, and using funds of the Rural Development Program of the European Commission (RDP funds).

In all three case studies, indicators related to food production and economic viability were assessed to have improved over time, except in BE-Dairy, where farm income was perceived to be on average stable, but with increased yearly variation. The perspective in all three case studies is more negative for indicators representing the environmental domain. In NL-Starch potato, soil quality was perceived to decline, and in BE-Dairy, emissions of carbon were perceived to go up again. In IT-Hazelnut, the indicator “Area of organic hazelnut production” was seen as positively related to the function “Biodiversity & habitat” and was perceived to be increasing. However, participants indicated that they perceived that biodiversity in the farming system was generally decreasing because of the limited habitat provision in the expanding area with hazelnut monocultures. In the expansion areas, hazelnut cultivation also requires more ground water extraction compared to the cultivations of chestnut and olives that it usually replaces.

Resilience strategies for the farming system

Overall, perceptions on implementation levels of strategies was scored most positive in IT-Hazelnut (Fig. 3A-C). In IT-Hazelnut, the strategies of establishing cooperatives and starting new value chain activities were perceived to be least well implemented. In BE-Dairy and NL-Starch potato implementation levels of different strategies were scored poor to good. In BE-Dairy, strategies related to the carbon footprint were less well implemented than for the indicators “real milk price” and “labor income”. In NL-Starch potato, strategies related to soil quality were less well implemented than for profit per hectare. (Figure 3A-C)

Strategies (Table 6) could mostly be related to the resilience attributes “Reasonably profitable,” “Infrastructure for innovation,” “Production coupled with local and natural capital,” “Socially self-organized,” and “Functional diversity.” Strategies linked to “Socially self-organized” were perceived to be well implemented in all case studies. Strategies related to innovation were perceived to be very well implemented in IT-Hazelnut, and moderately in the other case studies. Only in BE-Dairy and NL-Starch potato could several strategies be evaluated as contributing to functional diversity and the coupling of production with local and natural capital. In the case studies there were no strategies identified that could be linked to redundancy of farms or to policies (Appendix 4).

In general, perceived contribution of strategies to robustness was moderate. Contribution to adaptability was generally equal or lower, with a few exceptions. Contribution to transformability was considered positive as well as negative. When positive, contribution was equal or lower than moderate, with a few exceptions. When negative, contribution was considered weak and sometimes moderate. (Figure 3D-F)

In IT-Hazelnut, the perceived contribution of RDP funds was controversial, where participants scored negative as well as positive. Participants indicated that the RDP funds were de facto used as subsidies by farmers, without changing farming practices in the long term. Dependent on the indicator that was considered, mechanization was sometimes seen as being negative for transformability, but overall positive. Cooperatives, producer organizations, and value chain activities were all perceived to contribute to farming system resilience. In BE-Dairy, all strategies were perceived to contribute positively to robustness, and to a lesser extent to adaptability and transformability. Moreover, interventions from outside this farming system, such as exceptional financial support from the sectoral federation (Fedis-support), genetic improvement, and creation of milk powder stocks, as well as strategies that require on-farm investments, were perceived to negatively affect transformability. In NL-Starch potato, many strategies were perceived to contribute positively to robustness and adaptability. There were four strategies that were evaluated to weakly affect transformability in a negative way: scaling (of area and hence the production), increase value of starch products, have land available outside contract farming, apply precision agriculture. In NL-Starch potato, strategies related to soil quality and potato production were perceived to be good for transformability. With regard to strategies related to profit per hectare, only cost reduction, better varieties, and improved knowledge on soil and varieties were perceived to be good for transformability. However, strategies that require investments from mainly within the farming system, such as scaling, increased value of starch products and adopting precision agriculture were regarded as negatively affecting transformability.

Resilience attributes

As shown in Figure 4A, perceptions about presence of most resilience attributes followed similar patterns in the three case studies. Presence of resilience attributes was perceived to be low to moderate, with some exceptions, especially in IT-Hazelnut. For example, BE-Dairy and NL-Starch potato scored “Reasonably profitable” as low, while this attribute scored high in IT-Hazelnut, providing it with financial capital reserves. “Socially self-organized” scored high in IT-Hazelnut and moderate in other case studies, which has an effect on the social capital reserves. In IT-Hazelnut, the coupling of production to local and natural capital was perceived low, while it was perceived to score moderately in the two other case studies. Perceptions on legislation being coupled to local and natural capital were low in all three case studies, especially in NL-Starch potato. The studied farming systems were perceived to have a poor to moderate degree of openness. “Exposed to disturbance” was especially evaluated lower, as participants perceived that disturbances are threatening system functioning instead of making a system more resilient. Diversity was evaluated to be poorly present in all case studies, with the exception of “Spatial and temporal heterogeneity of farm types”, which was assessed to be moderately present. Modularity in the farming system was perceived to be moderately present in IT-Hazelnut and weakly present in BE-Dairy and NL-Starch potato.

Figure 4B reveals that the potential contribution of resilience attributes to robustness was perceived to be very weakly to moderately positive in all three case studies. High scoring attributes in all three case studies related to the profitability of the system and its production being coupled with local and natural resources. Being exposed to disturbance was evaluated negatively as well as positively by stakeholders in BE-Dairy and NL-Starch potato, respectively, explaining the low overall score. Scores for robustness were specifically low for some resilience attributes in NL-Starch potato, namely “Exposed to disturbance,” “Optimally redundant (farms),” “Supports rural life,” and “Legislation coupled with local and natural capital.”

Scores for contribution of resilience attributes to adaptability were similar or lower compared to contributions to robustness (Figures 4B and 4C). An exception was “Infrastructure for innovation,” which received similar scores for adaptability and for robustness in all three case studies. Other resilience attributes scoring relatively high in all three case studies related to the profitability, production being coupled with local and natural capital, response diversity, and diverse policies.

Finally, contribution of resilience attributes to transformability was assessed to be very weak to moderate (Figure 4D). Scoring patterns for transformability deviated from the patterns as observed for robustness and adaptability. In BE-Dairy, expectations were lower for resilience attributes contributing to transformability than to adaptability. In all three case studies, “Infrastructure for innovation” got relative high scores for contributing to transformability compared to other resilience attributes.


Farming system sustainability assessment

The studied intensive, specialized farming systems are perceived to primarily provide economic viability, to provide food and, in NL-Starch potato and BE-Dairy, to maintain natural resources. Farmers (participating in the workshops) put more emphasis on economic viability compared to other participants, who divided importance more equally over farming system functions. This difference in perception indicates that no optimal solution exists across all stakeholder groups and that a balance between differing views needs to be found (in line with Robards et al. 2011). Detecting the difference between stakeholder groups was possible because participants assessed the same set of functions. The top-down approach of our method, dividing 100 points over eight functions, forced participants to make choices between economic, environmental, and social functions and indicators. In our case studies, this revealed a lack of attention for social functions such as quality of life and attractiveness of the area, which might result in missing out on important feedbacks from the social domain. The identification of this knowledge gap is an important result of our participatory method. Mosse (1994) points out that the identification of the boundaries of local knowledge is an important, but often overlooked, goal of participatory research. Performance of functions was generally perceived to be moderate, with a few low and well performing functions in each case study. Contrary to perceptions on function importance, perceptions on function performance were similar between the stakeholder groups in most case studies. The remarkable variety in allocated importance of functions, together with an only moderate performance of more important functions, suggests the presence of interactions and trade-offs between functions. Further indications of trade-offs were found in the studied farming systems (Appendix 5). Existence of trade-offs may influence stakeholders’ perceptions, which emphasizes the importance to have both information on perceptions of stakeholders as well as observational data.

Farming system resilience assessment

Based on perceived presence and potential contribution of resilience attributes, the case studies were perceived to show more robustness than transformability, which is typical for specialized systems aiming to control external factors as much as possible (Hoekstra et al. 2018). Strategies applied in the past 20 years, mainly with regard to economic functions and food production, show that the studied farming systems mainly use their adaptability to increase robustness, e.g., increasing farm size in BE-Dairy and NL-Starch potato to better cope with small margins. It should be noted that regarding the strategies, the method is biased toward the interest of stakeholders for specific functions. This interest may also have resonated within the stakeholders’ mind when reflecting on resilience attributes.

All farming systems were perceived to have a relative low presence of functional and response diversity, and contribution of diversity to resilience was perceived low. According to Hoekstra et al. (2018), the lack of diversity is an indication that production systems are operating more under a control rationale rather than a resilience rationale. Hoekstra et al. (2018) pose that for systems to optimally perform, a balance needs to be found between the control and resilience rationales. In the case studies, this balance between rationales might be partly found in the spatial heterogeneity of farm types, which relates to diversity (see also Reidsma and Ewert 2008). Heterogeneity of farm types is assessed to have relatively high presence, and especially in BE-Dairy and IT-Hazelnut this resilience attribute was perceived to contribute to resilience. Common building blocks for resilience in all case studies were profitability, production coupled with local and natural capital, social self-organization, and infrastructure for innovation. Profitability was perceived as having a large potential for improving robustness and adaptability in all case studies, but was currently perceived to be low in BE-Dairy and NL-Starch potato. Higher profitability was perceived to mainly increase robustness and adaptability in BE-Dairy. To attain higher profitability in BE-Dairy, many strategies in the past required large investments, which can explain the perceived negative contribution of these strategies to transformability. Production being coupled with local and natural capital is also assessed to have a large contribution to resilience in all case studies, but was currently considered to be low in IT-Hazelnut and moderate in BE-Dairy and NL-Starch potato. Loss of natural capital such as loss of ecosystem quality might be more visible in the quickly intensifying and expanding IT-Hazelnut (Biasi and Botti 2010). This could explain why the other case studies, which actually have more intensive systems, scored higher. Self-organization is commonly accepted as enhancing resilience (Cabell and Oelofse 2012). However, too much connections between actors in a system can increase the risk for co-dependency and reduce modularity. This could be the case in NL-Starch potato where the transformative capacity of the local cooperative has provided a pathway toward higher profitability. However, to stay on this pathway, the cooperative maintains a high demand of starch potatoes, resulting in a very narrow rotation and an increased pressure from nematodes in the soil. Our study indicates that infrastructure for innovation is an important resilience attribute for specialized farming systems, especially for adaptability and transformability, and should receive more emphasis. Although Gunderson and Holling (2002) emphasized the importance of innovation for resilience, resilience literature often lays more emphasis on social and ecological aspects (e.g., Cabell and Oelofse 2012).

Assessing perceived sustainability and resilience allows to reflect on both concepts simultaneously. In the studied farming systems, function indicators relating to sustainability were on average perceived to perform moderately. This suggests that adaptations or even transformations need to be realized. Without those, sustainability might further decline, especially social sustainability that currently seems to receive relatively little attention compared to economic and environmental sustainability. On the one hand further decline could lead to undesired transformational change. On the other hand current perceived lower levels of adaptability and transformability do not seem to allow orchestrated transformations by stakeholders in the farming system: apart from infrastructure for innovation, no other attribute was assessed to support transformability well in any of the three farming systems. A promising resilience attribute for transformability is related to an enabling environment for shared learning and experimentation and should be included in further assessments (Appendix 6).


The FoPIA-SURE-Farm framework employed in this research captures essential steps to assess farming system resilience (see Meuwissen et al. 2019). The framework encompasses multiple dimensions and perspectives by including lists with many farming system functions and resilience attributes. These lists form a base for knowledge exchange between researchers and participants. Researchers exposed participants to the notion that sustainability and resilience need to be addressed in a structured, integrated approach. Based on this approach, participants could add local meaning to the still abstract system functions and resilience attributes. Adding local meaning enabled further discussions about sustainability and resilience. For instance, for both researchers and participants, it was informative that most important system parameters, according to participants’ perceptions, could be identified and directly discussed. This allowed for identification of important system dynamics and also strategies that were aimed to maintain or enhance system functions. Selection of important system parameters can also support further analyses on systems dynamics. For instance, by zooming in on a specific set of parameters, Kinzig et al. (2006) were able to study critical thresholds of agro-ecological systems. It should be noted, however, that by first identifying main sustainability indicators, a path-dependency is created, which in the application to the three case studies resulted in an emphasis on economic and production indicators in later steps of the workshop. In general, the selection of main system parameters is also a further simplification of reality. This increases the risk of not being able to understand the dynamics of the studied system (Quinlan et al. 2016). Related to adequately understanding farming system behavior, it should be noted that other levels than the farming system should be taken into account as well when studying multi-level concepts such as sustainability (e.g., Van Passel and Meul 2012, Delmotte et al. 2017) and resilience (e.g., Peterson et al. 2018, Meuwissen et al. 2019). In that sense FoPIA-SURE-Farm needs to be complemented with analyses at farm level (Spiegel et al. 2019) and the level beyond the farming system (Feindt et al. 2019). The level beyond the farming system is not well defined in agricultural literature because there are multiple possibilities, e.g., entire value chains, food systems, and the political or socio-technical environment.

The selected case studies were different in geographical size and had different positions in governmental hierarchy. On the one hand this shows the wide applicability of FoPIA-SURE-Farm to handle various farming systems with different geographical and political boundaries. On the other hand it made the comparison between the three case studies in this paper challenging, i.e., differences observed could be confounded with size and hierarchical level of the case study. Differences could also be confounded with the influence of the individual research teams in each case study. Also culturally defined inclinations, e.g., toward optimism or pessimism, could play a role. To deal with these challenges in this paper, we did not compare scores between case studies directly, e.g., stating that a function was performing better in one case study compared to another, or by performing statistical tests on differences between case studies. Instead, we treated scores as being relative to other scores within the same case study. This revealed certain patterns such as the relative absence of attention for social functions, the emphasis on robustness, and the relative importance of the resilience attributes related to profitability, social self-organization, and infrastructure for innovation in all case studies.

Summarizing weighted indicator scores into sustainability indices is a common practice in sustainability science (Mayer 2008), and for instance applied in the original FoPIA-approach (Morris et al. 2011, König et al. 2013). However, letting participants divide 100 points over functions and indicators is less commonly applied. We argue that this method helps to raise awareness of trade-offs between functions, in case such trade-offs are until then only implicit or part of a subconscious process. Dividing 100 points over functions and indicators resembles the Q-methodology (McKeown and Thomas 2013;, in which participants are forced to allocate scores to a number of items, while following a predefined distribution in which extreme values are more rare than moderate values. In the approach taken in this study, the participants themselves effectively determine their own distribution. This means that an observed imbalance between importance of functions or indicators can be interpreted as an outcome of the study rather than a design input as is the case for the Q-methodology.

Using Likert items and scales poses another challenge for interpreting the results from FoPIA-SURE-Farm. In this paper, the performance of functions can be seen as Likert scales, where the representative indicators are weighted Likert items, allowing for presenting the mean as summarizing statistic (Guerra et al. 2016). Regarding the scoring of strategies and resilience attributes, we chose to present the mean as well, which is not entirely correct according to some, but acceptable to others (e.g., Norman 2010) and more intuitive compared to using the median and quartiles in communicating results (Guerra et al. 2016). This is especially true for the strategies where the number of observations is low. Appendix 3 provides means as well as individual observations, which shows that means and medians, basically the observations in the middle of the data, do not differ much. Moreover, for testing significant differences, non-parametric tests were used that correspond with the ordinal nature of the data. It can be argued that, although participants may have different points of reference regarding whether a function for instance is performing poorly or perfectly, perceptions of performance between stakeholders and across indicators can be compared (e.g., Morris et al. 2011, König et al. 2013). The possibility that participants have avoided extreme values on the provided scoring scales, might be reflected in the moderate scorings that many functions and resilience attributes received. This is an additional reason to look at differences between scores and consequently focus on the patterns of higher and lower scoring items, as is done in this study. This is somewhat similar to the analyses on the outcomes of Q-methodology, where patterns of extreme values for a set of specific items can be interpreted as being expressions of mental models of stakeholders in a system (McKeown and Thomas 2013). A final point of attention relates to the notion that negative and positive values are for various reasons not true opposites of each other, possibly leading to a method bias (Alexandrov 2010). In our study, all Likert-type items were phrased in a positive way, thus reducing the impact of a possible method effect (Alexandrov 2010). Practically, this reduces the likelihood that, for instance, the positive scoring of resilience attributes is partly a methodological artefact.

Another point of discussion is that participants showed signs of fatigue toward the end of the FoPIA-SURE-Farm workshop. This coincided with the intellectually challenging scoring exercise on presence of resilience attributes and their contribution to robustness, adaptability, and transformability. Still, this exercise was completed correctly in six out of eight other European case studies (Paas et al. 2019, Reidsma et al. 2020). Dependent on the research question, scoring on presence and contribution could be combined into a score that summarizes the overall importance of the resilience attribute for the farming system. Filling out forms was especially experienced as tedious. Moreover this method was top-down, which is advised to be avoided in participatory approaches that deal with the topic of resilience (Callo-Concha and Ewert 2014). In FoPIA-SURE-Farm we could not avoid top-down questions in order to save time of participants, to enhance comparability between case studies, and to identify knowledge gaps. However, we stimulated participants to influence the content of the workshop, e.g., by providing feedback during the plenary and small group discussions that alternated the individual exercises. To further compensate for top-down questions, and to make sure that the right issues are addressed, stakeholders should be consulted again when main indicators of the farming system as identified in this study are used for further analyses, for instance, when moving to the planning phase of a project cycle (Ridder and Pahl-Wostl 2005) where concrete strategies for improved sustainability and resilience have been identified (Paas et al. 2020).

With the FoPIA-SURE-Farm framework, the underlying system mechanisms that bring current resilience were only revealed occasionally in plenary discussions, rather than being a fundamental part of the framework. For instance, interactions between resilience attributes through competition for resources or co-dependence was not addressed. To complete the resilience assessment and understand underlying mechanisms, further research is necessary that includes the impact of (new) shocks, adaptation measures, and future scenarios (Walker et al. 2002). For that reason we continued within the SURE-Farm project with participatory integrated assessments in which we assess performance, interactions, and thresholds of important system parameters in different possible futures (Paas et al. 2020).


The framework presented in this paper is based on existing sustainability and resilience frameworks. It provides a method to identify main indicators of a farming system and to obtain a qualitative assessment of its perceived sustainability and resilience, based on opinions of stakeholders from that system. This reveals stakeholder perspectives on importance and performance of functions and resilience attributes accounting for the complex nature of farming systems. Perspectives on importance were sometimes imbalanced, i.e., too little importance was allocated to social and environmental functions. Also attention to resilience attributes was imbalanced. The identification of imbalance is an important outcome of the method because it indicates the boundaries of local perspectives and knowledge. Perspectives on performance were sometimes deviating from findings presented in literature, which emphasizes the need to have input from quantitative analytical sources as well.

Assessing perceived sustainability and resilience simultaneously allows to reflect on pathways to higher sustainability. Taking the case of specialized systems in the EU, workshop outcomes suggest that function performance relating to sustainability was perceived to be moderate, while presence of resilience attributes was perceived to be low to moderate and contribution of these attributes to resilience was perceived to be weak to moderate. In the studied systems, robustness was perceived to be stronger than adaptability and transformability. This indicates that finding pathways to more sustainability, which requires adaptability and transformability, will be a challenging process.

Strategies to maintain performance of indicators of the studied systems were mainly related to keeping the system economically viable, partly through innovations in the system. Across case studies, profitability, production coupled with local and natural capital, infrastructure for innovation, and self-organization were perceived as important resilience attributes. Based on workshop results, we conclude that an additional resilience attribute related to an enabling environment for experimentation and learning is necessary. The relative importance and contribution of some resilience attributes in the studied systems differed from case to case, e.g., heterogeneity of farm types. This indicates that the local context in general, and stakeholder perspectives in particular, are important when evaluating general resilience and policy options based on resilience attributes. Overall, despite some methodological limitations, the case study specific results seem a good starting point for raising awareness, further assessments, and eventually for developing a shared vision and action plan for improving sustainability and resilience of a farming system.


Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.


We acknowledge the SURE-Farm Horizon 2020 project (Grant Agreement No 727520) for funding this study. We are thankful to all workshop participants, whose input was essential for this study. We are thankful to ILVO, UNITUS, and Experimental Farm Valthermond (WUR) in collaboration with Innovatie Veenkolonien for providing meeting facilities and catering in the Belgian, Italian, and Dutch case study, respectively. We are thankful for assistance to Erwin Wauters, Jo Bijttebier, and Jorrit Becking in the Belgian case study, Federico Antonioli, Giulio Paolini, Saverio Senni, and Elena Nera in the Italian case study, and Alisa Spiegel, Inge van der Wiel, Yannick Buitenhuis, and Bert Rijk in the Dutch case study. We are thankful to two anonymous reviewers and colleagues in the Plant Production Systems Group of Wageningen University for providing feedback on this work.


This human subject research in the context of the SURE-Farm project has been approved by the Social Sciences Ethics Committee of Wageningen University & Research. The data/code that support the findings of this study are available on request from the corresponding author, [WP]. The data/code are not publicly available because they contain information that could compromise the privacy of research participants.


Alexandrov, A. 2010. Characteristics of single-item measures in likert scale format. Electronic Journal of Business Research Methods 8(1).

Alkan Olsson, J., C. Bockstaller, L. M. Stapleton, F. Ewert, R. Knapen, O. Therond, G. Geniaux, S. Bellon, T. P. Correira, N. Turpin, and I. Bezlepkina. 2009. A goal oriented indicator framework to support integrated assessment of new policies for agri-environmental systems. Environmental Science and Policy 12:562-572.

Anderies, J. M., C. Folke, B. Walker, and E. Ostrom. 2013. Aligning key concepts for global change policy: robustness, resilience, and sustainability. Ecology and Society 18(2):8.

Andersen, E., B. Elbersen, F. Godeschalk, and D. Verhoog. 2007. Farm management indicators and farm typologies as a basis for assessments in a changing policy environment. Journal of Environmental Management 82(3):353-362.

Arodudu, O., K. Helming, H. Wiggering, and A. Voinov. 2017. Towards a more holistic sustainability assessment framework for agro-bioenergy systems — a review. Environmental Impact Assessment Review 62:61-75.

Ashkenazy, A., T. Calvāo Chebach, K. Knickel, S. Peter, B. Horowitz, and R. Offenbach. 2018. Operationalising resilience in farms and rural regions - findings from fourteen case studies. Journal of Rural Studies 59:211-221.

Balmann, A., K. Dautzenberg, K. Happe, and K. Kellermann. 2006. On the dynamics of structural change in agriculture: internal frictions, policy threats and vertical integration. Outlook on Agriculture 35(2):115-121.

Biasi, R., and F. Botti. 2010. Hazelnut landscape transformation in the northern Latium: the study case of the Monti Cimini’s hills. Corylus & Co. 2(1):39-48. [online] URL:

Cabell, J. F., and M. Oelofse. 2012. An indicator framework for assessing agroecosystem resilience. Ecology and Society 17(1):18.

Callo-Concha, D., and F. Ewert. 2014. Using the concepts of resilience, vulnerability and adaptability for the assessment and analysis of agricultural systems. Change and Adaptation in Socio-Ecological Systems 1:1-11.

Carpenter, S., B. Walker, J. M. Anderies, and N. Abel. 2001. From metaphor to measurement: resilience of what to what? Ecosystems 4(8):765-781.

Cumming, G. S., T. H. Morrison, and T. P. Hughes. 2017. New directions for understanding the spatial resilience of social-ecological systems. Ecosystems 20(4):649-664.

Cumming, G. S., and G. D. Peterson. 2017. Unifying research on social-ecological resilience and collapse. Trends in Ecology and Evolution 32(9):695-713.

Darnhofer, I. 2010. Strategies of family farms to strengthen their resilience. Environmental Policy and Governance 20(4):212-222.

Darnhofer, I. 2014. Resilience and why it matters for farm management. European Review of Agricultural Economics 41(3):461-484.

Delmotte, S., V. Couderc, J.-C. Mouret, S. Lopez-Ridaura, J.-M. Barbier, and L. Hossard. 2017. From stakeholders narratives to modelling plausible future agricultural systems. Integrated assessment of scenarios for Camargue, Southern France. European Journal of Agronomy 82(B):292-307.

Delmotte, S., S. Lopez-Ridaura, J. M. Barbier, and J. Wery. 2013. Prospective and participatory integrated assessment of agricultural systems from farm to regional scales: comparison of three modeling approaches. Journal of Environmental Management 129:493-502.

Dinno, A. 2017. conover.test: Conover-Iman test of multiple comparisons using rank sums. R package version 1.1.5.

Douxchamps, S., L. Debevec, M. Giordano, and J. Barron. 2017. Monitoring and evaluation of climate resilience for agricultural development - a review of currently available tools. World Development Perspectives 5:10-23.

Feindt, P. H., C. J. A. M. Termeer, J. Candel, and Y. Buitenhuis. 2019. D4.2 Assessing how policies enable or constrain the resilience of farming systems in the European Union: case study results. Sustainable and resilient EU farming systems (SureFarm) project report, EU Horizon 2020 Grant Agreement No. 727520. SureFarm, UK.

Folke, C., S. R. Carpenter, B. Walker, M. Scheffer, T. Chapin, and J. Rockström. 2010. Resilience thinking: integrating resilience, adaptability and transformability. Ecology and Society 15(4):20.

Ge, L., N. P. R. Anten, I. D. E. van Dixhoorn, P. H. Feindt, K. Kramer, R. Leemans, M. P. M. Meuwissen, H. Spoolder, and W. Sukkel. 2016. Why we need resilience thinking to meet societal challenges in bio-based production systems. Current Opinion in Environmental Sustainability 23:17-27.

Giller, K. E. 2013. Can we define the term ‘farming systems’? A question of scale. Outlook on Agriculture 42(3):149-153.

Gliessman, S. R. 2015. Agroecology: the ecology of sustainable food systems. Third edition. CRC Press, Taylor & Francis Group, Boca Raton, Florida, USA.

Godfray, H. C. J. 2015. The debate over sustainable intensification. Food Security 7:199-208.

Guerra, A. L., T. Gidel, and E. Vezzetti. 2016. Toward a common procedure using likert and likert-type scales in small groups comparative design observations. Pages 23-32 in D. Marjanović, M. Štorga, N. Pavković, N. Bojčetić, S. Škec, editors. Proceedings of the DESIGN 2016 14th International Design Conference. The Design Society, Glasgow, UK.

Gunderson, L. H., and C. S. Holling. 2002. Panarchy: understanding transformations in human and natural systems. Island Press, Washington, D.C., USA.

Herrera, H. 2017. Resilience for whom? The problem structuring process of the resilience analysis. Sustainability 9(7):1196.

Hoekstra, A. Y., R. Bredenhoff-Bijlsma, and M. S. Krol. 2018. The control versus resilience rationale for managing systems under uncertainty. Environmental Research Letters 13(10).

Kinzig, A. P., P. Ryan, M. Etienne, H. Allison, T. Elmqvist, and B. H. Walker. 2006. Resilience and regime shifts: assessing cascading effects. Ecology and Society 11(1):20.

König, H. J., S. Uthes, J. Schuler, L. Zhen, S. Purushothaman, U. Suarma, M. Sghaier, S. Makokha, K. Helming, S. Sieber, L. Chen, F. Brouwer, J. Morris, and H. Wiggering. 2013. Regional impact assessment of land use scenarios in developing countries using the FoPIA approach: findings from five case studies. Journal of Environmental Management 127:S56-S64.

Liberti, S. 2019. Il gusto amaro delle nocciole. Internazionale 1312:39-47.

Marchese, D., E. Reynolds, M. E. Bates, H. Morgan, S. S. Clark, and I. Linkov. 2018. Resilience and sustainability: similarities and differences in environmental management applications. Science of The Total Environment 613-614:1275-1283.

Mayer, A. L. 2008. Strengths and weaknesses of common sustainability indices for multidimensional systems. Environment International 34(2):277-91.

McKeown, B., and D. Thomas. 2013. Q Methodology. SAGE, Thousand Oaks, California, USA.

Meuwissen, M. P. M., P. H. Feindt, P. Midmore, E. Wauters, R. Finger, F. Appel, A. Spiegel, E. Mathijs, K. J. A. M. Termeer, A. Balmann, Y. De Mey, and P. Reidsma. 2020. The struggle of farming systems in Europe: looking for explanations through the lens of resilience. Eurochoices 19(2):4-11.

Meuwissen, M. P. M., P. H. Feindt, A. Spiegel, C. J. A. M. Termeer, E. Mathijs, Y. de Mey, R. Finger, A. Balmann, E. Wauters, J. Urquhart, M. Vigani, K. Zawalinska, H. Herrera, P. Nicholas-Davies, H. Hansson, W. Paas, T. Slijper, I. Coopmans, W. Vroege, A. Ciechomska, F. Accatino, B. Kopainsky, P. M. Poortvliet, J. Candel, D. Maye, S. Severini, S. Senni, B. Soriano, C. J. Lagerkvist, M. Peneva, C. Gavrilescu, and P. Reidsma. 2019. A framework to assess the resilience of farming systems. Agricultural Systems 176:102656.

Morris, J. B., V. Tassone, R. de Groot, M. Camilleri, and S. Moncada. 2011. A framework for participatory impact assessment: involving stakeholders in European policy making: a case study of land use change in Malta. Ecology and Society 16(1):12.

Mosse, D. 1994. Authority, gender and knowledge: theoretical reflections on the practice of participatory rural appraisal. Development and Change 25(3):497-526.

Nera, E., W. Paas, P. Reidsma, G. Paolini, F. Antonioli, and S. Severini. 2020. Assessing the resilience and sustainability of a hazelnut farming system in central Italy with a participatory approach. Sustainability 12(1):343.

Norman, G. 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education 15:625-632.

Paas, W., F. Accatino, F. Antonioli, F. Appel, I. Bardaji, I. Coopmans, P. Courtney, C. Gavrilescu, F. Heinrich, V. Krupin, G. Manevska-Tasevska, D. Neumeister, M. Peneva, J. Rommel, S. Severini, B. Soriano, M. Tudor, J. Urquhart, E. Wauters, K. Zawalinska, M. Meuwissen, and P. Reidsma. 2019. D5.2 Participatory impact assessment of sustainability and resilience of EU farming systems. Sustainable resilient EU farming systems (SureFarm) project report, EU Horizon 2020 Grant Agreement No. 727520. SureFarm, UK.

Paas, W., F. Accatino, F. Appel, J. Bijttebier, J. Black, C. Gavrilescu, V. Krupin, G. Manevska-Tasevska, F. Ollendorf, M. Peneva, J. Rommel, C. San Martín, S. Severini, B. Soriano, S. Valchovska, M. Vigani, E. Wauters, K. Zawalińska, C. Zinnanti, M. Meuwissen, and P. Reidsma. 2020. FoPIA-SURE-Farm 2. Page in F. Accatino, W. Paas, H. Herrera, F. Appel, C. Pinsard, S. Yong, L. Schutz, B. Kopainsky, K. Bańkowska, J. Bijttebier, J. Black, C. Gavrilescu, V. Krupin, G. Manevska-Tasevska, F. Ollendorf, M. Peneva, J. Rommel, C. San Martín, S. Severini, B. Soriano, S. Valchovska, M. Vigani, E. Wauters, K. Zawalińska, C. Zinnanti, M. Meuwissen, and P. Reidsma, editors. D5.5 Impacts of future scenarios on the resilience of farming systems across the EU assessed with quantitative and qualitative methods. Sustainable and resilient EU farming systems (SURE-Farm) project report. SureFarm, UK.

Peterson, C. A., V. T. Eviner, and A. C. M. Gaudin. 2018. Ways forward for resilience research in agroecosystems. Agricultural Systems 162:19-27.

Plumecocq, G., T. Debril, M. Duru, M. B. Magrini, J. P. Sarthou, and O. Therond. 2018. The plurality of values in sustainable agriculture models: diverse lock-in and coevolution patterns. Ecology and Society 23(1):21.

Pretty, J. 2008. Agricultural sustainability: concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences 363(1491):447-465.

Pretty, J., and Z. P. Bharucha. 2014. Sustainable intensification in agricultural systems. Annals of Botany 114:1571-1596.

Prosperi, P., T. Allen, B. Cogill, M. Padilla, and I. Peri. 2016. Towards metrics of sustainable food systems: a review of the resilience and vulnerability literature. Environment Systems and Decisions 36(1):3-19.

Quinlan, A. E., M. Berbés-Blázquez, L. J. Haider, and G. D. Peterson. 2016. Measuring and assessing resilience: broadening understanding through multiple disciplinary perspectives. Journal of Applied Ecology 53(3):677-687.

R Core Team. 2015. R: A language and environment for statistical computing. R Core Team, Vienna, Austria.

Reidsma, P., and F. Ewert. 2008. Regional farm diversity can reduce vulnerability of food production to climate change. Ecology and Society 13(1):38.

Reidsma, P., M. Meuwissen, F. Accatino, F. Appel, I. Bardaji, I. Coopmans, C. Gavrilescu, F. Heinrich, V. Krupin, G. Manevska-Tasevska, M. Peneva, J. Rommel, S. Severini, B. Soriano, J. Urquhart, K. Zawalinska, and W. Paas. 2020. How do stakeholders perceive the sustainability and resilience of EU farming systems? Eurochoices 19(2):18-27.

Reidsma, P., W. Paas, A. Spiegel, and M. Meuwissen. 2019. Guidelines for the Framework of Participatory Impact Assessment of SUstainable and REsilient EU FARMing systems (FoPIA-Surefarm). Supplementary Materials L in W. Paas, F. Accatino, F. Antonioli, F. Appel, I. Bardaji, I. Coopmans, P. Courtney, C. Gavrilescu, F. Heinrich, V. Krupin, G. Manevska-Tasevska, D. Neumeister, M. Peneva, J. Rommel, S. Severini, B. Soriano, M. Tudor, J. Urquhart, E. Wauters, K. Zawalinska, M. Meuwissen, and P. Reidsma, editors. D5.2 Participatory impact assessment of sustainability and resilience of EU farming systems. Sustainable and resilient EU farming systems (SURE-Farm) project report, EU Horizon 2020 Grant Agreement No. 727520.

Resilience Alliance. 2010. Assessing resilience in social-ecological systems: workbook for practitioners. Version 2.0.

Ridder, D., and C. Pahl-Wostl. 2005. Participatory integrated assessment in local level planning. Regional Environmental Change 5(4):188-196.

Robards, M. D., M. L. Schoon, C. L. Meek, and N. L. Engle. 2011. The importance of social drivers in the resilient provision of ecosystem services. Global Environmental Change 21(2):522-529.

Sieber, S., T. S. Amjath-Babu, P. Reidsma, H. Koenig, A. Piorr, I. Bezlepkina, and K. Mueller. 2018. Sustainability impact assessment tools for land use policy advice: a comparative analysis of five research approaches. Land Use Policy 71:75-85.

Spiegel, A., T. Slijper, Y. De Mey, M. Poortvliet, J. Rommel, H. Hansson, M. Vigani, B. Soriano, E. Wauters, F. Appel, F. Antonioli, H. Harizanova, C. Gavrilescu, P. Gradziuk, D. Neumeister, and M. Meuwissen. 2019. D2.1 Report on farmers’ perceptions of risk, adaptive capacity and resilience. Sustainable and resilient EU farming systems (SureFarm) project report, EU Horizon 2020 Grant Agreement No. 727520. SureFarm, UK.

Tendall, D. M., J. Joerin, B. Kopainsky, P. Edwards, A. Shreck, Q. B. Le, P. Kruetli, M. Grant, and J. Six. 2015. Food system resilience: defining the concept. Global Food Security 6:17-23.

Tilman, D., K. G. Cassman, P. A. Matson, R. Naylor, and S. Polasky. 2002. Agricultural sustainability and intensive production practices. Nature 418(6898):671-677.

Toth, F. L. 2001. Participatory integrated assessment methods: an assessment of their usefulness to the European Environmental Agency. Technical report 64. European Environment Agency, Copenhagen, Denmark.

Urruty, N., D. Tailliez-Lefebvre, and C. Huyghe. 2016. Stability, robustness, vulnerability and resilience of agricultural systems. A review. Agronomy for Sustainable Development 36:15.

Vaidya, A., and A. L. Mayer. 2014. Use of the participatory approach to develop sustainability assessments for natural resource management. International Journal of Sustainable Development and World Ecology 21(4):369-379.

Van Passel, S., and M. Meul. 2012. Multilevel and multi-user sustainability assessment of farming systems. Environmental Impact Assessment Review 32(1):170-180.

Walker, B., S. Carpenter, J. Anderies, N. Abel, G. S. Cumming, M. Janssen, L. Lebel, J. Norberg, G. D. Peterson, and R. Pritchard. 2002. Resilience management in social-ecological systems: a working hypothesis for a participatory approach. Ecology and Society 6(1):14.

Walker, B., C. S. Holling, S. R. Carpenter, and A. P. Kinzig. 2004. Resilience, adaptability and transformability in social-ecological systems. Ecology and Society 9(2):5.

Walker, B., and D. Salt. 2012. Resilience practice: building capacity to absorb disturbance and maintain function. Island Press, Washington, D.C., USA.

Westley, F., P. Olsson, C. Folke, T. Homer-Dixon, H. Vredenburg, D. Loorbach, J. Thompson, M. Nilsson, E. Lambin, J. Sendzimir, B. Banerjee, V. Galaz, and S. Van Der Leeuw. 2011. Tipping toward sustainability: emerging pathways of transformation. AMBIO 40:762.

Winkler, K. J., M. W. Scown, and K. A. Nicholas. 2018. A classification to align social-ecological land systems research with policy in Europe. Land Use Policy 79:137-145.

Address of Correspondent:
Wim Paas
Wageningen University
Plant Production Systems
P.O.Box 430
6700AK Wageningen
The Netherlands
Jump to top
Table1  | Table2  | Table3  | Table4  | Table5  | Table6  | Figure1  | Figure2  | Figure3  | Figure4  | Appendix1  | Appendix2  | Appendix3  | Appendix4  | Appendix5  | Appendix6