Research, part of a special feature on Multicriteria Assessment of Food System Sustainability

Scope and precision of sustainability assessment approaches to food systems

• Abstract
• Introduction
• Methods and Criteria Development
  • Identification of sustainability assessment approaches
  • Development of a typology for analyzing the scope of the approaches
  • Development of a framework for analyzing the precision of the approaches
  • Selection of approaches for the assessment of precision
  • Assessment of the precision of the six selected approaches
  • Identification of trade-offs
• Results
  • Overview of the scope of sustainability approaches
  • Precision of the six selected approaches
• Discussion and Conclusions
  • A way toward more transparency and harmonization in the future
• Responses to This Article
• Acknowledgments
• Literature Cited

INTRODUCTION

Food production, in particular its agricultural stages, has substantial impacts on climate change and biodiversity and on different environmental resources, such as water, soil, and air (Steinfeld 2006, Rockström et al. 2009, Foley et al. 2011). Moreover, the socioeconomic impacts of food production are important because social and economic conditions, especially on farms, are often unfavorable and agriculture is often one of few economic activities in rural areas (European Commission 2004). Sustainable development (WCED 1987) has become one of the most frequently used frameworks for analyzing the agricultural and food sector in a comprehensive and holistic way.

During the past 15 years, a variety of different approaches have been developed for assessing aspects of sustainability in the food sector and especially for agricultural production (Grimm 2009, Singh et al. 2009, Binder et al. 2010). However, these approaches have various limitations. For instance, life cycle assessment (LCA) tools quantitatively address many environmental aspects in a rather precise way and with a high resolution of data but largely ignore impacts on biodiversity and soil quality (Schader et al. 2012b) and economic and socio-cultural impacts (Finkbeiner et al. 2010). Approaches for holistic farm assessment, such as Response-Inducing Sustainability Evaluation (RISE; Grenz et al. 2009), comprehensively address sustainability but can only be applied to agricultural enterprises. Environmental management, i.e., International Organization for Standardization (ISO) 14001, Eco-Management and Audit Scheme, and sustainability reporting systems (GRI 2011) establish procedures for dealing with sustainability at the company level but do not allow for cross-company comparisons and lack a clearly defined science-based assessment methodology.

Several classification schemes are suggested in the literature to compare existing sustainability assessment approaches systematically. Gasparatos and Scolobig (2012) classified sustainability assessment tools into three principal families: monetary tools, biophysical tools, and indicator tools. Ness et al. (2007) gave a comprehensive overview of fundamental sustainability assessment approaches and categorized these according to their temporal scope, i.e., prospective and retrospective, and whether they were product related, were indicator based, or took an integrated view. De Ridder et al. (2007) classified the approaches into assessment frameworks, participatory tools, scenario analysis tools, multicriteria analysis tools, cost-benefit and cost-effectiveness analysis tools, accounting tools, physical analysis tools and indicator sets, and model tools. Marchand et al. (2012) compared 2 assessment tools, focusing on the complexity of farm-level assessment approaches, and used a list of 10 criteria including complexity, data correctness, effectiveness, and compatibility. Bockstaller et al. (2006) did cross-comparisons of 4 farm-level environmental assessment approaches using indicators for the assessment concept, feasibility, and usefulness as evaluation criteria. Schader (2009) compared 7 economic models that included environmental indicators. Binder et al. (2010) examined normative, procedural, and systemic dimensions of 7 sustainability assessment approaches at farm and regional levels. They categorized the approaches into 3 types: top-down farm assessments, top-down regional assessments with some stakeholder participation, and bottom-up integrated participatory or transdisciplinary approaches. To understand the possible differences in the results of sustainability assessment approaches, it is necessary to examine the large breadth of different sustainability assessment approaches in terms of purpose, assessment level, and thematic and geographical coverage.

We aim to shed light on the prevalent differences between sustainability assessment approaches in terms of their scope and precision. We specifically aim to classify sustainability assessment approaches used for evaluating farms, farming systems, or products according to their main characteristics; to assess differences in scope and precision between different approaches; and to discuss possible consequences for the explanatory power and target audiences of these approaches as well as implications for the current debate on sustainability in food production.

METHODS AND CRITERIA DEVELOPMENT

We conducted the comparison of different sustainability assessment approaches in six steps.

Identification of sustainability assessment approaches

First, a literature survey and expert consultations were conducted to identify different sustainability assessment approaches. We included all approaches that can be applied in the food sector, address at least the environmental dimension of sustainability, and cover more than one sustainability impact category. Approaches that cover only one impact category, such as carbon footprints (e.g., Hillier et al. 2011) or biodiversity assessment (Jenny et al. 2013), were explicitly excluded.

Although sustainability is a multidimensional concept that includes environmental, economic, social, and governance dimensions (see FAO 2012), the term “sustainability assessment” is often used even if only the environmental dimension of sustainability is assessed. Therefore, we also included sustainability assessment approaches for agricultural systems if they addressed at least the environmental dimension of sustainability. We identified 35 approaches to sustainability assessment but make no claim that this list is complete.

Development of a typology for analyzing the scope of the approaches

In a second step, we developed a typology for characterizing the scope of the 35 sustainability assessment approaches. From these approaches and other comparison papers (De Ridder et al. 2007, Ness et al. 2007, Zapf et al. 2009, Binder et al. 2010), we defined the following characterization criteria (Table 1): primary purpose of the assessment, level of assessment, geographical scope, sector scope, thematic scope, and perspective on sustainability. Reviewing publicly available documents and scientific papers enabled the actual characterization of the sustainability assessment approaches.

The primary purposes of a sustainability assessment approach can vary considerably and include the following: purely science-oriented approaches for research; monitoring and certification schemes intended to provide proof, such as to consumers, of the sustainability performance of companies; landscape planning tools that focus on the regional level and consider, for example, the environmental and socioeconomic surroundings of a number of farms and assess the impacts on sustainability; farm advisory tools to didactically assess the strengths and weaknesses of a farm and serve as a basis for management improvement or strategy development; and self-assessment tools that serve a similar purpose, but without the support of an adviser. Finally, some sustainability assessment approaches are designed primarily for policy advice.

The level of assessment describes whether a sustainability assessment addresses the entire agricultural sector of a country, region, or landscape; the entire farm or a field; or a product or group of products and standards. Approaches addressing the product level typically cover the entire supply chain or at least the relevant production processes.

Geographical scope describes the area in which the approaches have been applied and in which they can be applied. The geographical applicability of approaches depends on data availability and the applicability of model algorithms or indicators to different societies, regions, or ecosystems.

Similar to the geographical scope, the sector scope defines whether an approach is applicable only to specific farm types, e.g., dairy or meat, or to all branches of the agriculture and food sector. The applicability of an approach can be restricted because of model algorithms and indicators or because of data constraints.

To analyze the thematic scope of the sustainability assessment approaches in terms of impact assessment categories covered in each sustainability dimension, we used the environmental, social, and economic subthemes of the Sustainability Assessment in Food and Agriculture Systems (SAFA) Guidelines (FAO 2012) as a reference. SAFA is a globally applicable guiding framework for sustainability assessments in the food and agricultural sector. We employed a simplistic approach by calculating the share of SAFA subthemes addressed in the respective sustainability assessment method (see Appendix 1). We stress that this approach allows indication of whether a SAFA subtheme is addressed but does not allow assessment of whether the subtheme is addressed adequately and accurately. Although the SAFA framework describes “Governance” as a fourth dimension, we base this typology on the WCED (1987) classical structure with only the economic, environmental, and social dimensions because many of the themes in the governance dimension refer to companies rather than farms.

An important distinction between the approaches needs to be made with respect to their perspective on sustainability. In forestry, where the concept of sustainability was first formulated by Carlowitz in 1713, the term originally described a “minimum criterion” that prevents the overuse and degradation of forest resources. However, there are now two prevalent perspectives of the notion “sustainability” in the context of food and agriculture systems:

• The business or farm perspective of sustainability describes whether the farm is able to sustain itself for an extended period of time. This interpretation focuses on whether the farm, as an economic entity, is resilient enough to use its natural, social, and economic resources without depleting them and is also able to cope with possible future changes in the societal, economic, and environmental framework.
• The societal perspective, as defined in the report Our Common Future (WCED 1987), assesses whether a farm contributes to a sustainable development of society. This means that the assessments focus on the impacts of farm management on the economic, social, and environmental resources of society. These impacts can be either positive, i.e., services delivered, or negative, i.e., damages or costs induced. The societal perspective is often also employed in the concept of multifunctionality of agriculture (OECD 2001).

Development of a framework for analyzing the precision of the approaches

In a third step, we developed a framework for assessing the precision of the approaches. As the term precision is itself often used imprecisely, a clear distinction must be made between its different meanings:

1. Precision in the sense of accuracy describes the proximity of a measured or calculated value to the true value of the considered variable.
2. Precision in the sense of reproducibility refers to the degree of similarity between the values obtained from repeated measurements of the same variable.
3. Precision in the sense of measurement resolution reflects the smallest change in the considered variable that can be detected by the given method.

In research, precision is most often used in the sense of accuracy. For our aim, this would mean comparing the real or actual sustainability performance, such as of a farm, with the sustainability performance results calculated by an assessment approach. However, this would require that the “real values” of the sustainability variables are known. Assessing the precision of the sustainability assessment approaches in such a direct way is beyond our scope and is, in fact, impossible in many situations. For this research, we therefore adopted the interpretation of precision with respect to measurement resolution. This interpretation pertains to the ability of an approach to distinguish the outcome of different situations, such as before and after an action intended to improve sustainability has been implemented on a farm. From a practical point of view, this ability is very important in numerous contexts: from impact assessment to farm advisory. To operationalize precision in the sense of measurement resolution for one of the sustainability dimensions, we used the following criteria: (1) whether qualitative, semiquantitative, or quantitative assessments, where applicable, are used to generate results for a sustainability dimension; (2) the thematic coverage of impact assessment categories within each sustainability dimension, i.e., the wider the coverage of topics within a sustainability dimension is, the more precisely the dimension can be described; (3) appraisal of the complexity of model algorithms; and (4) the time required for on-site data collection.

In the appraisal of complexity of model algorithms, we distinguish between three levels: low, medium, and high. Model algorithms are deemed to be of high complexity if many input variables are taken into account when compared to the Intergovernmental Panel on Climate Change (IPCC) Tier 3 approaches (IPCC 2006) for modeling greenhouse gases. Medium complexity means that greenhouse gases are covered but in less detail. A low level of complexity for this example means that only a Tier 1 approach, or a set of indicators, is used for evaluation of greenhouse gas emissions. Other indicators can be appraised accordingly. However, the complexity of model algorithms is mostly valid for indicators that can be covered quantitatively, mainly environmental and some economic indicators.

On-site data collection causes opportunity costs for the person, such as a farmer or a manager of a processing operation, who provides information as the basis for sustainability assessment. However, on-site data collection is not only determined by the sustainability assessment approach itself but also by the availability of secondary data. Indeed, for product-related approaches, data and background information is available through life cycle inventory databases, such as ecoinvent (Frischknecht et al. 2007). According to Zapf et al. (2009), the willingness of farmers to participate in sustainability assessments, therefore, very much depends on the amount of time required for on-site data collection. On-site data collection in the sustainability assessment approaches compared by Zapf et al. (2009) ranged from six hours to up to three days. Taking this range as a basis, we rated an on-site data collection time of less than three hours to be low, between three to six hours to be moderate, and more than six hours to be high.

Selection of approaches for the assessment of precision

The 4th step was to select 6 sustainability assessment approaches out of the 35 identified approaches for the precision assessment decided in the 5th step. We selected approaches in which the availability and transparency of information allowed an informed assessment of precision. The second selection criterion was that the approach had been developed for different primary purposes, e.g., research, policy advice, or farm advice, so as to cover a wide range of potential application areas. Based on these criteria, we selected the following: the Swiss agricultural life cycle assessment (SALCA); the German method REPRO, which focuses on the environmental dimension of sustainability and which was then integrated into the sustainability certification tool “DLG-Zertifikat”; the farm advisory method RISE; the farm-level impact assessment Committee on Sustainability Assessment (COSA); the public monitoring system Agrar-Umweltindikatoren (Agri-environmental indicators, AUI); and the policy impact assessment method Farm Modeling Information System (FARMIS).

Assessment of the precision of the six selected approaches

In a fifth step, we assessed the precision of the six selected approaches against the criteria defined in the third step. For some approaches, the precision assessment could be done on the basis of the authors’ own experiences with the approaches or from available documentation (RISE, FARMIS, and SALCA). However, because we had limited experience in application and there was insufficient available documentation, for others of these sustainability approaches, we used phone interviews (J. Frei, Federal Office of Agriculture, 5 September 2013, personal communication; D. Giovannucci, The COSA, 6 September 2013, personal communication; K.-J. Hülsbergen, Technische Universität München, 2 September 2013, personal communication) to gain the information required.

To ensure a fair assessment of the approaches, the judgment of the precision of the approaches was carried out by the first two authors independently according to the framework developed in the fourth step. To maximize validity, we adapted Jankowicz’s (2005) co-recategorization procedure, which serves to quantify, and thereby illustrate, the degree of agreement in the allocated ratings. Differences in the rating judgment were discussed until a consensus was reached. It was not necessary to use the complete procedure because the categories had been decided in advance, so we were able to omit the first step in which categories are identified and simply compare the judgments for each category by the individual researchers.

Identification of trade-offs

In a final 6th step, trade-offs between scope and precision were identified on the basis of the data on scope of the 35 sustainability assessment approaches and the data on precision of the 6 selected approaches. First, we looked for interdependencies between different types of scopes and identified patterns. Second, we looked for these interdependencies among the 6 models with respect to precision. Finally, we compared the data on scope and precision and identified reasons for potential trade-offs.

RESULTS

In presenting the results, we apply the typology of scopes to the 35 approaches that were found in literature (Table 2), present the results of the analysis of the precision of the six selected approaches, and finally compare the scope and precision of the tools to identify potential trade-offs.

Overview of the scope of sustainability approaches

The 35 approaches identified in the literature survey were classified according to the criteria presented in Table 1. The primary purpose of the tools, as stated in the cited publications or as derived from their real-world applications, is presented in Table 2. Of the 35 approaches, 14 are primarily used for research (Agri-LCA, CAPRI, DRAM, FARMIS, MMF, MODAM, PASMA, PROMAPA.G, PROSA, RAUMIS, REPRO, SAFE, SALCA, and SPA), 12 approaches can be used for policy advice (CAPRI, DRAM, FARMIS, ISAP, MODAM, PASMA, PROMAPA.G, RAUMIS, SAFE, SDA, SILAS, and SSP), 3 for farm monitoring (AUI, KSNL, and Sustainability Monitoring and Assessment Routine [SMART]), 4 for farm advice (IFSC, OCIS PG, RISE, and SSP), 2 for certification (DLG-Zertifikat and KSNL), 2 for self-assessment (DairySAT and Fieldprint calculator), 2 for landscape planning (FESLM and SSP), and 1 for consumer information (Labelguide WWF). There are 9 approaches that are used for more than one purpose, of which 7 are used for both research and policy advice.

The approaches identified cover a large spectrum with regard to the level of assessment. We found 18 approaches that can be applied at the farm level (AUI, AVIBIO, COSA, DairySAT, DLG-Zertifikat, IDEA, IFSC, ISAP, KSNL, MMF, MOTIFS, OCIS PG, REPRO, RISE, SAFE, SALCA, SMART, and SSP), 6 at the product level (Agri-LCA, GEMIS, PROSA, REPRO, SALCA, and SEEbalance), 9 at the sector level (CAPRI, DRAM, FARMIS, MODAM, PASMA, PROMAPA.G, RAUMIS, SDA, and SILAS), 5 at the landscape or regional level (FESLM, MMF, SAFE, SPA, and SSP), and 1 for assessing standards (Labelguide WWF).

The geographical scope of most of the sustainability assessment approaches is limited, with 28 approaches that have been designed for a specific geographical context and that are not globally applicable. However, even though these approaches focus on a specific geographical context, they could be made applicable to other geographical contexts via extension of algorithms or background data. In total, 7 approaches have been explicitly designed for global application and have been applied in many different countries, e.g., RISE and SMART. Apart from COSA, which is targeted to developing countries, all approaches with limited geographical scope focus on European countries.

Admittedly, geographical scope and level of the assessment cannot be clearly disentangled. Not all approaches can be unequivocally assigned to a specific level of assessment. For instance, REPRO can assess environmental impacts at the farm level and at the product level, whereas FARMIS also allows for a farm-level comparison in addition to sector-level assessments.

With respect to the sector scope, 27 are not restricted to a specific branch of the agriculture and food sector. There are 4 approaches that are limited to a specific branch, such as dairy (DairySAT), poultry (AVIBIO), coffee and cocoa (COSA), and corn, cotton, rice, wheat, potatoes, and soybeans (Fieldprint calculator). Finally, AVIBIO and Fieldprint calculator are limited to their sector and to their geographical scope.

The analysis of the thematic scope of the approaches revealed that all approaches include the environmental dimension of sustainability. The economic dimension is additionally covered by 27 approaches, and the social dimension is additionally covered in 19 of the approaches. Overall, 17 approaches cover all 3 dimensions of sustainability. Most product-related approaches were based on an LCA framework (ISO 2006a, b) and focus on indicators of environmental sustainability, except for PROSA; whereas farm- and sector-level approaches cover either the social or the economic dimension of sustainability, or both.

A societal perspective on sustainability was taken by 25 of the 35 approaches, whereas none of them took a clear business perspective. However, 11 approaches adopt a mix of both societal and business perspectives (Table 2). OCIS PG, RISE, and COSA take a mixed perspective but with a farm-level perspective on sustainability being the main area. Indeed, the farm advisory tool RISE considers economic sustainability, i.e., economic stability, profitability, and liquidity, from a farm perspective; whereas it considers, for instance, child labor and working conditions from a societal perspective. Economic sustainability in REPRO is assessed primarily from the farm perspective, but elements of the societal perspective are also employed. SMART explicitly covers the environmental and the social dimension from a societal perspective, whereas the governance and economic dimension are covered from a farm or business perspective. A fully consistent societal perspective on sustainability was found for FARMIS, i.e., environmental and economic dimension, as well as for the product-level approaches, e.g., SALCA, and AUI (monitoring), which, however, only address the environmental dimension of sustainability.

Precision of the six selected approaches

Scope and precision of environmental subthemes

Comparing the precision of the tools in assessing the environmental dimension, it became evident that RISE (all) and REPRO (17) covered the most of the 19 environmental SAFA subthemes; whereas medium coverage is achieved by SALCA and COSA (each 12), and AUI (7) and FARMIS (8) cover less than half of the 19 environmental subthemes specified in the SAFA guidelines (Table 3). RISE is the only approach that evaluates animal welfare, which is part of the environmental dimension of sustainability in the SAFA guidelines.

The complexity of environmental model algorithms is highest in the approaches developed for research: in SALCA and even slightly more complex in REPRO. For instance, REPRO models daily nutrient flows and also takes composition and decomposition of organic matter into account.

RISE uses quantitative and semiquantitative algorithms that are less complex than those of SALCA and REPRO. According to the developers, the aim of COSA is to cover the environmental topics in a way that is scientifically sound while allowing a quick overview of a large number of farms at the same time. Both RISE and COSA evaluate some environmental impacts indirectly via farm management practices implemented on the farm or by asking for the farmers’ view. For instance, soil erosion is not quantified per se, but the implementation of erosion prevention measures is assessed. Because of its high aggregation level, the sector-level impact assessment approach FARMIS can only process limited farm-specific environmental information because it uses accountancy data as a main data source. Its precision for environmental farm-level assessments is therefore low.

Scope and precision of social subthemes

The social dimension of sustainability is covered by 16 subthemes in the FAO SAFA guidelines. Only the farm-level advisory tool RISE (10 social SAFA subthemes), the assessment tool COSA (11), and, to a lesser extent, REPRO (3) cover the social dimension of sustainability. RISE semiquantitatively addresses working conditions, including gender issues and working hours; farm management, including strategy and planning; and quality of life. COSA additionally assesses participation and transparency, which in the SAFA framework are assigned to the sustainability dimension “governance.” REPRO focuses on farmers’ working conditions because this subtheme is straightforward to quantify by checking working hours, salaries, and number of leave days. The reason for this is that the certification system, DLG-Zertifikat, needs to be based on unambiguously verifiable information. On the contrary, COSA and RISE explicitly widen the focus and include “soft” factors such as farmers’ perceptions on environmental issues, quality of life, and product quality. To assess the social subthemes, all tools use relatively simple algorithms. If possible, topics are assessed quantitatively, e.g., number of working days, whereas most assessments are done semiquantitatively.

Scope and precision of economic subthemes

The economic dimension of sustainability is not addressed by AUI and SALCA. However, there have been efforts to link SALCA to farm accounting data to cover the economic performance as well (Hersener et al. 2011). FARMIS includes a detailed assessment of farm incomes and its determinants, i.e., revenues and costs. In total, 5 of the 14 economic SAFA subthemes are covered by FARMIS, whereas both COSA and RISE cover 7 economic SAFA subthemes. The DLG-Zertifikat employs a classical farm-level perspective on the economic dimension of sustainability by analyzing liquidity, stability, and profitability. However, the complexity of model algorithms is low. As FARMIS originally was an economic sector model, the economic model algorithms in FARMIS are very complex and are based on farm accountancy data. All models assess the economic dimension in a quantitative way.

Time required for data collection

The time required for data collection in the research approaches SALCA and REPRO is very high. SALCA requires more than six hours for data collection, and REPRO takes two days of on-farm data collection. Less than six hours is required by RISE, which is evaluated to be moderately time consuming. According to its developers, COSA requires only one to one and a half hours for data collection on an average family farm. For AUI, we judged the time requirements for data collection to be low because the large number of farms enables efficient data handling, although the time for data entry by farmers cannot be quantified because the system is in development (J. Frei, Federal Office of Agriculture, 5 September 2013, personal communication). FARMIS is based on farm accountancy data, whereas data for the environmental subthemes are derived from SALCA calculations as fixed factors per farm. Thus, FARMIS does not require any primary data collection.

DISCUSSION AND CONCLUSIONS

It is clear that all approaches have their limitations in scope. No approach is perfect in the sense that it comprehensively covers all sustainability dimensions, can be globally applied, and can be used for all levels of assessment from sector level over farm to the product level.

For instance, SALCA focuses on the environmental dimension and is limited in its geographical scope; RISE and COSA cover all sustainability dimensions but can only be applied for farm-level assessments in contrast to the entire supply chain; and the coverage of sustainability dimensions and the geographical scope in both FARMIS and AUI are limited. No approach serves all purposes of the sustainability assessment. For instance, FARMIS is a research approach that can also be used for policy advice but not for providing advice to farmers. RISE, on the other hand, is a farm advisory tool that has limited capabilities for answering research questions. All approaches have a special focus in scope while putting less emphasis on other aspects of scope. Thus, there is a trade-off between the different characteristics of scope.

Furthermore, the approaches either cover a limited number of SAFA themes per sustainability dimension, or the depth of analysis of the themes is limited. Most approaches make a compromise between coverage of sustainability dimensions and depth of analysis. However, this compromise often lies either on one side or the other. For instance, RISE covers 100% of all environmental topics, whereas SALCA covers only 12 of 19 topics. On the other hand, the model algorithms of SALCA are much more detailed than those of RISE. REPRO implemented complex model algorithms for 17 of 19 environmental topics at the costs of high data collection requirements and a low depth of analysis for the social and economic dimensions. So, we can identify a trade-off between the coverage of sustainability themes, the depth of analysis, and the time required for data collection: in other words, between the criteria we used to operationalize precision.

There are sustainability assessment approaches with a high level of precision that are research oriented, e.g., SALCA and REPRO, and designed for farm- or product-level impact assessment but that are limited with respect to the geographical scope, i.e., validated only for one specific country, and the thematic scope of the sustainability dimensions that are covered, i.e., focus on the environmental dimension. For instance, REPRO, in its full version, is only applicable in Europe, and SALCA does not cover the social and economic dimensions of sustainability. On the other hand, the sustainability impact assessment approaches that were designed for extension show a lower level of precision but both a wide geographical, e.g., RISE aims at globally applicability, and thematic scope. So, we found a trade-off between the scope and the precision of sustainability assessment approaches.

In summary, our analysis on scope and precision has revealed three types of trade-offs: trade-offs between the different characteristics of scope, between the criteria of precision, and between the scope and the precision. As a consequence of the trade-offs that have been identified, we can furthermore conclude that one-size-fits-all solutions are not feasible; therefore, the approaches must be tailored to a specific purpose. This conclusion is in line with the finding by De Ridder et al. (2007) who stressed that different or specific tasks require different or specific tools. Thus, sustainability assessment approaches need to be carefully chosen against the background of the task for which they will be applied. Furthermore, the design of sustainability impact assessment approaches should be tailored to the specific question or problem, its purpose, and the geographical coverage.

We also identified some misperceptions of the term sustainability, which result in the potential for confusion among stakeholders and consumers about the term. The misperceptions are because of (1) a lack of distinct use and transparency on the perspective of sustainability; and (2) use of the term sustainability even when only the environmental dimension is covered.

1. The sustainability approaches in this analysis mix the social and the business perspectives of sustainability (see Table 2) and do often not consciously distinguish between them. The two perspectives, business oriented and society oriented, can employ the same impact categories or indicators. For instance, protecting soil and water resources is beneficial to both the individual farm and the society. Whether the operation of a single farm can be sustained is of “private business interest,” but the operational sustainability of the single farm might not necessarily be of “societal interest.” Thus, the business perspective does not always correlate with the sustainability of a society.
2. There are approaches that only cover the environmental dimension of sustainability. This limited interpretation of the notion that “sustainability” is synonymous with environmental performance is a further reason for stakeholder confusion about the term sustainability. Furthermore, the indicator sets used in the approaches are not consistent for the same impact category. This means that high-precision indicators do not necessarily produce the same assessment result as indicators with low precision. We therefore recommend only using the term sustainability if at least the three dimensions of sustainability, i.e., environment, social, and economic, are covered in sufficient breadth and depth.

A further conclusion is that harmonization of models and indicators is necessary for enhancing comparability of sustainability assessment approaches. The heterogeneity described previously leads to the situation that different sustainability impact assessment approaches, e.g., when applied to the same farm, may arrive at differing impact assessment results. To avoid contradictory impact assessment results, indicator selection procedures should be harmonized to have thematically coherent indicator sets. A linguistic distinction between farm/company-level sustainability assessment and a sustainability assessment from a societal perspective is therefore essential.

As a consequence, initiatives for harmonization of approaches or for comparing approaches would be of great value for a more transparent and credible competition on food markets and among farming systems. However, as some aspects, e.g., animal welfare, are driven by ethical values, which are different among cultures, a full harmonization of sustainability assessment approaches is unlikely. Some authors even consider such a harmonization as disadvantageous because they argue that it is important that a sustainability assessment in a specific context reflects the specific values of the stakeholders (Gasparatos et al. 2008, Gasparatos 2010). Nevertheless, confusion arises even among farmers if a sustainability assessment in the context of a farm advisory service, e.g., with RISE, arrives at a different result than a sustainability assessment using an LCA-based approach such as SALCA.

A way toward more transparency and harmonization in the future

The SAFA guidelines pursue a different concept (FAO 2012). They aim at rendering approaches and results of sustainability assessments in the food sector more transparent and comparable. This is in line with the call for disclosing the values and assumptions behind sustainability (Binder et al. 2010, Gasparatos 2010). The SAFA guidelines contribute to overcoming the previously described challenges by establishing a comprehensive, widely accepted language for sustainability in agriculture and food; facilitating transparent comparisons of the sustainability performance of companies; and emphasizing the need to take the varying scope of influence of enterprises into account, which may stretch beyond the physical borders of a production site and even include suppliers and stakeholders outside the supply chain. Thus, despite being operator rather than product centered, the SAFA approach maintains a supply chain perspective. Although the SAFA guidelines provide a standard set of sustainability themes and goals that all enterprises in the sector should pursue, they allow for flexibility in selecting indicators for measuring sustainability performance. So although they aim for being globally applicable for all food, the practical applicability of the SAFA guidelines is yet to be tested and evaluated under a diversity of conditions. SAFA-consistent assessment tools, e.g., SMART, can help in operationalizing the SAFA guidelines to establish transparent and comparable reporting systems for food companies (Schader et al. 2012a, Jawtusch et al. 2013).

Still, communication of sustainability requires full transparency about the perspectives, values, and assumptions. Therefore, developing methods for communicating sustainability in both an efficient and transparent way remains a transdisciplinary challenge for researchers, developers, and relevant stakeholders. However, if the variability among the assessment perspectives remains or even increases, it is simply a matter of time before the concept of “sustainability” will lose the central influence that it currently has in the public and scientific debate.

LITERATURE CITED

Asensio, L., R. G. de Barreda, M. Ruiz, J.-L. M. de Diego and E. Miqueleiz. 2011. An application of a positive mathematical programming model to analyse the impact of agricultural policy measures in the Spanish agricultural sector. Pages 175-198 in H. A. do Prado, A. J. B. Luiz, and H. C. Filho, editors. Computational methods for agricultural research: advances and applications. IGI Global, Hershey, Pennsylvania, USA.

Bertelsmeier, M. 2005. Analyse der Wirkungen unterschiedlicher Systeme von direkten Transferzahlungen unter besonderer Berücksichtigung von Bodenpacht- und Quotenmärkten. Dissertation. Humboldt-Universität zu Berlin, Berlin, Germany.

Binder, C. R., G. Feola, and J. K. Steinberger. 2010. Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture. Environmental Impact Assessment Review 30(2):71-81. http://dx.doi.org/10.1016/j.eiar.2009.06.002

Binder, C., A. Wiek, and M. Fenchel 2001. Sustainability spaces: a new concept to evaluate development using indicator systems. Pages 33-34 in The science & culture of industrial ecology: abstracts from the inaugural meeting (Netherlands, 12-14 November 2001). International Society for Industrial Ecology, New Haven, Connecticut, USA.

Bockstaller, C., G. Gaillard, D. Baumgartner, R. Freiermuth-Knuchel, M. Reinsch, R. Brauner, and E. Unterseher. 2006. Abschlussbericht zum Projekt 04 - “Comete” 2002-2005: Betriebliches Umweltmanagement in der Landwirtschaft. Vergleich der Methoden INDIGO, KUL/USL, REPRO, SALCA. Grenzüberschreitendes Institut zur rentablen umweltgerechten Landbewirtschaftung (ITADA), Colmar, France.

Breitschuh, T., H. Eckert, U. Maier, U. Gernand, and A. Müller. 2009. Folgenabschätzung einer zunehmenden Bereitstellung von Bioenergieträgern auf die Nachhaltigkeit landwirtschaftlicher Unternehmen, bewertet mit dem Kriteriensystem Nachhaltige Landwirtschaft (KSNL). Umweltbundesamt, Bonn, Germany.

Britz, W. 2005. CAPRI modelling system documentation. Common agricultural policy regional impact analysis. “Development of a regionalised EU-wide operational model to assess the impact of current Common Agricultural Policy on farming sustainability,” J05/30/2004 – Deliverable 1. Bonn, Germany.

De Ridder, W., J. Turnpenny, M. Nilsson, and A. Von Raggamby. 2007. A framework for tool selection and use in integrated assessment for sustainable development. Journal of Environmental Assessment Policy and Management 9(4):423-441.

England, D., and S. M. White. 2009. Industry development: involving the whole industry in practice change. Extension Farming Systems Journal 5(1):207.

European Commission. 2004. Rural development in the EU. European Commission, Brussels, Belgium.

Finkbeiner, M., E. M. Schau, A. Lehmann, and M. Traverso. 2010. Towards life cycle sustainability assessment. Sustainability 2(10):3309-3322. http://dx.doi.org/10.3390/su2103309

Foley, J. A., N. Ramankutty, K. A. Brauman, E. S. Cassidy, J. S. Gerber, M. Johnston, N. D. Mueller, C. O’Connell, D. K. Ray, P. C. West, C. Balzer, E. M. Bennett, S. R. Carpenter, J. Hill, C. Monfreda, S. Polasky, J. Rockström, J. Sheehan, S. Siebert, D. Tilman, and D. P. M. Zaks. 2011. Solutions for a cultivated planet. Nature 478:337-342. http://dx.doi.org/10.1038/nature10452

Food and Agriculture Organization of the United Nations (FAO). 2012. Sustainability assessment in food and agriculture systems (SAFA) guidelines. Test version 1.0. Natural Resources Management and Environment Department, FAO, Rome, Italy. [online] URL: http://www.fao.org/fileadmin/templates/nr/sustainability_pathways/docs/SAFA_Guidelines_12_June_2012_final_v2.pdf

Frischknecht, R., N. Jungbluth, H.-J. Althaus, G. Doka, R. Dones, T. Heck, S. Hellweg, R. Hischier, T. Nemecek, G. Rebitzer, M. Spielmann, and G. Wernet. 2007. Overview and methodology. Data v.2.0 (2007). Ecoinvent Report No. 1. Swiss Centre for Life Cycle Inventories, Dübendorf, Switzerland.

Fritsche, U., L. Rausch, and K. H. Simon. 1989. Umweltwirkungsanalyse von Energiesystemen: Gesamt-Emissions-Modell Integrierter Systeme (GEMIS). Endbericht. Institut für angewandte Ökologie, Darmstadt, Germany.

Gasparatos, A. 2010. Embedded value systems in sustainability assessment tools and their implications. Journal of Environmental Management 91(8):1613-1622. http://dx.doi.org/10.1016/j.jenvman.2010.03.014

Gasparatos, A., M. El-Haram, and M. Horner. 2008. A critical review of reductionist approaches for assessing the progress towards sustainability. Environmental Impact Assessment Review 28(4-5):286-311. http://dx.doi.org/10.1016/j.eiar.2007.09.002

Gasparatos, A., and A. Scolobig. 2012. Choosing the most appropriate sustainability assessment tool. Ecological Economics 80:1-7. http://dx.doi.org/10.1016/j.ecolecon.2012.05.005

Giovannucci, D., J. Potts, B. Killian, C. Wunderlich, G. Soto, S. Schuller, F. Pinard, K. Schroeder, and I. Vagneron. 2008. Seeking sustainability: COSA preliminary analysis of sustainability initiatives in the coffee sector. Committee on Sustainability Assessment, Winnipeg, Manitoba, Canada.

Global Reporting Initiative (GRI). 2011. Sustainability reporting guidelines. Version 3.1. GRI, Amsterdam, The Netherlands.

Grenz, J., C. Thalmann, A. Stämpfli, C. Studer, and F. Häni. 2009. RISE, a method for assessing the sustainability of agricultural production at farm level. Rural Development News 1(2009):5-9.

Grimm, C. 2009. Nachhaltige Landwirtschaft: Indikator en, Bilanzierungsansätze, Modelle. Volume 74. Erich-Schmidt-Verlag, Berlin, Germany.

Helming, J. F. M. 2005. A model of Dutch agriculture based on positive mathematical programming with regional and environmental applications. Dissertation. Wageningen University, Wageningen, The Netherlands.

Hersener, J.-L., D. Baumgartner, and D. Dux. 2011. Zentrale Auswertung von Ökobilanzen landwirtschaftlicher Betriebe (ZA-ÖB). Schlussbericht. Agroscope Reckenholz-Tänikon (ART), Zurich, Switzerland.

Hillier, J., C. Walter, D. Malin, T. Garcia-Suarez, L. Mila-i-Canals, and P. Smith. 2011. A farm-focused calculator for emissions from crop and livestock production. Environmental Modelling & Software 26(9):1070-1078. http://dx.doi.org/10.1016/j.envsoft.2011.03.014

Hülsbergen, K.-J. 2003. Entwicklung und Anwendung eines Bilanzierungsmodells zur Bewertung der Nachhaltigkeit landwirtschaftlicher Systeme. Shaker, Aachen, Germany.

Intergovernmental Panel on Climate Change (IPCC). 2006. 2006 IPCC guidelines for national greenhouse gas inventories. Volume 4, agriculture, forestry and other land use. Prepared by the National Greenhouse Gas Inventories Programme. Institute for Global Environmental Strategies, Hayama, Kanagawa, Japan.

International Organization for Standardization (ISO). 2006a. Environmental management – life cycle assessment – principles and framework. ISO, Geneva, Switzerland.

International Organization for Standardization (ISO). 2006b. Environmental management – life cycle assessment – requirements and guidelines. ISO, Geneva, Switzerland.

Jankowicz, D. 2005. The easy guide to repertory grids. John Wiley and Sons, Chichester, UK.

Jawtrusch, J., C. Schader, M. Stolze, L. Baumgart, and U. Niggli. 2013. Sustainability monitoring and assessment routine: results from pilot applications of the FAO SAFA guidelines. International Symposium on Mediterranean Organic Agriculture and Quality Signs Related to the Origin (Agadir, Morocco, 2-4 December 2013). Réseau – Echanges- Développement Durable (REDD), Lausanne, Switzerland.

Jenny, M., J. Zellweger-Fischer, O. Balmer, S. Birrer, and L. Pfiffner. 2013. The credit point system: an innovative approach to enhance biodiversity on farmland. Aspects of Applied Biology 118:23-30.

Julius, C., C. Moller, B. Osterburg, and S. Sieber. 2003. Indicators for a sustainable land use in the “Regionalised Agricultural and Environmental Information System for Germany” (RAUMIS). Agrarwirtschaft 52(4):184-194.

Kloepffer, W. 2008. Life cycle sustainability assessment of products. International Journal of Life Cycle Assessment 13(2):89-95. http://dx.doi.org/10.1065/lca2008.02.376

Lang, D. J., R. W. Scholz, C. R. Binder, A. Wiek, and B. Stäubli. 2007. Sustainability potential analysis (SPA) of landfills – a systemic approach: theoretical considerations. Journal of Cleaner Production 15(17):1628-1638. http://dx.doi.org/10.1016/j.jclepro.2006.08.004

Lillywhite, R., C. Keeling, P. Courtney, N. Lampkin, B. Pearce, F. Rayns, M. Reed, U. Schmutz, C. Watson, and A. Williams. 2012. Assessing economic, environmental and social characteristics of UK farming systems. In Proceedings of the 10th European International Farming Systems Association (IFSA) Symposium (Aarhus, Denmark, 1-4 July 2012). IFSA Europe Group, Vienna, Austria.

López-Ridaura, S., H. van Keulen, M. van Ittersum, and P. Leffelaar. 2005. Multi-scale sustainability evaluation of natural resource management systems: quantifying indicators for different scales of analysis and their trade-offs using linear programming. International Journal of Sustainable Development & World Ecology 12(2):81-97. http://dx.doi.org/10.1080/13504500509469621

Mack, G., and C. Flury. 2006. Auswirkungen der Agrarpolitik 2011. Modellrechnungen für den Agrarsektor mit Hilfe des Prognosesystems SILAS. Agroscope Reckenholz Tänikon (ART), Im Auftrag für das Bundesamt für Landwirtschaft, Bern, Switzerland.

Marchand, F., L. Debruyne, and L. Lauwers. 2012. A comparison of complex expert-based assessments versus quickscan assessment. In Proceedings of the 10th European International Farming Systems Association (IFSA) Symposium (Aarhus, Denmark, 1-4 July 2012). IFSA Europe Group, Vienna, Austria.

Meul, M., S. Van Passel, F. Nevens, J. Dessein, E. Rogge, A. Mulier, and A. Van Hauwermeiren. 2008. MOTIFS: a monitoring tool for integrated farm sustainability. Agronomy for Sustainable Development 28(2):321-332. http://dx.doi.org/10.1051/agro:2008001

Nemecek, T., O. Huguenin-Elie, D. Dubois, G. Gaillard, B. Schaller, and A. Chervet. 2011. Life cycle assessment of Swiss farming systems: II. Extensive and intensive production. Agricultural Systems 104:233-245. http://dx.doi.org/10.1016/j.agsy.2010.07.007

Ness, B., E. Urbel-Piirsalu, S. Anderberg, and L. Olsson. 2007. Categorising tools for sustainability assessment. Ecological Economics 60(3):498-508. http://dx.doi.org/10.1016/j.ecolecon.2006.07.023

Organisation for Economic Co-operation and Development (OECD). 2001. Multifunctionality - towards an analytical framework. OECD, Paris, France.

Pottiez, E., P. Lescoat, and I. Bouvarel. 2012. AVIBIO: a method to assess the sustainability of the organic poultry industry. In Proceedings of the 10th European International Farming Systems Association (IFSA) Symposium (Aarhus, Denmark, 1-4 July 2012). IFSA Europe Group, Vienna, Austria.

Rigby, D., P. Woodhouse, T. Young, and M. Burton. 2001. Constructing a farm level indicator of sustainable agricultural practice. Ecological Economics 39(3):463-478. http://dx.doi.org/10.1016/S0921-8009(01)00245-2

Rockström, J., W. Steffen, K. Noone, Å. Persson, F. S. Chapin, III, E. Lambin, T. M. Lenton, M. Scheffer, C. Folke, H. Schellnhuber, B. Nykvist, C. A. De Wit, T. Hughes, S. van der Leeuw, H. Rodhe, S. Sörlin, P. K. Snyder, R. Costanza, U. Svedin, M. Falkenmark, L. Karlberg, R. W. Corell, V. J. Fabry, J. Hansen, B. Walker, D. Liverman, K. Richardson, P. Crutzen, and J. Foley. 2009. Planetary boundaries: exploring the safe operating space for humanity. Ecology and Society 14(2): 32. [online] URL: http://www.ecologyandsociety.org/vol14/iss2/art32/

Saling, P., R. Maisch, M. Silvani, and N. König. 2005. Assessing the environmental-hazard potential for life cycle assessment, eco-efficiency and SEEbalance. International Journal of Life Cycle Assessment 10(5):364-371. http://dx.doi.org/10.1065/lca2005.08.220

Sanders, J. 2007. Economic impact of agricultural liberalisation policies on organic farming in Switzerland. Dissertation. Institute of Rural Sciences, University of Wales, Aberystwyth, UK.

Sattler, C., J. Schuler, and P. Zander. 2006. Determination of trade-off-functions to analyse the provision of agricultural non-commodities. International Journal of Agricultural Resources, Governance and Ecology 5(2-3):309-325.

Schader, C. 2009. Cost-effectiveness of organic farming for achieving environmental policy targets in Switzerland. Dissertation. Institute of Biological, Environmental and Rural Sciences, Aberystwyth University, Aberystwyth, UK. Research Institute of Organic Farming (FiBL), Frick, Switzerland.

Schader, C., J. Jawtusch, D. Emmerth, R. Bickel, J. Grenz, and M. Stolze. 2012a. Sustainability assessment of operators in the food chain based on the FAO SAFA-guidelines. In Zukunft der Ökolebensmittelverarbeitung: Nachhaltigkeit - Qualität - Integrität. 2. IFOAM EU-Verarbeiterkonferenz zum Thema Ökolebensmittelverarbeitung und Umweltleistungen (Oberursel/Frankfurt, Germany, 26-27 November 2012). Research Institute of Organic Farming (FiBL), Frick, Switzerland. [online] URL: http://www.ifoam-eu.org/sites/default/files/event/files/ifoameu_processingconference-frankfurt2012_jawtusch_bickel_poster-safa_fib.pdf

Schader, C., and M. Stolze. 2011. Bewertung der Nachhaltigkeit der biologischen Landwirtschaft in der Schweiz durch Experten. Pages 332-335 in 11. Wissenschaftstagung Ökologischer Landbau (Giessen, Germany, 15-18 March). University of Giessen, Research Institute of Organic Farming (FiBL), SÖL, Giessen, Germany.

Schader, C., M. Stolze, and A. Gattinger. 2012b. Environmental performance of organic agriculture. Pages 183-206 in J. Boye and Y. Arcand, editors. Green technologies in food production and processing. Springer, New York, New York, USA.

Schmid, E., and F. Sinabell. 2006. Modelling organic farming at sector level - an application to the reformed CAP in Austria. International Association of Agricultural Economists Conference (12-18 August 2006). International Association of Agricultural Economists (IAAE), Gold Coast, Australia.

Singh, R. K., H. R. Murty, S. K. Gupta, and A. K. Dikshit. 2009. An overview of sustainability assessment methodologies. Ecological Indicators 9(2):189-212. http://dx.doi.org/10.1016/j.ecolind.2008.05.011

Smyth, A. J., and J. Dumanski. 1995. A framework for evaluating sustainable land management. Canadian Journal of Soil Science 75(4):401-406. http://dx.doi.org/10.4141/cjss95-059

Steinfeld, H. 2006. Livestock’s long shadow: environmental issues and options. Food and Agriculture Organization of the United Nations, Rome, Italy.

Van Cauwenbergh, N., K. Biala, C. Bielders, V. Brouckaert, L. Franchois, V. Garcia Cidad, M. Hermy, E. Mathijs, B. Muys, J. Reijnders, X. Sauvenier, J. Valckx, M. Vanclooster, B. Van der Veken, E. Wauters, and A. Peeters. 2007. SAFE—a hierarchical framework for assessing the sustainability of agricultural systems. Agriculture, Ecosystems & Environment 120(2-4):229-242. http://dx.doi.org/10.1016/j.agee.2006.09.006

Williams, A. G., E. Audsley, and D. L. Sandars. 2006. Determining the environmental burdens and resource use in the production of agricultural and horticultural commodities. Department for Environment, Food and Rural Affairs (Defra) Project Report IS0205. Cranfield University and Defra, Bedford, United Kingdom.

World Commission on Environment and Development (WCED). 1987. Our common future, report of the World Commission on Environment and Development. Annex to General Assembly document A/42/427 Development and International Co-operation: Environment. United Nations, New York, New York, USA.

World Wildlife Fund (WWF) Schweiz. 2010. Hintergrundbericht Labels für Lebensmittel. WWF Schweiz, Zürich, Switzerland.

Zahm, F., P. Viaux, L. Vilain, P. Girardin, and C. Mouchet. 2008. Assessing farm sustainability with the IDEA method – from the concept of agriculture sustainability to case studies on farms. Sustainable Development 16(4):271-281. http://dx.doi.org/10.1002/sd.380

Zapf, R., U. Schultheiss, R. Oppermann, H. van den Weghe, H. Döhler, and R. Doluschitz. 2009. Bewertung der nachhaltigkeit landwirtschaftlicher Betriebe. Volume Kuratorium für Technik und Bauwesen in der Landwirtschaft e.V. (KTBL) Band 473. KTBL, Darmstadt, Germany.