Ecology and Society
Home | Archives | About | Login | Submissions | Subscribe | Contact | Search

 E&S Home > Vol. 16, No. 2 > Art. 14

Copyright © 2011 by the author(s). Published here under license by The Resilience Alliance.
Go to the pdf version of this article

The following is the established format for referencing this article:
Bottrill, M. C., M. Hockings, and H. P. Possingham. 2011. In pursuit of knowledge: addressing barriers to effective conservation evaluation. Ecology and Society 16(2): 14. [online] URL: http://www.ecologyandsociety.org/vol16/iss2/art14/


Research

In Pursuit of Knowledge: Addressing Barriers to Effective Conservation Evaluation

Madeleine C. Bottrill 1, Marc Hockings 2 and Hugh P. Possingham 1


1School of Biological Sciences, The University of Queensland, 2School of Geography, Planning and Environmental Management, The University of Queensland



ABSTRACT


Evaluation, the process of assessing the effectiveness of programs and activities, has gained increasing attention in the conservation sector as programs seek to account for investments, measure their impacts, and adapt interventions to improve future outcomes. We conducted a country-wide evaluation of terrestrial-based conservation programs in Samoa. Though rarely applied, the benefit of evaluating multiple projects at once is that it highlights factors which are persistent and influential across the entire conservation sector. We found mixed success in achieving goals among conservation programs; yet this result is surrounded by uncertainty because of the quality of existing evidence on project outcomes. We explore the role of different components of the conservation management system, i.e., context, planning, inputs, processes, and outputs, in facilitating and/or constraining collection of data on project outcomes, and thereby assessment of whether projects were successful. Our study identified a number of direct and indirect barriers that affected the capacity of projects to carry out informative evaluations and generate knowledge on conservation progress in Samoa. These attributes and mechanisms include: the availability and management of data, design and planning of projects, and systems for reporting among donors and proponents. To overcome these barriers to evaluation, we believe that a shift in institutional approaches to reporting outcomes is needed, from a reflective way of thinking to a more prospective outlook.


Key words: conservation; effectiveness; evaluation; Pacific Islands; success



INTRODUCTION


Despite an increase in response measures to protect biodiversity, the condition of species and habitats continues to decline (Butchart et al. 2010). Existing resources for conservation are inadequate to reverse human-induced rates of extinction back to natural levels (James et al. 1999). To meet conservation goals within constrained budgets, there is a need therefore to efficiently allocate resources between different actions based on the expected net benefit of those actions for biodiversity, the probability of success, and cost (Possingham et al. 2001, Joseph et al. 2009). Among these variables, information on whether an action is likely to succeed is rarely comprehensively assessed (Ferraro and Pattanayak 2006), yet these data are critically important to prevent valuable resources being wasted on ineffective interventions and for measuring progress toward conservation goals that we ultimately aspire to achieve.

Evaluating conservation projects is difficult because definitions of success depend on the values of those individuals or organizations assessing outcomes of a project and is therefore context-specific and subjective (Patton 2008). Outputs are the products delivered at the end of a process, whereas outcomes represent the level of performance of a project (Owen 2006). Measures of conservation performance are diverse including: (1) persistence of species populations; (2) improvements to ecological condition; (3) reduction of threatening processes; (4) financial return on investment; and (5) enhancement of social or human capital (Axford et al. 2008). Recognizing, identifying, and reporting the multidimensional attributes of successful and failed conservation projects contributes to evidence, or a knowledge base, of what works (Saterson et al. 2004) and encourages transparent reporting of actions that do not work (Redford and Taber 2000). Knowledge of past successes and failures influences how future interventions are designed, implemented, managed, and assessed (Axford et al. 2008). The quality and strength of available knowledge on project outputs and outcomes also determines the level of confidence about which factors lead to effective conservation (Pullin and Knight 2009).

Evaluation, the process of assessing and measuring effectiveness of projects, provides the information to enable us to determine the factors that influence outcomes (Rossi et al. 2004), which is in turn essential for choosing future projects. The use of evaluation in conservation has grown considerably in the last decade, with an increasing number of case studies (Sayer et al. 2007, Andam et al. 2008), reviews of trends and approaches (Kleiman et al. 2000, Stem et al. 2003), guidelines and toolkits (Conservation Measures Partnership 2007, Kapos et al. 2008), and a recent special issue of the academic journal, New Directions for Evaluation, dedicated to environmental evaluation (Mickwitz and Birnbaum 2009). Interest in evaluation is also represented by a number of transdisciplinary societies including the American Evaluation Association (www.eval.org), the European Evaluation Society (www.europeanevaluation.org), and the Australasian Evaluation Society (www.aes.asn.au), which promote best practice and innovation in the sector and provide resources to support professionals working in this field. The increase in conservation evaluation has coincided with an age of accountability in which nongovernment organizations and government agencies are placing greater emphasis on tracking and reporting how conservation funds are spent so as to demonstrate return on investments to donors and the general public (Christensen 2003). Alongside performance auditing, evaluation in the conservation context is used to assess the status of a biological feature, build support for a program among stakeholders, and develop guidance for future investments (Hockings et al. 2006). In particular, measurement of project outputs and outcomes through evaluation can help managers and policy makers identify successful interventions (Kapos et al. 2009) or modify those that are less effective through a process of continuous learning (Gerber et al. 2005, McCarthy and Possingham 2007, Grantham et al. 2010).

As conservation evaluation gathers momentum, a number of theoretical and practical challenges constrain its implementation and uptake. The initiation of evaluation processes in the first instance is constrained by a shortfall in funding and time allocated specifically for the purpose of monitoring and evaluation, alongside a reluctance to divert resources from implementation (Kapos et al. 2008). In addition, hesitancy among managers to expose the shortcomings of a project can curb enthusiasm for evaluation (Redford and Taber 2000). Once underway, successful evaluation is often undermined by lack of resources and poor design. Frequently, objectives of conservation projects are not clearly stated or linked directly to individual actions that might be monitored later (Margoluis et al. 2009a). Few organizations gather key ecological and social data needed to monitor the impact of conservation interventions (Kapos et al. 2008). In the absence of appropriate baseline information, other data, unfit for purpose, are used to measure progress (Pullin and Knight 2009). There is often a lack of trained staff or capacity among organizations to undertake evaluation, and experimental methodologies applied in other disciplines, e.g., randomly controlled experiments, might be inappropriate to the dynamic multidimensional systems in which conservation is applied (Hockings et al. 2009, Margoluis et al. 2009b). Measuring progress is difficult because the time required to demonstrate positive change to social and ecological systems is lengthy and often extends well beyond the lifespan of most conservation projects (Hildén 2009). These challenges act synergistically, resulting in both limited implementation of evaluation or uptake of evaluation findings. Advocates of evaluation argue that these issues should not stop us evaluating, but rather should be explicitly considered in how evaluation processes are designed and implemented to suit the context and needs of the area of interest (Stem et al. 2005, Kapos et al. 2008, Mickwitz and Birnbaum 2009).

We carried out a country-wide evaluation on the Pacific island of Samoa in 2009 using document content analysis and semistructured interviews. The benefit of evaluating multiple projects at once is that it enables identification of attributes and processes that are influential and persistent across the national conservation sector. Our objective was to identify factors that determine whether a project is successful or not in the hope of informing future decision making. These factors might encompass all aspects of project design and delivery including context, e.g., governance, level of development, village size, level of threat; inputs, e.g., availability of data, staff capacity, and level of funding; processes, e.g., consultation with stakeholders; outputs, e.g., extent of implementation; and characteristics of targeted space biodiversity space features, e.g., recovery potential of species. We encountered firsthand many challenges associated with accessing and collating existing data on project effectiveness. Illustrated by data from our own evaluation, we explore aspects of the management system that might potentially constrain or facilitate effective conservation evaluation. We use our experience to suggest general strategies to address barriers that constrain implementation of evaluation methodologies, thereby mainstreaming knowledge of success into decision making.



METHODS


Study area

The archipelago of Samoa (13°35’S, 172°20’W; Fig. 1) lies at the heart of a global biodiversity hot spot in the Pacific region of Polynesia and Micronesia (Mittermeier et al. 2005, Conservation International 2007). The terrestrial ecosystems of Samoa support 720 species of native vascular plants, 30% of which are found nowhere else in the world (Whistler 1992). Historically, the greatest changes to the region’s natural ecosystems have been caused by logging for timber and conversion of forests for agriculture and plantations (Atherton 1994). More recently, the spread of invasive species have threatened the healthy conditions of habitats and populations of endemic species (Kingsford et al. 2009). In addition, it is predicted that climate change will exacerbate threats to already vulnerable species and habitats, and dramatically impact human livelihoods across these low-lying nations with unforeseen biodiversity consequences (IPCC 2007). To address declines in biological, cultural, and economic values, substantial conservation efforts over the past 30 years have been made in Samoa to restore and safeguard biodiversity and natural resources from past and existing threats (Fig. 2).

Selection of projects for evaluation

Different types of conservation interventions, actions, activities, and programs, (henceforth “projects”) have been implemented in Samoa to date. Our evaluation aimed to build a comprehensive, inclusive, and representative dataset of all terrestrial projects. We used projects that met the following criteria:
  • completed between 1990 and 2008, more than one year but less than 20 years;
  • a focus on terrestrial ecosystems (Pearsall and Whistler 1991) inclusive of coastal mangrove areas;
  • based wholly or partly in Samoa;
  • a focus on conservation of biodiversity and/or sustainable use of natural resources.
We did not include large-scale strategic initiatives such as the Convention on Biological Diversity’s Programme of Work on Protected Areas (Dudley et al. 2005), but rather focused on more specific, site-scale projects that might be embedded within such initiatives. Identification of projects was assisted by literature and database searches, and consultations with scientists, managers, and officials in Samoa.

Evaluation design

We used a mixed method approach to collect and analyze data on project characteristics, outputs, and outcomes (Sechrest and Sidani 1995). Information obtained from each source was cross-examined to develop a comprehensive understanding of the impact of projects.

Content analysis

We collected and reviewed all available and accessible written documents, reports, book chapters, journal articles, studies, and monitoring surveys related to each of the selected projects. We used content analysis, or assessment of written texts and interview transcripts for patterns and trends, to extract information on the objectives of the project, extent of implementation, and reported outcomes or achievements of the project to date.

Semistructured interviews

Evidence to support the assessment of project outcomes was also collected through semistructured interviews, which followed an open framework of themed questions that were conversational and flexible in nature (Mason 2002). Interviewees included conservation managers, government officials, community members, regional experts, representatives of donor agencies, academics, or those individuals involved in the development or implementation of a particular conservation project. Over two visits to Samoa, 30 participants were interviewed. Individual responses are anonymously coded in the text as “S...” and specific projects are coded as “P...”.

Systems analysis

Natural resource governance in Samoa represents a complex social-ecological system with interactions between humans and the biophysical world through the exploitation of natural resources, agriculture, and fisheries. The Samoan people and the natural environment are closely connected with 78% of the population living outside urban areas. The conservation sector in Samoa represents a management system composed of people, institutions, structures, and processes that interact together to deliver a set of project outputs and outcomes. The complexity of this system means that there is uncertainty and diversity surrounding the outcomes produced by projects. Several evaluations have examined specific parts of this system in isolation, for example, assessing completion of project activities (output) in relation to the amount of funding provided (input). In reality, a diversity of social, ecological, and political elements, in addition to economic factors, is likely to influence project outputs and outcomes (Maani and Cavana 2007). We used a systems thinking approach to articulate the different and interdependent elements, i.e., funding agencies, project proponents, village councils, and government agencies, and their contribution as a part, and as a whole, to the success and failure of conservation projects. Mapping structures, inputs, and processes of the management system in relation to the observed endpoints (project outputs and outcomes) can provide a better understanding of how well the management system functions and which factors influence project effectiveness and informative evaluation (Fig. 3). The pathway to success might be navigated along a number of different trajectories and there is likely to be no specific formula for effective conservation.



RESULTS


Overview of projects assessed

We identified 39 terrestrial conservation projects, with a combined budget of US$5.4 million, completed in Samoa over the past 20 years. Among these projects, a subset of 27 projects had adequate information on project objectives, activities, and a sufficient lag time since completion, i.e., minimum one year. Site management, e.g., protection or restoration of an area, was the primary objective of the majority of projects (68%). Both capacity-building and livelihoods were also important secondary objectives of many projects, 30% and 54%, respectively. Excluding five projects, which involved 50 year covenants between donors and a community, the average duration of projects was three years. The budgets of projects varied over two orders of magnitude, from US$5000 up to US$740,000, with a median budget size of US$41,500.

The current state of knowledge

Outputs are products, events, or services delivered through implementation of project activities (Rossi et al. 2004). Among the 27 projects originally assessed, 22 projects had sufficient information to assess the types of outputs achieved. The other five projects had only a brief report or proposal of activities with no follow-up information on outputs or outcomes. The types of outputs produced by projects tend to reflect their objectives. Site management projects implemented activities including fencing of areas, removal of invasive species, and replanting of trees. Education and awareness projects held training workshops and distributed teaching materials and booklets. Livelihood-based project activities included construction of walkways for tourism and socioeconomic household surveys. There was considerable variation in the extent of implementation of planned activities across all projects. Five projects implemented all activities with another four projects implementing over half of their activities. However, over half of all projects implemented less than 50% of activities listed and one project implemented no activities at all.

An outcome is the state or condition of a target that a project ultimately intends to influence or change. Empirical data on outcomes were not available for many activities. Some projects measured outputs related to social and economic targets, e.g., number of educational workshops (n = 6), yet no projects measurably demonstrated specific outcomes. We therefore assessed the likelihood a project achieved its intended outcomes based on three criteria: (1) the extent of activity implementation; (2) the perception of success by a proponent or donor engaged in the project; and (3) available qualitative or quantitative evidence extracted from documents, surveys, or evaluation reports. In Table 1, we summarize the extent to which assessed projects were likely to achieve their outcomes with some examples. We found that a substantial number of projects achieved only some contribution to their intended objectives.

Quality and strength of existing knowledge

The types of evidence available to assess programs or actions varied from the results of quantitative monitoring to an informed observation by a project manager (Table 2). Eight projects of the 22 projects assessed had printed information from monitoring surveys on outputs, e.g., growth of replanted mangrove areas. We acquired the remaining information on outputs and outcomes from interviews.

There were 10 projects with some type of external evaluation or completion report. Six additional projects had some type of assessment of results such as a narrative report or monitoring form. Eight projects had formal terminal evaluation processes, required by the donor. As an evaluator discusses below, the purpose of these evaluations was mainly to assess compliance by the project with the approved work plan, measure deliverables, and to account for project expenditures, rather than focus on outcomes.
It is all output-based monitoring. I developed a monitoring template for them which is really based on the approved work plan and budget. So when you go out, you simply go out to check on the work plan and budget to see if they have done what they are supposed to do. (S20)
In general, evaluations focused on the inputs and process-related aspects of project activities. As one respondent describes, evaluations were only able to capture the short-term tangible outputs.
The actual tangible project outputs have not been achieved so that’s why you assess more the design of the project. In terms of end of project evaluations then you look mostly at the objectives in terms of tangible outputs, those are measured 3 or 4 or 5 years after the project has been implemented. (S28)
The consequence of evaluations focused on the products delivered by a project, or output-based evaluation, is that these approaches examine the design of projects and not whether projects ultimately contribute to improved biodiversity or other desired values.



FACTORS AFFECTING INFORMATIVE EVALUATION


Among assessed projects, we found there were limited data available to measure whether projects achieved their ultimate goals. Using a systems thinking approach, we explored which factors affect measurement of project outcomes, and therefore if a project was successful or not. We describe below, supported by empirical data from our evaluation and qualitative responses by interview respondents, the role of different components of the management system, i.e., context, planning, inputs, processes, and outputs, in contributing to these factors. We found that availability of outcome data was affected by the quality of existing datasets available prior to interventions, the extent and collection of new data, and the size of the budget; yet these factors were influenced by underlying factors linked to project planning and design and the context for conservation in Samoa.

Extent and quality of baseline data

Information on the status of a project target, e.g., condition of an important habitat, prior to the start of the project, is useful for comparing outcomes against project objectives (Rossi et al. 2004). Existing knowledge of ecological systems and features in Samoa is however limited and unevenly distributed (Doherty and Atherton 2008). Recognizing the gaps in existing biodiversity data, many projects collected further information on ecological systems as a project activity (n = 13). Knowledge on social and economic attributes related to conservation is also patchy and information such as the financial benefits from ecosystem services or the distribution of skills in conservation management is neither quantifiable nor accessible at a national level (Ward et al. 1999). Despite the prominence of socioeconomic objectives in project proposals, existing baseline data on social or economic trends, e.g., current state of knowledge among community prior to education program, was infrequently considered (n = 4). “We didn’t do the before analysis...[there were] no proper measures for evaluation on how capacity improved” (S6).

As the project manager explains above, a project with a primary objective of capacity building did not assess existing skills of trainees or collect baseline data on current levels of knowledge.

Implementation of monitoring strategies

Baseline data provide a benchmark for comparison with data collected through monitoring of subsequent activities. The process of monitoring therefore plays an important role in the ability of projects to collate information about outputs and outcomes (Salafsky et al. 2002). Among assessed projects with sufficient information (n = 22), monitoring was considered implicitly or explicitly in over half of project work plans (12 of 22 projects). Among projects with monitoring listed in their work plan, just over half of these actually implemented any monitoring activities (7 of 12 projects).

Resource availability

The collection of preintervention data were directly influenced by the availability of funding, capacity, and to the length of time needed to implement conservation projects. The lack of resources available for conservation was cited by interview respondents as a major constraint to effective implementation and successful conservation outcomes. Among project managers, lack of funds was perceived as one of the key barriers to successful implementation, monitoring, and reporting of projects.
People are aware and there are enough action plans and policy. There is everything out there to do something but there is not enough money to do, to implement. If there was enough money to do all that was needed, then it would be done. (S28)
There is nobody specifically working there, but if we could get some funds to do some further monitoring work then we go out there and do some work there. (S2)
The short duration of projects also affected the ability of projects to monitor changes over time. Observable changes to ecological features or social processes tend to have long time horizons.
It is difficult to talk about outcomes when you are working in a short space and time. It takes much longer than two years to really see the impact of the project especially conservation outcomes. (S20)
However, other respondents emphasized limited capacity among government staff and also community leaders.
A big problem ...is that there are just not enough people in the Ministry. It is a big challenge to constantly follow up that things are progressing. So you need to appreciate the capacity constraints. That’s why they tend to bring in consultants but the problem with that is that there is no real capacity. No transfer of skills. (S25)
Inadequacy of resources affected all aspects of project management. In terms of data availability, resource shortages lead to inconsistent datasets, incomplete project activities, and prevent further data collection.

Project design and planning

To understand how resource constraints led to incomplete work plans and inadequate data on project outcomes, we examined the way in which projects were planned and designed from the outset, and the types of objectives they hoped to achieve. Nearly 90% of assessed projects stated multiple objectives combining social, economic, and ecological values associated with conservation. This leads projects to broaden their scope and develop lengthy work plans to address the diversity of activities required for achieving multidisciplinary objectives. As a consequence, projects must monitor all these activities to demonstrate any progress toward broader objectives. The expansion of work plans to capture the complexity of the activities also requires more data, a diversity of skills, and substantial resources. Coupled with ambitious work plans, projects had short time frames to plan, implement, manage, and monitor their conservation activities.

Values and motivations

There is a growing trend for projects to state multiple objectives. Ecological values of biodiversity, such as protecting endemic species and maintaining representative habitats, were the primary goals of the majority (85%) of assessed projects (n = 27). Social and economic benefits were also perceived by interview respondents as key motivators for communities to engage in conservation, and therefore, as one respondent discusses, such objectives are desirable.
We were trying to ensure that while protecting all of this biological diversity in the region we were also looking at ways in which the community can benefit from conservation. Many countries have limited land and you just can’t tell them to conserve. We need to provide something for the community. (S1)
The intention of many projects in Samoa is that ecological sustainability is achieved alongside local economic development (Hunnam 2002). Almost 80% of Samoa’s population lives outside the two major towns in the country (PRB 2008) and therefore maintaining healthy subsistence-level farming and fishing systems plays a vital role in the majority of communities.

Sources of funding for conservation

Where funding for conservation comes from influences project success in many ways. More and more, international funders seek multiple objectives in project proposals. It is assumed, as described by a respondent below, that projects include multiple objectives to win funding yet these might differ from national priorities and/or what the project eventually implements:
It changes every day depending on the donor. There’s already a national strategy but there are enough national strategies out there, but what really are the country priorities? Are they the priorities that are being implemented? No. It’s what the donor is willing to pay for. (S28)
International donor-assisted projects also have short funding cycles, between three to five years, that restrict time available both to implement activities and to see changes to project targets. As a monitoring report for one project stated:
The project design is complicated for a local community to deliver on and coordinate effectively. It consists of many outputs with overlapping time lines. All of them are crammed into a fairly tight two-year time frame which demands disciplined and dedicated implementation on the part of the community, as well as the timely contribution from collaborating agencies. (Monitoring report 2008, P54)
Ambitious work plans are commonplace among international aid-funded projects as they attempt to integrate multiple objectives. When coupled with short project duration, we found some projects simply ran out of time and did not complete all their activities (n = 9).

We have represented the interactions between components and processes that generate these factors in a conceptual model (Fig. 3). Available quantitative data on achievement of stated objectives is directly related to the occurrence of specific processes, e.g., implementation of project activities, data management, or monitoring of project objectives, and the input of sufficient resources, e.g., funding, good quality existing data, capacity, and time. Constraints on these parts of the management system were the result of underlying organizational and socioeconomic attributes of the conservation sector in Samoa related to the design and planning of projects, cultural and institutional values, and availability of funding.



DISCUSSION


Our outcomes-based evaluation of conservation projects in Samoa found effectiveness varied considerably, with many projects making only a partial contribution (41%) to their primary objectives. These projects shared similar characteristics including their primary objective of biodiversity conservation and their sources of funding. However there were also differences, for example, some were single-site interventions and others were multistakeholder, multiple-site strategies. There were benefits and limitations to the methodology used for our evaluation in estimating these results. This comparative analysis of multiple projects at a national scale enabled us to examine the conservation management system in Samoa as a whole, bringing together the perspectives of donors, government, project managers, and to a lesser extent, project beneficiaries. This type of analysis is useful because it allows us to highlight common ground among seemingly divergent projects, from shared aspirations, such as finding a balance between local economic development and biodiversity conservation, to persistent challenges like sustainable financing of projects, which are faced by many actors within this system. Our application of this approach was aided by the relatively small scale of the conservation sector in Samoa, an island country with a population less than 180,000. Despite the benefits of this approach, examples of multiple project evaluations are rare in the conservation literature (but see Sayer et al. 2007, Axford et al. 2008, Lima et al. 2010), in part due to the difficulties in collating sufficient data on a diversity of projects covering multiple types of interventions, but also because there are generally relatively few examples of any conservation evaluation case studies in peer-reviewed publications.

Our assessment did not intend to be an arbitrary judgment about any one project or intervention and its positive or negative achievements. Time and resource constraints meant this study could not collect primary data on projects or interview intended project beneficiaries; therefore we relied upon data readily accessible and available. The lack of preintervention data on project outcomes posed substantial challenges in developing a comprehensive evaluation of those interventions. In this study, both quality and availability of evidence was highly variable between projects with most sources being subjective, based on participant observations and anecdotes. These data brought limitations and we acknowledge that some of our assertions might be imperfect. We used a measure, “percentage of goal achievement,” to determine the results emerging from projects, which was not formally generated by projects themselves and was our own construction. In addition, our assessment of success was underpinned by the goals stated by projects, which in reality might not be appropriate or tractable to perceptions of success. One consequence might be that projects that aim low are scored highly and vice versa. An alternative might have been to conduct a goal-free evaluation in which the evaluator avoids learning stated goals and focuses on observing processes and outcomes (Scriven 1991).

A major, though unintended, result of our initial study was the identification of a number of direct and indirect barriers to informative monitoring and evaluation results. These attributes and mechanisms affected the capacity of projects in Samoa to design and implement informative evaluations by which to measure the contribution of their actions to overall goals. These barriers included the availability and management of data, design and planning of projects, and systems for reporting among donors and proponents. In the remaining discussion, we investigate how these barriers are manifested and perpetuated, and what solutions are available to minimize their impact on future assessments. In particular, we make the case for a radical change in institutional thinking about measuring effectiveness and promoting adaptive management. Finally, we outline a series of recommendations for improving monitoring and evaluation in Samoa, which acknowledge the existing constraints and opportunities in the current management system.

The insufficient availability and limited quality of existing baseline data and subsequent monitoring data constrained the ability of proponents to measure the effectiveness of their projects (Fig. 3, Table 2). Absent or inadequate monitoring resulted in reduced confidence that projects did in fact achieve their goals. In addition, most projects did not have a system for organizing, retaining, and capturing data and knowledge gained through conservation activities. In particular, data related to social targets were not well organized or managed, which made them difficult to analyze. Without baseline information, it is impossible to track whether any project activities are delivering outcomes.

The shortfall of resources and availability of data fundamentally inhibited the ability of projects themselves to measure progress toward goals, which was primarily due to the way that projects were designed to meet the expectations of communities and donors. From the local proponent perspective, the onus is on conservation projects to contribute to ecological values but also provide multiple socioeconomic benefits to communities in terms of income generation, training, and education. From the donor perspective, funding agencies expect projects to address changing regional or global priorities, alongside local or national goals, within relatively short time frames. As a result, projects continue to address these expectations of donors and communities by including many different objectives into their work plans. The preference for multiple objectives is based on perceptions of designing conservation projects that are more comprehensive, improve efficiency, promote more sustainable outcomes, and contribute to both social and ecological goals (McShane and Wells 2004). Multiple objective projects however invariably require multilayered work plans, a diverse set of skills among staff, and dynamic project management. Conversely, as demonstrated in Samoa, without adequate resources and capacity, multiple objective projects can lead to incomplete implementation, over-stretched staff, and disappointed donors and beneficiaries. A significant contributing factor was the length of project time frames. The majority of projects were funded for less than three years. As a consequence, many projects had inadequate time to complete activities which in turn meant that the evaluations could only measure processes and some outputs.

Current attitudes to reporting of conservation outcomes in Samoa favor promoting successes and burying failure. This poses a significant barrier to informative evaluation by discounting the value of learning from unsuccessful outcomes. These attitudes are based on how success is perceived by both donors and proponents. The tendency is for success to be viewed in terms of short-term, rather than long-term, impacts. Among proponents, success is perceived as further investment in the next funding cycle; whereas among donors, success is perceived as disbursement of funding. By taking this short-term outlook, everyone loses. Donors cannot learn about problems in their strategy or the true impact of projects and therefore they are left with a lack of knowledge on how to direct funds in the future. Proponents on the other hand face a continuous dilemma. If proponents focus on success, then expectations of donors are raised by their ambitious projects and thereby, proponents must keep offering more to compete for funds. Ultimately, it is intended beneficiaries that are the biggest losers when conservation needs are not met.

Opportunities do however exist in Samoa for changing and adapting current systems of monitoring and evaluation, which might not be present in countries with seemingly adequate resources and better capacity. For example, the adaptive capacity of Samoan institutions or the ability of the system to adjust to changing internal demands and external circumstances, might be less constrained by “rigidity traps” observed in other systems (Carpenter and Brock 2008). Rigidity traps, found in bureaucratic systems with highly connected institutions, are characterized by rigidity and inflexibility where command and control has compromised diversity and innovation (Holling et al. 2002, Allison and Hobbes 2004). Natural resource governance in Samoa has the opportunity to avoid this clutter and instead leverage the messiness of existing institutional arrangements to learn and adapt project management and monitoring without being entangled in layers of bureaucracy. In some respects, given the dependence of projects on international donor funding, many projects are in fact stuck in global rigidity traps. Working within existing institutions rather than implementing programs in parallel could help minimize this constraint. Another significant opportunity in Samoa is the social structure and use patterns associated with “Fa’a Samoa” (the “Samoan way”), a traditional system of expected behavior and responsibilities. The hierarchical “matai” system holds social norms associated with rights of access to resources, which are granted by village chiefs and provide a mechanism for social insurance based on reciprocal generosity (Fiske 1992, Cox and Elmqvist 1997). Community-based resource governance is likely to be most effective when there are local authorities, either informal or formal, such as the village matais in place to enforce restrictions (Barrett et al. 2001). With increasing globalization in the Pacific, it should be recognized that there are benefits to maintaining such traditional institutions.

The current system of natural resource management in Samoa is faced with limited capacity and resources, inadequate project planning, and systemic problems related to donor and stakeholder expectations; yet there are also opportunities within existing institutions to facilitate flexible and sustainable solutions. The design of projects will also involve a balancing act between national social, economic, and environmental priorities. Undoubtedly, there is a need for local economic development to occur alongside biodiversity conservation, such as Integrated Conservation and Development Programmes that have proven to be a valued approach to sustainable resource management in the Pacific (Hunnam 2002). Two approaches to how this might be realized include: (1) longer and more expensive projects that have the capacity to deliver on the theoretically sound ICDP approach, and (2) promoting the ICDP outcomes by linking more directed short-term projects toward some broader overall goal but while keeping individual projects more targeted and simple. The second approach is increasingly evident among recent regional funding initiatives by the Critical Ecosystem Partnership Fund (CEPF) and the GEF Small Grants Programme (SGP). The multidimensional aspects of sustainable conservation are unavoidable. We therefore advocate that systems thinking, as demonstrated in this study, be applied more widely in Samoa. These existing funding and resources constraints are likely to persist in Samoa for the foreseeable future. It is therefore necessary to devise realistic and practical recommendations that acknowledge them when trying to improve monitoring and evaluation in Samoa. In the remainder of this discussion, we speculate on ways for Samoan institutions to embrace complexity and provide a series of recommendations for improving knowledge in Samoa on conservation outcomes in the coming decade.

The existing management system in Samoa takes a static approach to reporting that is focused on accountability and compliance, i.e., were all of the funds disbursed? Was the work plan completed? Instead, we encourage managers to make a shift in their approaches to reporting to be more adaptive and prospective. This approach would ask instead, what activities, processes, or decisions are needed to improve conservation outcomes? How could activities or project management be improved? Improvements to space mechanistic components space alone, e.g., more funds or better trained staff, will not necessarily achieve better outcomes. Inherently, future decisions are based on the past and without prospective thinking, managers and decision makers are likely to make the same mistakes. If all the resources and expertise that were needed were available, continuing along the same trajectory as past prescriptions would not deal with the dynamic world. A culture of learning and forward thinking would instead promote transparency about unsuccessful activities and hopefully encourage donors and proponents to report and learn failures that would benefit everyone.

A shift toward a prospective approach to evaluation would be required at multiple scales, from individual projects to broader strategies. Some of these changes could be addressed at a national or institutional level by Samoan implementing agencies whereas others are beyond the scope of national institutions alone and must be considered by the broader global conservation sector. For example, to address the difficulties that come with multiple objective projects, adjustments are needed in how aid is disbursed and the ways international initiatives are designed. In light of the disconnection between global development priorities and local needs, there has been some effort to change this situation through the Paris Declaration on Aid Effectiveness (2005) and the subsequent Accra Agenda for Action (2008), which has been ratified by over 100 countries. These international agreements represent a commitment by donor countries and agencies to help developing country governments formulate and implement their own national development plans.

On a national level, we believe if institutions in Samoa embrace a prospective perspective to reporting success and failure and use a systems thinking approach to measuring outcomes, that they can produce more effective and informative evaluations. In the following section, we outline a number of practical and realistic changes, within system limitations and opportunities, that could bring institutions closer to this goal over the next 10 years. These changes represent conceptual, institutional, methodological, and logistical adjustments to the way that projects are currently designed and planned:
  • Acknowledgement of the complexity of social-ecological systems in Samoa. The concept of systems thinking is a useful way to understand the interactions between different components of the natural resource management system in Samoa. Current evaluation approaches that focus on ecological aspects alone are likely to underestimate or ignore the socio-political and economic drivers influencing outcomes. Systems thinking would help managers and evaluators take a more open and exploratory approach to analyzing factors affecting conservation effectiveness;
  • Working within existing institutional structures. Many current projects run in parallel to existing institutions with external evaluators assessing programs over short time frames. If a culture of evaluation and self-assessment was integrated and supported as part of core business within existing institutions then ownership over evaluation findings and adaptive management might be more likely. Samoa has the advantage of existing traditional institutions with social norms that foster environmental responsibility and cooperation that should be maximized in project design;
  • Collection and management of appropriate ecological and social data from quantitative and qualitative sources using mixed methodologies. A centrally administered, for example, within the Ministry of Natural Resources and Environment, and regularly updated system is needed to improve accessibility and transparency of data layers from projects. A system for managing knowledge would provide information of existing baselines, but also represent a mechanism for learning and enable an information exchange between proponents and end users;
  • Prioritization of data collection through a selection of nationally agreed indicators. There is a trade-off between resources invested in monitoring and the amount of information gained through monitoring. Monitoring less, but better, would provide more useful information for managers than spreading resources too thinly trying to measure everything. A national dialogue about the critical social and ecological indicators to track progress needs to be initiated;
  • Application of evaluation methodologies that recognize the complexity of systems and encourage prospective thinking, including theories of change (ToC), conceptual models, and self-learning tools. A ToC approach tries to articulate a systematic and cumulative interpretation of the links between activities, outcomes, and contexts of an initiative (Connell and Kubisch 1998). The advantages of a ToC approach is that it does not judge complex interventions as a whole but rather attempts to uncover which aspects of the process have or have not been successful (MacKenzie and Blamey 2005). Using conceptual models to map out the entire system and then select crucial components to focus monitoring activities could help efficiently allocate available resources (Margoluis et al. 2009a). In addition, self-evaluation in which project proponents assess their own performance could help foster ownership of evaluation results and counter aversion to reporting failures (Taylor 1994).
We have highlighted a series of recommendations that we believe are feasible given the existing resources and capacity available and the socioeconomic context for conservation in Samoa. The majority of these adjustments corresponds to the “planning and design” and “processes” stages of the evaluation cycle (Fig. 3) and might not require substantial additional inputs than those that already exist. The most difficult part will be changing attitudes toward how project outcomes are reported and encouraging the conservation sector to embrace prospective evaluation. Funding and training will go some way in achieving more informative evaluation, but we make the case that much greater gains might be made by thinking more positively about learning from successes and failures.



CONCLUSION


In recent years, the field of biodiversity conservation has entered an age of accountability in which scientists and policy makers have called for an “effectiveness” revolution that builds evidence demonstrating that conservation programs are making an impact, but also are cost-effective (Pullin and Knight 2001, Sutherland et al. 2004, Ferraro and Pattanayak 2006). People want to know that biodiversity is better off from these efforts. The reality on the ground, as described in this study within the context of terrestrial conservation projects in Samoa, is that there are substantial barriers to achieving informative evaluation outcomes. In the short term, the impact of simple direct interventions are perhaps more measurable than multidimensional integrated initiatives, but neglect the complexity of social-ecological systems. Projects have been overwhelmed by the expectations of donors and community agendas, causing them to lose sight of their ultimate goals. Institutional change is therefore required, particularly among those organizations and agencies responsible for funding conservation programs, which incentivizes and promotes a culture of adaptive management and prospective ventures in trial and error.



RESPONSES TO THIS ARTICLE


Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.




ACKNOWLEDGMENTS

We would like to thank all the participants in Samoa who generously gave time to share their experiences and knowledge of conservation issues in Samoa and the Pacific region. We would also like to thank James Atherton, Leilani Duffy, Faleafaga Toni Tipamaa, Ana Tiraa, and Alan Tye for assistance during the researcherís visits to Samoa. We also thank Joanna Axford, James Atherton, Dan Segan, and two anonymous reviewers for their comments on an earlier version of this manuscript. This study has been made possible through funding from the Critical Ecosystem Partnership Fund with assistance provided by Conservation Internationalís Pacific Islands Program. MCB is supported by a Northcote Graduate Scholarship and funding from the Commonwealth Environmental Research Facility to the Applied Environmental Decision Analysis centre.




LITERATURE CITED


Allison, H. E., and R. J. Hobbs. 2004. Resilience, adaptive capacity, and the "Lock-in Trap" of the Western Australian Agricultural Region. Ecology and Society 9(1): 3. [online] URL: http://www.ecologyandsociety.org/vol9/iss1/art3/.

Andam, K. S., P. J. Ferraro, A. Pfaff, G. A. Sanchez-Azofeifa, and J. A. Robalino. 2008. Measuring the effectiveness of protected area networks in reducing deforestation. Proceedings of the National Academy of Sciences of the United States of America 105:16089-16094.

Atherton, J. 1994. Planning new conservation areas in Western Samoa: a case study. Thesis. University College of North Wales, Bangor, UK.

Axford, J. C., M. T. Hockings, and R. W. Carter. 2008. What constitutes success in Pacific Island community conserved areas? Ecology and Society 13(2): 45. [online] URL: http://www.ecologyandsociety.org/vol13/iss2/art45/.

Barrett, C. B., K. Brandon, C. Gibson, H. Gjertsen. 2001. Conserving tropical biodiversity amid weak institutions. Bioscience 51:497-502.

Butchart, S. H. M., M. Walpole, B. Collen, A. van Strien, J. P. W. Scharlemann, R. E. A. Almond, J. E. M. Baillie, B. Bomhard, C. Brown, J. Bruno, K. E. Carpenter, G. M. Carr, J. Chanson, A. M. Chenery, J. Csirke, N. C. Davidson, F. Dentener, M. Foster, A. Galli, J. N. Galloway, P. Genovesi, R. D. Gregory, M. Hockings, V. Kapos, J.-F. Lamarque, F. Leverington, J. Loh, M. A. McGeoch, L. McRae, A. Minasyan, M. H. Morcillo, T. E. E. Oldfield, D. Pauly, S. Quader, C. Revenga, J. R. Sauer, B. Skolnik, D. Spear, D. Stanwell-Smith, S. N. Stuart, A. Symes, M. Tierney, T. D. Tyrrell, J.-C. Vie, and R. Watson. 2010. Global biodiversity: indicators of recent declines. Science 328:1164-1168.

Carpenter, S. R., and W. A. Brock. 2008. Adaptive capacity and traps. Ecology and Society 13(2): 40. [online] URL: http://www.ecologyandsociety.org/vol13/iss2/art40/.

Christensen, J. 2003. Auditing conservation in an age of accountability. Conservation in Practice 4:12-18.

Connell J. P., and A. C. Kubisch. 1998. Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects and problems. Pages 15-44 in A. Fulbright-Anderson, A. C. Kubisch, and J. P. Connell, editors. New approaches to evaluating community initiatives: theory, measurement and analysis. Aspen Institute, Washington, D.C., USA.

Conservation International. 2007. Ecosystem profile: Polynesia-Micronesia biodiversity hotspot. Conservation International-Melanesia Center for Biodiversity Conservation, Atherton, Australia.

Conservation Measures Partnership. 2007. Open standards for the practice of conservation. Version 2.0. Conservation Measures Partnership, Washington, D.C., USA.

Cox, P. A., and T. Elmqvist 1997. Ecocolonialism and indigenous-controlled rainforest preserves in Samoa. Ambio 26:84-89.

Doherty, N., and J. Atherton. 2008. Literature review on terrestrial biological survey information in Samoa. Ministry of Environment and Natural Resources and Conservation International Pacific Islands Program, Apia, Samoa.

Dudley, N., K. Mulongoy, S. Cohen, S. Stolton, C. Barber, and S. Gidda. 2005. Towards effective protected area systems: an action guide for implementing the convention on biological diversity programme of work on protected areas. Technical Series No. 18. Secretariat of the Convention on Biological Diversity, Montreal, Quebec, Canada.

Ferraro, P. J., and S. K. Pattanayak. 2006. Money for nothing? A call for empirical evaluation of biodiversity conservation investments. Plos Biology 4:482-488.

Fiske, S. J. 1992. Sociocultural aspects of establishing marine protected areas. Ocean & Coastal Management 18:25-46.

Gerber, L., M. Beger, M. McCarthy, and H. P. Possingham. 2005. A theory for optimal monitoring of marine reserves. Ecology Letters 8:829-837.

Grantham, H. S., M. Bode, E. McDonald-Madden, E. T. Game, A. T. Knight, and H. P. Possingham. 2010. Effective conservation planning requires learning and adaptation. Frontiers in Ecology and the Environment 8:431-437.

Hildén, M. 2009. Time horizons in evaluating environmental policies. New Directions for Evaluation 122:9-18.

Hockings, M., S. Stolton, N. Dudley, and R. James. 2009. Data credibility: what are the right data for evaluating management effectiveness of protected areas? New Directions for Evaluation 122:53-63.

Hockings, M., S. Stolton, F. Leverington, N. Dudley, and J. Courrau. 2006. Evaluating effectiveness: a framework for assessing management effectiveness of protected areas. International Union for Conservation of Nature, Gland, Switzerland and Cambridge, UK.

Holling C. S., L. H. Gunderson, and G. D. Peterson. 2002. Sustainability and panarchies. Pages 63-102 in L. H. Gunderson and C. S. Holling, editors. Panarchy: understanding transformations in human and natural systems. Island Press, Washington, D.C., USA.

Hunnam, P. 2002. Lessons in conservation for people and projects in the Pacific Islands region. United Nations Development Programme, New York, New York, USA.

Intergovernmental Panel on Climate Change (IPCC). 2007. Climate Change 2007: Impacts, adaptation and vulnerability. Contribution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. M. L. Parry, O. F. Canziani, J. P. Palutikof, P. J. van der Linden, and C. E. Hanson, editors. Cambridge University Press, Cambridge, UK.

James, A. N., K. J. Gaston, and A. Balmford. 1999. Balancing the Earth's accounts. Nature 401:323-324.

Joseph, L. N., R. F. Maloney, and H. P. Possingham. 2009. Optimal allocation of resources among threatened species: a project prioritization protocol. Conservation Biology 23:328-338.

Kapos, V., A. Balmford, R. Aveling, P. Bubb, P. Carey, A. Entwistle, J. Hopkins, T. Mulliken, R. Safford, and A. Stattersfield. 2009. Outcomes, not implementation, predict conservation success. Oryx 43:336-342.

Kapos, V., A. Balmford, R. Aveling, P. Bubb, P. Carey, A. Entwistle, J. Hopkins, T. Mulliken, R. Safford, A. Stattersfield, M. J. Walpole, and A. Manica. 2008. Calibrating conservation: new tools for measuring success. Conservation Letters 1:155-164.

Kingsford, R. T., J. E. M. Watson, C. J. Lundquist, O. Venter, L. Hughes, E. L. Johnston, J. Atherton, M. Gawel, D. A. Keith, B. G. Mackey, C. Morley, H. P. Possingham, B. Raynor, H. F. Recher, and K. A. Wilson. 2009. Major conservation policy issues for biodiversity in Oceania. Conservation Biology 23:834-840.

Kleiman, D. G., R. P. Reading, B. J. Miller, T. W. Clark, M. Scott, J. Robinson, R. L. Wallace, R. J. Cabin, and F. Felleman. 2000. Improving the evaluation of conservation programs. Conservation Biology 14:356-365.

Lima, F. P., J. N. Muniz, and P. de Marco, Jr. 2010. Evaluating Brazilian conservation projects: the weak link between practice and theory. Natureza & Conservação 8:41-45.

Maani, K. E., and R. Y. Cavana. 2007 Systems thinking, system dynamics: managing change and complexity. Pearson Prentice Hall, Auckland, New Zealand.

MacKenzie M., and A. Blamey. 2005. The practice and the theory: lessons from the application of a theories of change approach. Evaluation 11:151-168.

Margoluis, R., C. Stem, N. Salafsky, and M. Brown. 2009b. Design alternatives for evaluating the impact of conservation projects. New Directions for Evaluation 122:85-96.

Margoluis, R., C. Stem, N. Salafsky, and M. Brown. 2009a. Using conceptual models as a planning and evaluation tool in conservation. Evaluation and Program Planning 32:138-147.

Mason, J. 2002. Qualitative researching. Sage, London, UK.

McCarthy, M., and H. P. Possingham. 2007. Active adaptive management for conservation. Conservation Biology 21:956-963.

McShane, T. O., and M. P. Wells. 2004. Getting biodiversity projects to work: towards more effective conservation and development. Columbia University Press, New York, New York, USA.

Mickwitz, P., and M. Birnbaum. 2009. Key insights for the design of environmental evaluations. New Directions for Evaluation 122:105-112.

Mittermeier, R. A., P. R. Gil, M. Hoffman, J. Pilgrim, T. Brooks, C. G. Mittermeier, J. Lamoreux, and G. A. B. da Fonseca. 2005. Hotspots revisited: Earth’s biologically richest and most endangered terrestrial ecoregions. Conservation International, Washington, D.C., USA.

Owen, J. M. 2006. Program evaluation: forms and approaches. Allen & Unwin, Crows Nest, New South Wales, Australia.

Patton, M. Q. 2008. Utilization-focused evaluation. Fourth Edition. Sage, Thousand Oaks, California, USA.

Pearsall, S. H., and W. A. Whistler. 1991. Terrestrial ecosystem mapping for Western Samoa. Government of Samoa and South Pacific Regional Environment Programme, Apia, Samoa.

Population Reference Bureau (PRB). 2008. 2008 World Population Data Sheet. Population Reference Bureau, Washington, D.C., USA.

Possingham, H. P., S. Andelman, B. R. Noon, S. Trombulak, and H. R. Pulliam. 2001. Making smart conservation decisions. Pages 225-244 in M. E. Soulé and G. Orians, editors. Conservation biology: research priorities for the next decade. Island Press, Washington, D.C., USA.

Pullin, A. S., and T. M. Knight. 2001. Effectiveness in conservation practice: pointers from medicine and public health. Conservation Biology 15:50-54.

Pullin, A. S., and T. M. Knight. 2009. Data credibility: a perspective from systematic reviews in environmental management. New Directions for Evaluation 122:65-74.

Redford, K. H., and S. Taber. 2000. Writing the wrongs: developing a safe-fail culture in conservation. Conservation Biology 14:1567-1568.

Rossi, P. H., M. W. Lipsey, and H. Freeman. 2004. Evaluation: a systematic approach. Seventh Edition. Sage, Thousand Oaks, California, USA.

Salafsky, N., R. Margoluis, K. H. Redford, and J. G. Robinson. 2002. Improving the practice of conservation: a conceptual framework and research agenda for conservation science. Conservation Biology 16:1469-1479.

Saterson, K. A., N. L. Christensen, R. B. Jackson, R. A. Kramer, S. L. Pimm, M. D. Smith, and J. B. Wiener. 2004. Disconnects in evaluating the relative effectiveness of conservation strategies. Conservation Biology 18:597-599.

Sayer, J., B. Campbell, L. Petheram, M. Aldrich, M. R. Perez, D. Endamana, Z. L. N. Dongmo, L. Defo, S. Mariki, N. Doggart, and N. Burgess. 2007. Assessing environment and development outcomes in conservation landscapes. Biodiversity and Conservation 16:2677-2694.

Scriven, M. 1991. Prose and cons about goal-free evaluation. American Journal of Evaluation 12:55-62.

Sechrest, L., and S. Sidani. 1995. Quantitative and qualitative methods: is there an alternative? Evaluation and Program Planning 18:77-87.

Stem, C., R. Margoluis, N. Salafsky, and M. Brown. 2003. A review of monitoring and evaluation approaches and lessons learned in conservation. Fifth World Parks Congress 2003: benefits beyond boundaries. International Union for Conservation of Nature, Durban, South Africa.

Stem, C., R. Margoluis, N. Salafsky, and M. Brown. 2005. Monitoring and evaluation in conservation: a review of trends and approaches. Conservation Biology 19:295-309.

Sutherland, W. J., A. S. Pullin, P. M. Dolman, and T. M. Knight. 2004. The need for evidence-based conservation. Trends in Ecology & Evolution 19:305-308.

Taylor, L. 1994. Reflecting on teaching: the benefits of self-evaluation. Assessment & Evaluation in Higher Education 19:109-122.

Ward, T., F. Kingstone, and S. Siwatibau. 1999. Indicators of success: South Pacific biodiversity conservation programme. Volume One: Technical Report. South Pacific Regional Environment Programme, Apia, Samoa.

Whistler, W. A. 1992. Vegetation of Samoa and Tonga. Pacific Science 46:159-178.


Address of Correspondent:
Madeleine C. Bottrill
School of Biological Sciences
University of Queensland
St Lucia, QLD 4072
Australia

m.bottrill@uq.edu.au

Up

Home | Archives | About | Login | Submissions | Subscribe | Contact | Search