Perception-based Methods to Evaluate Conservation Impact in Forests Managed Through Popular Participation

We reviewed construct validity in perception-based methods assessing status and/or trend of forest condition as applied in 19 empirical studies that evaluated the conservation impact of popular participation in forest management. Perception-based methods focus on eliciting peoples’ assessment of the status and/or trend in forest condition or indicators of forest condition through interviews, surveys, or participatory rural appraisal techniques. We found that individual studies generally did not attend to the issue of construct validity in relation to each particular approach to perception-based assessment of status and/or trend in forest condition. Furthermore, the studies provided very little documentation of the construct validity of the perception-based methods as applied to assessments of forest condition in the specific context of popular participation in forest management. This scarcity of evidence implies that any support for the construct validity of these methods must be found outside the literature in which it was applied. A quick review of the literature on local assessments, monitoring, and local ecological knowledge supports the construct validity of such approaches as applied in various contexts; however, we argue that this support cannot be directly transferred to the context of popular participation in forest management. Accordingly, we conclude that there is a need for research to refine and validate perception-based methods as applied in the specific context of popular participation in forest management.


INTRODUCTION
Popular participation in the management of forests is an important policy, particularly in developing countries.Popular participation signifies peoples' participation in the management of the forests they live in and around and thus includes many management regimes found around the world, such as decentralized forest management, participatory forest management, joint forest management, community-based forest management, indigenous forestry, and social forestry.A large number of developing countries are engaged officially in promoting some form of popular participation in forest management, and recent studies estimate the share of the world's natural forests that are managed officially with some degree of popular participation at 10-12% and steadily increasing (e.g., Sunderlin et al. 2008).
At the inception of the most recent wave of promoting popular participation in forest management, which began in the 1970s, the primary objective was to promote forest conservation.Since then, the scope has been widened to encompass the improvement of rural livelihoods and poverty alleviation, and more recently, the promotion of good governance and democracy (Roe et al. 2009).The objective of forest conservation, however, remains important.Accordingly, a good number of studies have been conducted to evaluate the effects of popular participation on forest conservation (for a recent review, see Lund et al. 2009).These studies are framed as impact evaluation studies, but many of them suffer from difficulties of establishing both internal validity and construct validity (Lund et al. 2009).Both types of validity are prerequisites to establishing the degree to which any observed changes in the forest can be attributed to popular participation, rather than to other factors (Ferraro 2009).Similarly, there are many examples of impact evaluations of general conservation interventions that struggle with both types of validity (Ferraro 2009).Internal validity indicates whether one is Ecology and Society 15(3): 5 http://www.ecologyandsociety.org/vol15/iss3/art5/actually estimating a causal relationship, rather than the effect of confounding factors, whereas construct validity indicates whether the evaluation method achieves measurement of the outcome and treatment that it claims to evaluate.Hence, in relation to the evaluation of conservation impact in forests managed by popular participation, construct validity indicates whether the evaluation empirically establishes two things: the degree to which a given forest is managed by popular participation, and the degree to which it is conserved.
Here, we focus on a subset of evaluations of conservation impact of popular participation in forest management, namely, those applying perception-based methods.Perception-based methods focus on eliciting peoples' assessment of the status and/or development of forest condition or indicators of forest condition through interviews, surveys, or participatory rural appraisal techniques.Such methods seek to elicit what is often referred to as local ecological knowledge (LEK; e.g., Huntington 2000).We focus on the issue of construct validity in relation to the use of perception-based methods to establish conservation impact.We focus on such studies and construct validity for three reasons.First, perception-based methods are applied widely in studies of collaborative forest management (Conley and Moote 2003).Lund et al. (2009) found that 21 of 60 studies used perception-based methods in their review of studies of the conservation impact of popular participation in forest management in developing countries.Second, perception-based methods constitute a low-cost and low-technology approach to forest assessment, implying that they are a potentially useful tool in research and development projects in developing countries.A number of studies have observed that approaches to assess ecological indicators that build upon local knowledge and perceptions may be highly cost effective (Danielsen et al. 2005, Anadón et al. 2009).These advantages of perception-based methods should be viewed in relation to inventories and remote-sensing methods.The latter is also low cost, but thus far is not very effective in mountainous areas and in detecting changes in forest quality beyond changes in forest cover.Another argument for perception-based methods is that assessment methods used in popular participation processes should be easy to use and to comprehend by both local communities and forest officers in developing countries, who, in relation to forests managed under popular participation, will engage in negotiations of rights and responsibilities of forest management based on forest assessments at regular intervals.This is so because of the requirement in many, if not the majority, of processes of popular participation in forest management in the developing world that the managing communities must comply with requirements in relation to ecological sustainability (Ribot 2004, Lund andTreue 2008).Finally, although widely used and potentially important, it could be argued that perception-based methods are particularly prone to problems in relation to construct validity because of the subjective and opinion-based nature of the data produced by these methods.People may perceive the condition of the same forest very differently for reasons of cognition and differences in understanding of the concept of "forest condition.&#8221 Further, people may answer strategically to questions about forest condition for instrumental or normative reasons.
We explore how perception-based methods have been implemented to assess the contribution of popular participation to forest conservation with the overall purpose of finding evidence of tests of various approaches that may guide future studies.We accomplish this by reviewing the studies' use of perception-based methods guided by three research questions.First, what are the respondents asked to assess, i.e., what indicators are used?We explore whether the studies assess the current status and/or trends/development in forest condition and what indicators are used, i.e., whether the respondents are simply asked to assess forest condition/development or whether it is broken down into indicators such as stem density, average tree size, etc.Second, who is asked?We explore whose perceptions are elicited, e.g., the local management body, women, men, forest officers, etc. Third, how are respondents' perceptions validated?We explore whether perception-based assessments are validated through triangulation, i.e., asking multiple individuals or groups the same questions, or inventories, i.e., structured measurements in the forest.We look at how the results of different assessments are aggregated within an individual study to yield a final estimate.

METHODS
We began the review with a literature search in the bibliographic database Web of Science, using the search criteria: "local; participation; community; decentralized; indigenous; village; common; joint" http://www.ecologyandsociety.org/vol15/iss3/art5/combined with "forest; woodland."The review was then expanded by adding any relevant studies referenced in the articles identified in the initial search, as well as studies identified through the authors' collegial networks.The main inclusion criterion was that the article should report empirical data on forest management or conservation aspects gathered by perception-based methods and from a forest area managed with some degree of popular participation, encompassing local management setups formally supported by legislation or existing in spite of legislation, but in lieu of low enforcement efforts by the State.In this context, empirical data are understood as information collected through a structured method that is reported in the study.Hence, articles that casually pass a comment on the status or trend in forest condition based on the authors' own perceptions or anecdotal information were excluded (e.g., Klooster and Masera 2000, Webb and Gautam 2001, Bray et al. 2003, Matta et al. 2005, Southwold-Llewellyn 2006).Because our focus was the methods applied by the studies, metastudies such as those by Gibson et al. (2005), Hayes (2006), and Nagendra and Gokhale (2008) were also excluded.Finally, only peer-reviewed articles in English-language journals were included.This yielded 19 articles (Agrawal and Yadama 1997, Varughese and Ostrom 2001, Conroy et al. 2002, Nagendra 2002, Husain and Bhattacharya 2004, Misra and Kant 2004, Gautam and Shivakoti 2005, Nagendra et al. 2005, Pérez-Cirera and Lovett 2005, Adhikari and Lovett 2006, Agrawal and Chhatre 2006, Bajracharya et al. 2006, Meshack et al. 2006, Antinori and Rausser 2007, Balooni et al. 2007, Gautam 2007, Palmer and Engel 2007, Thoms 2008, Zulu 2008).Although we cannot claim to have made a complete review of all potentially relevant studies, we think that we have covered enough ground to be able to provide an analysis that reflects the status of this research area.
Of the 19 studies, eight are from Nepal, six from India, two from Mexico, and one each from Tanzania, Malawi, and Indonesia.Seven of the studies use the approach and methods laid out in the "field manual" of the International Forestry Resources and Institutions (IFRI) Program at Indiana University in Bloomington, Indiana, USA (International Forestry Resources and Institutions 2008): Varughese and Ostrom (2001), Nagendra (2002), Gautam and Shivakoti (2005), Nagendra et al. (2005), Agrawal and Chhatre (2006), Balooni et al. (2007), and Gautam (2007).However, the extent of rigour in the employment of this detailed set of research instruments that were developed over more than a decade varies among these studies.For instance, Agrawal and Chhatre (2006) had a sample size of 95 forests and did not apply the full set of research instruments.

RESULTS
All 19 studies assessed the status and/or trend in forest condition using perception-based methods.Four studies assessed only the status of forest condition, eight assessed only the trend in forest condition, and seven assessed both status and trend in forest condition using perception-based methods (Table 1.)

What are the indicators?
For the status of forest condition, various indicators of forest condition were applied.The IFRI protocol requires that a forestry professional who has assisted in an inventory of the forest assesses the condition of forest as compared to the topography and ecological zone in which the forest is located.Specifically, the forester assesses the density of vegetation, species diversity, and commercial and subsistence value on a five-point Likert scale (International Forestry Resources and Institutions 2008).Four studies simply required respondents to assess "forest condition" (Agrawal and Yadama 1997, Conroy et al. 2002, Agrawal and Chhatre 2006, Thoms 2008).Other indicators included forest area (Husain andBhattacharya 2004, Antinori andRausser 2007), forest area with commercial potential (Antinori and Rausser 2007), forest degradation (Pérez-Cirera and Lovett 2005), forest regeneration (Meshack et al. 2006), canopy cover (Misra and Kant 2004), biodiversity (Antinori and Rausser 2007), and various indicators of environmental quality (Meshack et al. 2006, Palmer andEngel 2007).
In a few studies, respondents were asked to evaluate intermediary variables between management regime and forest condition, such as the occurrence of illegal harvesting (Antinori and Rausser 2007), indicators of logging impacts on the environment (Palmer and Engel 2007) 2008).Otherwise, the indicators resembled those used to assess the status of forest condition.
We were unable to evaluate whether there is evidence that some indicators work better than others because none of the studies in our sample tested multiple different approaches.

Whose perceptions are elicited?
The IFRI studies asked a forestry professional who had participated in an inventory of the forest in question to assess the status of the forest condition, whereas the trend in forest condition was assessed by forest users (International Forestry Resources and Institutions 2008).The other studies obtained assessments from forestry experts (Misra and Kant 2004), households (Bajracharya et al. 2006, Zulu 2008), nongovernmental organization representatives and government officials (Peréz-Cirera and Lovett 2005), and forest councils (Agrawal andYadama 1997, Balooni et al. 2007), or the authors themselves estimated the indicators (Conroy et al. 2002, Adhikari andLovett 2006).Agrawal and Chhatre (2006) asked six different groups to assess "forest condition": upper and lower caste men and women (four groups), forest department guard (one group), and enumerator group (one group), and averaged the responses into a single measure.Unfortunately, none of the studies presented tests of the variation and/or relative accuracy of estimates provided by different groups of people.

How are perceptions validated?
Of the 27 instances of perception-based assessment of either status or trend in forest condition in our sample of 19 studies, 13 were validated in one way or another (Table 1).The approaches to validation included: triangulation of perception-based assessments by asking multiple groups of individuals to assess forest condition and/or development (Agrawal and Chhatre 2006), checking management plans and other records (Conroy et al. 2002), and doing forest walks and/or inventories (Gautam and Shivakoti 2005).The most common validation approach was to ask more than one group to assess the same indicator (Table 1).Generally, however, the results of these validation approaches and how the estimates from different sources were combined to yield a final estimate of status or trend in forest condition were not reported.In some cases, the reader is referred to previous research for more details on the validation approaches (e.g., Varughese and Ostrom 2001, Bajracharya et al. 2006, Balooni et al. 2007).
Four studies provided a basis for assessing the construct validity of the perception-based method.
Varughese and Ostrom (2001) asked villagers and forest officers for their perception of forest condition development in 18 forests: worsening, stable, or improving.Varughese (1999) provided details of validation of the assessment of forest condition development in six forests using repeated forest inventories.In three of the six sites, the two http://www.ecologyandsociety.org/vol15/iss3/art5/approaches yielded similar results.In their comprehensive study of 95 forests in India, Agrawal and Chhatre (2006) used an average of the perceived value of forest condition (measured using a Likert scale) by six groups (including the enumerator group), and validated these perceptions using forest plot measurements of stem density and tree diameter for 30 of the forests.They found a correlation between perceived value and plot measurements of 0.68.The assessment of indicators of forest condition by forest officers in relation to the two case-study forests reported by Gautam and Shivakoti (2005) was confirmed by the inventory results, as was a similar comparison in the two casestudy sites reported in Nagendra (2002).Nagendra et al. (2005) reported on repeated inventories from three sites in Nepal, but unfortunately, did not report users' assessment of perceived change in forest condition over time, which would have allowed for a validity assessment, although they state that such perceptions were recorded during the field work.

DISCUSSION
The reviewed studies generally did not attend to the issue of construct validity in relation to each particular approach to perception-based assessment.Agrawal and Chhatre's (2006) triangulation using inventory information provides some support for that particular approach, whereas the relatively poor fit between the inventory and perception-based methods found by Varughese and Ostrom (2001) is less supportive.Furthermore, none of the studies provided documentation, e.g., references to other applications, of the construct validity of perceptionbased methods as applied to assessments of forest condition in the context of popular participation in forest management.The scarcity of evidence implies that support for the construct validity of these methods must be found outside the literature in which it is applied.
The perception-based approaches for assessing forest condition, as applied here, have many similarities with the approaches described in the literature on LEK (e.g., Gilchrist et al. 2005), as well as that on local approaches to assessment and monitoring (e.g., Danielsen et al. 2005, Setty et 2008).Huntington (2000), Sáenz-Arroyo et al. (2005a, 2005b, 2006), and Gerhardinger et al. (2009) also provide supportive evidence in relation to fisheries.However, it is unclear whether these results can be transferred to land-based ecosystems.
In particular, there seems to be a gap in the literature on LEK concerning forestry.Hellier et al. (1999:869) compared estimates of change in vegetation cover in two communities obtained from rapid rural assessment exercises with aerial photographs and satellite images and found that the disparity between the two estimates "indicates the need for caution in the use of indigenous knowledge for this purpose."Setty et al. (2008) found complementarity between LEK and scientific approaches when evaluating a participatory resource monitoring system for harvesting of a nontimber forest product (Phyllanthus spp.) by indigenous Soliga harvesters in the Biligiri Rangaswamy Temple Wildlife Sanctuary in South India.This system, which was developed and used over a 10-year period, included visual estimates of fruit production made by harvesters at the forest level, which were found to correspond well with those obtained using scientific transect measurements.Setty et al. (2008) attributed the good accuracy to two factors: visual estimates represented a consensus among 10 to 15 experienced harvesters in each village, and the people making the estimates were largely the same ones during the entire monitoring and evaluation program.This long-term participation, contributing to adaptive management of nontimber forest products, was maintained by ensuring adequate incentives, including resource tenure for monitoring activities (Setty et al. 2008).
High levels of knowledge among local stakeholders, however, are no guarantee of local consensus on this knowledge.There can, for various reasons, be substantial differences among local stakeholders' perceptions of the same trend in resource status or availability, and that makes perception-based studies vulnerable to various sources of bias.Gilchrist et al. (2005) interviewed Inuit people to assess marine bird population changes and compared that information with empirical data derived from scientific studies of the same populations.They found that the accuracy of http://www.ecologyandsociety.org/vol15/iss3/art5/information often varied among interviewees and that interviewees generally gave inaccurate quantitative assessments of population changes.
Analyzing the perceptions of various stakeholders and their proposed management solutions in the context of forest resource decline in Sissili province of southern Burkina Faso, Paré et al. (2010:289) found that the stakeholders identified a relatively large number of forest species and their uses, indicating a fairly high level of local knowledge; however, "there [were] substantial differences among stakeholder groups in terms of their opinion about sustainable forest management of the province's forest resources, which could be explained by the level of stakeholder experience in forest resource management issues."Silvano et al. (2005) investigated the perceptions of stream health for farmers with landholdings down to the Macabuzinho River, Brazil, using structured questionnaires.Farmers failed to estimate properly the stream's ecological status, particularly water quality and quantity.The authors suggest that this might be because the farmers do not depend on the river for drinking water; that is, the authors suggest the general hypothesis that knowledge is highly related to the use of a particular product or service from an ecosystem.
Peoples' perceptions of landscapes are also influenced by their historical development.Dhubháin et al. (2009) found striking differences in peoples' perceptions of amenity, recreation, and economic functions of forestry in two sites in Shillelagh and Newmarket, Ireland, which had similar forest cover in terms of species composition and extent, but differences in demographic characteristics and history of forestry.Participatory management processes also may influence perceptions of natural resources through changes in human and social capital in the evolution of participatory processes (Pretty and Ward 2001).More generally, Ingold and Kurttila (2000), who studied perceptions of weather among Sami people in Finland, argue that knowledge is intimately connected to personal experience and life history.The implications of this are that individual perceptions will vary, not only because of interests in specific resources, but also because of context, including seemingly unrelated experiences in individuals' lives.
Our review of the results of these varied experiments and studies indicate that people can assess changes in their environment.However, there are limits to the usefulness of this ability in estimating the status and trends of natural resource condition and availability.This limitation has several causes.First, variation in peoples' knowledge about the condition of natural resources may be attributed partially to variation in their individual relations to the natural resources, e.g., use, dependence, tenure, and experience in resource management (Ingold and Kurttila 2000, Pretty and Ward 2001, Silvano et al. 2005, Setty et al. 2008, Paré et al. 2010).Second, differences in cognitive abilities, both in relation to understanding what one is being asked to assess and actually being able to assess it, cause variation.Third, resource management is often politicized, and competing claims to legitimate access and resource tenure rights may shape responses.
Based on these findings, we argue that further experimentation and testing of LEK in the context of evaluating conservation impact in processes of popular participation in forest management are needed for several reasons.First and foremost, the evaluation of conservation impact in the context of these processes is a contentious and political issue.
In our experience, it is standard that the management rights to locally managed forest resources can be taken over by higher authorities if management fails to meet silvicultural and environmental criteria specified in national legislation and/or management plans (e.g., Ribot 2004, Lund andTreue 2008).Hence, in cases in which conservation impact is negative, asking local communities to evaluate their conservation impact is asking them to provide evidence that can be used to reverse their management rights.Hence, one would expect a high risk of strategic responses; communities will tend to exaggerate positive and mask negative conservation outcomes.Second, our review of the literature on local approaches to assessment and monitoring and LEK found few studies with a specific focus on forests.We think that forest ecosystems have special characteristics that should caution against the uncritical transfer from other applications of experiences about the validity of LEK.Forests are more complex than grasslands, for instance.And whereas water quality in a river can be assessed from any location along it, determining the condition of vast forest areas requires measurements in many locations.
A word of caution is needed, as our point is not to say that LEK has no role in natural resources management before it has been subjected to scientific scrutiny.(Davis and Wagner 2003).

Speculation
With this review, we have merely touched upon the issue of methods to elicit LEK, sometimes on the basis of rather short and rudimentary descriptions in the publications reviewed.Hence, we were not able to go beyond the level of detail reported in the publications.One issue we were not able to evaluate is the implementation of research tools.Lewis and Sheppard (2006), for instance, show that the results of eliciting LEK are highly dependent upon how it is done.Showing people map or landscape visualization, as Lewis and Sheppard (2006) did with the Cheam First Nation community in the Fraser Valley in British Columbia, Canada, to evaluate the effectiveness of landscape visualization relative to conventional mapping media in eliciting perceptions and understanding of forest management issues, or doing a forest walk before asking questions affects the answers given.An example of an interesting approach is the studies that followed the IFRI protocol, whereby a professional forester is asked to assess forest condition after having participated in an inventory of that forest (e.g., Nagendra 2002, Gautam and Shivakoti 2005, Nagendra et al. 2005).Such an approach should enable more informed answers, but it would be interesting to test whether this is actually the case.Further, we think that asking respondents to assess specific components of forest condition, rather than forest condition per se, runs less risk of having mulitple respondents interpret the question in different ways.Finally, we think that asking respondents to assess indicators of forest condition that are directly related to their daily livelihood activities, e.g., indicators of the availability of firewood for subsistence use, might provide more informed answers than asking them to assess more abstract indicators such as biodiversity or forest degradation.All of these ideas are subjects for future research.

CONCLUSION
We reviewed studies that used perception-based methods to evaluate the conservation impact of popular participation in forest management.We found that the construct validity of these methods is questionable, as very few studies investigated this aspect, and those that did had varying outcomes.Further, the studies generally provided little documentation of their methods and how they were applied in the research process.A quick review of the literature on local ecological knowledge found some support for the notion that people can assess changes in their environment.However, the contexts from which these experiences were drawn are different from popular participation processes in forest management.We therefore argue that the results are not directly transferable to forest management.Accordingly, there is scope for more research on assessment methods that can be used by local communities and forest departments to evaluate forest condition and development in sufficient detail that can be used in negotiating rights and responsibilities in this mode of forest management.Such methods should provide reliable data and be low in cost to facilitate assessments covering large forest tracts.Thus, we maintain that refinement and validation of perception-based methods is a potentially promising field of research.If these methods are refined and validated to provide reliable estimates under diverse conditions, they would be a very efficient and low-cost assessment tool.In the absence of such refinement and validation, we caution against the continued use of perception-based methods.As Lynam et al. (2007:5) argue, "given the complexity of natural resources and their management, picking the right tool does not guarantee that the data desired will be produced, but selecting the wrong tool does make success unlikely."

Table 1 .
, indicators of nontimber forest product availability (Husain and Bhattacharya http://www.ecologyandsociety.org/vol15/iss3/art5/Overview of approaches to validation in 11 studies that measured the status of forest condition and 16 studies that measured the trend in forest condition, from a total of 19 studies.
For trends in forest condition, the IFRI studies asked forest users to estimate changes in forest area and in the density of trees, shrubs, and ground cover (International Forestry Resources and Institutions fisheries that seems to support the hypothesis that fishers are able to detect changes in stocks of fish over time (e.g., small-scale fishers in Guinea, West Africa: Poizat and Baran 1997; fishers in the Gulf of California: Lozano-Montes et al. Paré et al. 2010) local traditional knowledge (e.g.,Paré et al. 2010), although to a lesser extent.It is possible to draw some lessons regarding the validity of perception-based approaches from these bodies of literature.There is much literature on LEK in relation to On the contrary, LEK holds http://www.ecologyandsociety.org/vol15/iss3/art5/