Using Consensus Analysis to Assess Mental Models about Water Use and Management in the Crocodile River Catchment , South Africa

The content, structure, and distribution of mental models can be elicited and measured using a variety of methods. In this article we explore a method for eliciting mental models within the context of water use and management in South Africa. This method is consensus analysis, a technique developed in cognitive anthropology. We used it to analyze qualitative data from semistructured interviews, pilesorts, and questionnaires to test quantitatively the degree of sharing and diversity of mental models within and across social groups. The consensus analysis method focused on comparing the mental models of two key stakeholder groups in the Crocodile River catchment in South Africa, i.e., conservationists and irrigators, to better understand the level of consensus between these groups. We specifically investigated the level of agreement regarding: (1) major water users of the Crocodile River, (2) causes of the current problems with flows in the river, (3) consequences of the river not flowing, and 4) priorities for future use. We discuss the results and examine the strengths and challenges of consensus analysis for eliciting and measuring mental models. We also evaluated the usefulness of this method in assisting natural resource managers to identify strategies for improving integrated management of water resources.


INTRODUCTION
It has long been recognized in the field of natural resource management that creating shared understanding among people is critical if collective decision-making processes and actions aimed at resolving social-ecological problems are to be reached (Röling 2002, Adams et al. 2003).This is particularly true in contested arenas where a plurality of views may hinder the development of solutions.As such, it is vital that natural resource management practitioners, and other key partners, have access to approaches and tools that enable them to better comprehend different social groups' (and individuals') beliefs, opinions, and knowledge.It is particularly important that they gain an understanding of the similarities and differences in perspectives within and among different groups regarding the natural resource problem at hand, including its causes and solutions.With this knowledge, natural resource management practitioners are better placed to identify the most appropriate strategies for facilitating negotiation and fostering collective action for improved social and ecological outcomes.
One approach to "getting into people's heads" is the notion of mental models.A mental model refers to the internal representations of external reality that people carry around with them, and it forms a cognitive basis for reasoning, decision making, and behavior (Jones et al. 2011).According to Quinn (2005), mental models have a social cognition component.Over time people internalize experiences they share with the group(s) to which they belong to create cultural meaning, or a shared mental model.They use these collective mental models to perceive and relate to the world around them.Mental models draw on a broad set of theoretical and applied fields which include 1. Are the mental models regarding water use and management in the Crocodile River catchment: q Shared?(Is there consensus?)q Weak? (Is there "weak agreement", or tendency towards agreement but with considerable variability?)q Multicentric?(Are there multiple group mental models, either nonopposing/overlapping or diametrically opposed/contested?) q Fragmented? (Is there no agreement?Does each individual have a different mental model?) 2. What is the content and structure (i.e., similarities and differences in how components are organized) of the mental models?
A second objective of the paper is to evaluate the usefulness of the consensus analysis method for decision makers and practitioners working towards sustainable, equitable, and efficient water management under South Africa's National Water Act, Act No. 36 of 1998(Republic of South Africa 1998).The central objective of the National Water Act, popularly paraphrased as "some, for all, forever", is the decentralization of water management to the catchment (river basin) scale, primarily through the establishment of water management areas and catchment management agencies.Highly progressive and ambitious at the time it was passed into law, the National Water Act calls for, among other things, ecological integrity of water resources; equity in access to water resources, benefits, and services; and participation of stakeholders in decision-making processes about water resources (see Republic of South Africa 1998, Wester et al. 2003).However, more than ten years later, the implementation of the National Water Act, and of the catchment management agencies more specifically, has proven to be a difficult and slow process.Among the problems encountered is the stronghold of old, primarily bureaucratic, and topdown institutional structures and processes (Rogers et al. 2000).These appear to have further institutionalized the power of well-organized groups (primarily mining and industry sectors, water suppliers in larger towns, and commercial farmers) in the catchment management agency establishment processes (Wester et al. 2003, Goldin 2010).By contrast, the rural poor, the small-scale http://www.ecologyandsociety.org/vol16/iss1/art45/irrigation sector, and the local domestic water entities have tended to be marginalized (Wester et al. 2003, Goldin 2010).This has been compounded by confusing public engagement processes (du Toit and Pollard 2008).Combined, these have led to disagreements, conflicts, and delays in the development of catchment management agency proposals and to growing disillusionment and participation fatigue among stakeholders (Wester et al. 2003, du Toit andPollard 2008).As highlighted by Wester et al. (2003:809), "there are many conflicting views-and real conflicts-among stakeholders over water issues, which need to be articulated clearly as part of the CMA [catchment management agency] establishment process" (italics added).MacKay et al. (2003:353) also emphasize the need for "new institutions, new tools, a new mindset and a robust implementation plan" (italics added) if South Africa's "some, for all, forever" vision for water management is to be achieved.Consensus analysis may be one practical tool that can assist water resources decision makers, managers, users, and other key partners, in the development and establishment of the catchment management agencies in the Crocodile River catchment and other catchments in South Africa.

CULTURAL CONSENSUS MODEL AND CONSENSUS ANALYSIS
Consensus analysis is a technique used in association with the cultural consensus model in anthropology (Romney et al. 1986, Romney et al. 1987, Romney 1999).While the model, and the consensus analysis method, do not use the term mental model, they use concepts that are similar such as schema, knowledge domains, and cultural domains.The model defines culture as knowledge shared by a group (Romney 1999).The assumption is that cultural domains-comprised of systems of knowledge which jointly refer to a single conceptual sphere that individuals use to interpret and respond to the world of experience (Weller andRomney 1988, Handwerker 1998)-are learned and shared.The extent of sharing on the particular items comprising a domain can vary and consensus analysis allows investigation of the content and distribution (or sharing) of words and concepts among individuals.Consensus analysis thus enables one to "infer whether there is a culturally central or a more diversified understanding of the domain within the local knowledge system" (Caulkins and Hyatt 1999).
The cultural consensus model and consensus analysis have their origins in cognitive and mathematical anthropology and in psychometrics (Romney and Weller 1984, Romney et al. 1986, Romney et al. 1987, Romney 1999).Consensus analysis consists of an analysis of "informant responses to systematic interview questions" (Romney et al. 1986) which statistically assesses the amount of agreement among a group of people about a domain of culture (or knowledge).More specifically, consensus analysis involves a factor analysis of an intersubject agreement matrix that has been corrected for guessing and for Bayes' theorem to produce statistical estimates that address three questions (Romney et al. 1986 The model, and by extension consensus analysis, makes three critical assumptions (Romney et al. 1986, Weller 2007): 1. Respondents share a common culture (i.e., there exists a single set of answers to the question(s) being asked).

2.
Answers are provided by each respondent independently of all other respondents (not by a group nor in consultation with others).

3.
The questions asked should all be on a single topic and at the same level of difficulty.
If consensus is not found, one can assume that one or more of these assumptions was not met.http://www.ecologyandsociety.org/vol16/iss1/art45/

Study area: the Crocodile River catchment
This study was conducted in the Crocodile River catchment in South Africa (Fig. 1).The Crocodile River drains a catchment of about 10 400 km 2 , with a total main-stem length of approximately 320 km.1998, Rogers et al. 2000, MacKay et al. 2003, Wester et al. 2003, Goldin 2010;and D. du Toit, H. Biggs, and S. Pollard, unpublished manuscript).

Data collection and analysis: consensus analysis
Consensus analysis can be conducted through one or more phases of data collection and analysis.Typically, when there is little information available about a given domain, or when a domain is potentially not very coherent (i.e., heterogeneous or complex), two phases of data collection and analysis are undertaken.The objective of the first phase is to elicit more knowledge about the domain under inquiry and this generally entails open-ended and semistructured questions such as freelisting.The second phase uses the responses given in the first phase to create a set of structured questions that are analyzed using consensus analysis.These include dichotomous, ordering, and similarity tasks such as pilesorts, and questionnaires (multiple choice, yes/ no and true/false, paired-comparison, and triadic) (see Weller andRomney 1988, Borgatti 1996b).
Unlike most conventional survey methods, consensus analysis can produce a high level of statistical confidence with small sample sizes (4 to 30 individuals) (Romney et al. 1986, Weller andRomney 1988:77).In our study, two phases of data collection and analysis were undertaken with 33 individuals and 43 individuals, respectively (Fig. 2).We first carried out semistructured interviews.The data from these interviews were used to construct yes/no questionnaires and pile-sorting exercises which generated the data that were then analyzed using consensus analysis (as well as multidimensional scaling and hierarchical cluster analyses).The diagnostic criteria we used as a guide to interpret our data are presented in Table 1.The data collection and analysis for phases 1 and 2 are described more fully in the appendix.

Theme 1: water users
From the first round of interviews with 33 people, a wide and diverse range of water users were identified.Interviewees listed 25 different water users in the Crocodile River catchment which ranged from diverse social (human) groups to plants and animals (Table 2).
The most frequently cited water user was commercial farmers, who were mentioned by 97% (i.e., 32) of the interviewees.Commercial farmers were also perceived as the most salient water users -i.e., interviewees listed them near the top of the list, perceiving them as the "most important" water users or water users with which they were the most familiar.
The yes/no questionnaire, comprised of a list of 18 water users (all water users that had been mentioned more than once in the first phase of interviews), was filled out by 43 people (16 irrigators and 27 conservationists).The data were analyzed in Anthropac (Borgatti 1996a) with the consensus analysis module.The results of the analyses ( As indicated in Table 3, for all three groups, the ratio between the first and second eigenvalues was below the 3 to 1 ratio, thus indicating that there was not a single-factor solution or a shared mental model regarding the major water users of the Crocodile River catchment.For example, the ratio for the group as a whole (irrigators and conservationists) was 2.16 to 1.At first glance, this ratio suggests that the group as a whole mildly agreed with each other or shared a "weak" mental model (see Table 1 for guiding diagnostic criteria).However, there was high variability in individual competence scores (SD±0.53)and a very low average (group) competence score of 0.20, the latter indicating that conservationists and irrigators as a group shared only 20% of the beliefs regarding who are the major water users.In addition there were 17 individuals with negative scores.People with negative scores have a very different or unusual understanding of the issue under inquiry, indicating the potential presence of more than one group with different beliefs or mental models (Weller 2007).A closer inspection revealed that seven of the individuals with negative scores were irrigators and the remaining ten were conservationists.As we did not collect any sociodemographic information about our interviewees, it was not possible to explore this difference further.However, the absence of two or more factors of similar strength suggests that there were not subgroups (e.g., men and women, irrigators and conservationists, etc.) holding distinct mental models.Moreover, consensus analysis performed separately on the subgroups of irrigators and conservationists also indicated a lack of consensus (more specifically, "little to no agreement", see Table 1) with an eigenvalue ratio between the first and second factor of less than 2.0 to 1 for each group.
The lack of agreement among the group as a whole (and within each group) was checked, visually, with multidimensional scaling (MDS) (Fig. 3 for whole group only) which showed irrigators and conservationists scattered across the map.This was confirmed with hierarchical clustering (not shown) which showed no distinctive clustering of people.http://www.ecologyandsociety.org/vol16/iss1/art45/Fig. 2. Phases of data collection and analysis, for Crocodile River catchment data.
We also were interested in assessing if irrigators and conservationists, as a whole group and as separate groups, conceptualized water users in similar categories.A MDS map of the similarities among water users according to the whole group is shown in Fig. 4. The stress score of 0.082 was below the 0.172 stress value cutoff suggested by Sturrock and Rocha (2000) for an 18-item representation in three dimensions.Looking at the MDS map, in the upper center and right corner, rural (rural populations) and urban groups (urban areas, municipal and city authorities, domestic users, black townships, and housing and residential development sector) appeared close together, meaning that conservationists and irrigators perceived them as being more or less similar.Another grouping appears in the upper left corner, with the tourism industry, recreational users (trout fishers), aquaculture, environmental stakeholders (the environment and Kruger Park), and international groups (Mozambique) clustered together.In order to further explore why these groups were perceived to be similar, additional interviews with conservationists and irrigators would be needed.Ratio between the first and second eigenvalue factors is ≥3.0 to 1.

No consensus
If ratio between first and second factors is (Caulkins 2004): q 2.0 to 2.9 to 1, then have weak agreement.q <2.0 to 1, then have little/no agreement.
q Presence of two (or more) factors of similar strength, then have two (or more) mental models (either overlapping/nonopposing, or contested).
Average (group) competence score q Scores normally range from 0 to 1. q The higher the average competence score, the higher the consensus.
q If <0.3, then assumptions of cultural consensus model probably not met, i.e., no consensus (Romney et al. 1986).

2.
To what extent does each respondent agree with the group's cultural beliefs (mental model)?
Individual competence score q Scores normally range from 0 to 1. q The higher the score, the more knowledgeable and in agreement with the group is the person.

q
A person with a negative score indicates that s/he has a very different or unusual understanding of the issue.
3. What are the "culturally appropriate answers" to each item mentioned by the group?
Answer key q Meaningful only if the data show consensus.
q Tells you the culturally agreed upon or "correct" answers, i.e., what the group agreed on were the components of the domain (or mental model).

Theme 2: causes of problems with current flows
A total of 33 factors were mentioned as responsible for the causes of the problems with the current flows in the Crocodile River (Table 4).Of these, the most frequently mentioned was illegal use of water, which was mentioned by 31% (10 people) of the 33 people interviewed.This was closely followed by low rainfall (25%), forest plantations (22%), commercial agriculture (22%), and Kwena dam (19%).If the frequency of times Kwena dam was mentioned was combined with the frequency of mentions of dams in general (16%), then dams were mentioned the most frequently (35%).However, although dams were most frequently mentioned, they were not perceived as the most important or salient cause of problems with current flows.Illegal use of water was identified as the most salient cause of the problems.
A consensus analysis of the yes/no responses to the question "What are the causes of the problems with current flows in the Crocodile River?" showed similar findings to those found for "major water users".Irrigators and conservationists as a whole group did not hold a shared understanding, or mental model, about the underlying factors causing the problems with the current flows in the Crocodile River (Table 5; ratio between first and second eigenvalues 2.14 to 1, average competence score 0.44 ± 0.43).Similar results were found for conservationists and irrigators as separate groups.
Irrigators agreed with each other only 25% of the time (average competence score 0.25 ± 0.47).Among conservationists, the average level of shared beliefs regarding underlying causes was also low and the answers were highly variable (0.46 ± 0.45).For all three groups, the first to second eigenvalue ratios were greater than 2.0 to 1 (but below the 3.0 to 1 threshold for consensus), suggesting that there may be some tendency towards agreement ("weak agreement").However, again, the presence of several negative scorers (eight for the group as a whole, three for irrigators, and five for conservationists) indicate that there are considerable differences in beliefs regarding the factors that have led to the Crocodile River's current flow problems.This is also evident in the low average competence scores and in the high standard deviation in individual competence scores.
A MDS map (Fig. 5) confirmed that irrigators and conservationists as a group did not have a shared mental model regarding the causes of the river flow problems.The group of irrigators (represented by blue squares) and conservationists (red circles) formed clusters indicating that there may be consensus within each of these groups.We ran MDS maps and hierarchical cluster analyses (not shown) for the two groups separately which showed individuals scattered across rather than in close proximity to each other, indicating a lack of agreement.We also ran MDS maps and hierarchical cluster analyses of the causes of the flow problems in the Crocodile River (not shown) to see if conservationists and irrigators organized problems into separate clusters.Neither conservationists nor irrigators-as separate groups or as a whole group -grouped the list of causes of the river flow problems into any distinct, coherent categories.

Theme 3: consequences of the river not flowing
A total of 34 different negative impacts were mentioned in response to the question "What are the consequences of the Crocodile River not flowing?".
The consequences mentioned covered a broad range of ecological, social, and economic impacts.The impact on biodiversity was mentioned most frequently, i.e., by 30% of the people interviewed (see Table 6).It was also perceived as the most salient or important of consequences.This was followed by impact on the whole economy (24%), on animals (18%), and on Mozambicans (18%).
Because it is not possible to run consensus analysis on unconstrained pilesorts, a MDS map of the 43 people who participated in the pile-sorting exercise suggests that there was no agreement among irrigators and conservationists with respect to the consequences of the river not flowing (see Fig. 6; stress score = 0.132).This is the classic "fried egg" pattern of disagreement noted by Romney and Weller (1984) which shows that individuals in the center of the MDS map had the highest competence scores, whereas those toward the periphery had the lowest.Additional background information about each of the interviewees would have been useful to reveal if those conservationists and irrigators clustered more closely together in the center of the MDS map shared some common characteristic (e. g., higher education levels, lived longer period of time in the region, etc.).We also ran a MDS map and cluster analysis on conservationists and irrigators separately.The MDS map of the conservationists interviewed suggests that they generally agreed with each other with the exception of two individuals (indicated by numbers 24 and 27 in Fig. 7) who had different perspectives from the rest of the group regarding what would happen if the river stopped flowing.The MDS map of irrigators (not shown) seemed to show a weaker level of agreement among irrigators.
We also wanted to know if people had pile sorted the consequences into similar piles.Fig. 8 shows the MDS map of the pilesorts of consequences in two dimensions (stress score = 0.100).The consequences of the Crocodile not flowing were clearly separable into social, economic, and ecological categories (red and black, blue, and green, respectively).
The consequences circled are those that were found to be grouped together in the cluster analysis (Fig. 9).Looking at the x-axis of the MDS map from left to right, it appears that the consequences clustered on the left side are ecological impacts and the consequences clustered on the right side are socioeconomic impacts.The y-axis suggests that the cluster of socioeconomic consequences is further broken down into social impacts (top right-hand corner) to more economic impacts (bottom left-hand corner).Within the cluster of social impacts, there also appears to be a distinction between social impacts that will occur at the level of the catchment and those that will occur at the international scale.
Two items-"less water to use" and "everyone impacted"-did not group well within the social category.Compared to the other consequences, which were very specific, both of these are very broad and potentially vague terms, which may have Similar findings were revealed for irrigators and conservationists separately (not shown, MDS stress scores in two dimensions were 0.090 for irrigators and 0.136 for conservationists).In other words, both groups generally pile sorted the consequences into ecological categories and subcategories of socioeconomic impacts.The only difference between the two was that irrigators tended not to see "negative impacts on tourism" as an economic impact; rather, they saw this as a separate item.

Theme 4: priorities for future use
Over 70 different priorities were listed.The top 40 are presented in Table 7.At the top end, over 40% listed "basic human needs" as a priority for future water use, followed by equity balance which was mentioned by 34%."Water to meet ecological and environmental needs" was mentioned by 28%.
For the three groups, the low ratios (below the 3 to 1 threshold) between the first and second eigenvalue factors, combined with extremely low average competence scores, indicate that there is not a single, shared mental model regarding the priorities for future water use (Table 8).The eigenvalue ratio of 2.21 to 1 for the conservationists interviewed suggests that they may potentially weakly agree with each other.However, the average competence score of 0.065 (i.e., shared only 6.5% of priorities listed) clearly indicates that there is very little agreement among them.For all three groups, the lack of consensus was confirmed with MDS maps and hierarchical cluster analysis (not shown).
We also mapped the priorities, using MDS and cluster analysis, to see if the individuals we had interviewed had pile sorted them similarly, but these analyses did not enable us to identify any clear patterns among the group as a whole or for irrigators as a separate group.For conservationists, however, the MDS map (not shown; 3 dimensions; stress = 0.139) and the cluster analysis results suggest that conservationists distinguished among priorities.They tended to pile sort the priorities focused on water management for conservation or sustainability together, and to pile sort the priorities for water management for human activities in another group.

Summary
We were interested in eliciting and comparing the mental models of conservationists and irrigators with respect to: (1) who were the major water users of the Crocodile River catchment, ( 2) what were the causes of the current problems with flows in the Crocodile River, (3) what would be the consequences of the river not flowing, and ( 4) what should be the priorities for future use.Were the mental models regarding water use and management in the Crocodile River catchment shared, weak, multicentric, or fragmented?Among the irrigators and conservationists we interviewed there was no consensus, as a whole group or as separate groups, regarding major water users, the causes of the river flow problems, and future priorities for the Crocodile River catchment.With eigenvalue ratios of less than the 3 to 1 recommended cutoff for consensus, the groups did not have a shared mental model regarding these issues.While there may not have been consensus, the data suggested that there may be "weak agreement", i.e., a tendency towards agreement but with great variability in beliefs or knowledge (as evidenced by eigenvalue ratios of 2.0 to 2.9 to 1 (Caulkins 2004, on (1) major water users for the whole group, (2) causes of the flow problems for all three groups, and (3) priorities for conservationists.However, in almost all of these cases, the group and individual competence scores were too low and variable, respectively, to conclude that they shared a "weak" mental model.Rather, the mental models in these instances would be considered "fragmented" (Caulkins 2004) or "turbulent" (Caulkins and Hyatt 1999).The exception was the whole group and conservationists with regard to the causes of the problems with the river flows.In both http://www.ecologyandsociety.org/vol16/iss1/art45/Fig. 5.A three-dimensional MDS map of the similarities among respondents, with respect to their identification of the causes of problems with current flows in the Crocodile River (stress = 0.16).
cases, approximately 45% of the group shared similar beliefs regarding the causes.Over time, this weak agreement may increase into consensus or a shared mental model or, alternatively, may weaken further (Caulkins 2004).
Given that we used unconstrained pilesorts to assess the level of agreement regarding the consequences of the Crocodile River not flowing, we were unable to analyze the data with consensus analysis.However, MDS and hierarchical cluster analysis indicated that there seemed to be little agreement within the whole group, some agreement among irrigators, and stronger agreement among conservationists.
What was the content and structure of these mental models?I.e., were there similarities or differences in how items were grouped together by interviewees?With respect to major water users, causes of river flow problems, and future priorities, some items were perceived by irrigators and conservationists as being similar and grouped together.However, some of the groupings of items were difficult to interpret, and additional discussions with interviewees are needed to further explore these clusters.With regards to the consequences of the Crocodile River not flowing, irrigators and conservationists clearly separated them into three broad categories of social, economic, and ecological impacts.

Discussion
As an analytical tool for eliciting and measuring mental models, consensus analysis has both strengths and weaknesses.Its strength lies in its quantitative analysis of people's knowledge and beliefs (qualitative data).It allows one to assess, in quantitative terms, who agrees with whom, about what, and to what degree.Moreover, because it is not an invasive method, i.e., it minimizes the involvement of the researcher via use of prompt Table 6.Results of analysis of freelists of "consequences of the Crocodile River not flowing?",showing frequency with which interviewees identified consequences, the percentage of interviewees who identified that consequence, the average ranking of that consequence, and the salience (Smith's S)  questions and the like, it also permits the elicitation of mental models with minimal interference, and therefore minimal potential influence, by the researcher.
On the other hand, consensus analysis poses significant challenges.Careful preparation is needed in designing and carrying out the two phases of fieldwork.The wording of the questions must be well thought out in order to capture people's beliefs and knowledge regarding conventional truths or judgments and not their personal preferences or histories (Gatewood 1999, Horowitz 2009).If the domain is potentially complex and comprises a high diversity of issues, the first phase of data collection may necessitate time-consuming methods such as open-ended or semistructured interviews.While the data collected from such interviews are rich, it entails content analysis which may bring in bias unless multiple coders are used to cross-check and validate the codes produced (see Guest et al. 2006).http://www.ecologyandsociety.org/vol16/iss1/art45/Fig. 6.A three-dimensional MDS map of the similarities among conservationists and irrigators as to their perceptions of the consequences of the Crocodile River not flowing (stress=0.132) In both phases of data collection, sampling must also be carefully considered.In phase 1, it is recommended to collect data until one reaches either "saturation of issues", i.e., no new issues appear, or an identification of a core set of issues that people seem to agree on.In phase 2, sample size is dependent on the average (group) competence score which is often not known beforehand (see appendix for further details).Moreover, it is important to sample either across differences in life experiences and/or sociodemographic variables such as gender, age, and ethnicity, or to focus on a very specific group (Handwerker 2005).In our research, we cannot exclude sampling problems as a potential explanation for the lack of consensus.Our samples were quite modest by survey research standards but, as previously mentioned, consensus analysis can produce very reliable results from small samples (4 to 30 individuals) (Weller and Romney 1988).However, when sample sizes are small, the average level of competence must be 0.4 or above to obtain reliable results (Weller 2007).While our sample sizes for the whole group (43) and conservationists (27) were on the higher end, our sample of irrigators was small (17).Moreover, the average competence scores were consistently very low (below 0.4) for irrigators, as well as for conservationists and for the whole group in certain domains (major water users and priorities).This suggests that either we violated one or more of the three assumptions of the cultural consensus model (discussed further below) or that our sample sizes were too small.
Other challenges for applying consensus analysis revolve around the analysis and interpretation of the results.As pointed out by Garro (2000) and Handwerker (2002), the three assumptions of the cultural consensus model are highly restrictive.One must ensure that: (1) there is a common culture, or domain of knowledge, that people in the group share; (2) the responses of individuals in a group are given independently of others; and (3) all the questions asked are on a single topic and are at the same level of difficulty.These are often not met in field research (Garro 2000, Handwerker 2002).Our findings (lack of consensus across all four domains -water users, causes of problems, consequences, and priorities) suggest that one or more of these assumptions were not met or that "something else [went] wrong" (Romney et al. 1986:166).Given that the data were collected from independent interviews, it is likely that the lack of consensus was due to the absence of a common culture regarding water users, causes of problems, consequences, and priorities; and/or because questions that varied in difficulty or were conceptually too broad; and/or due to some other factor.Upon further reflection, our assumption of a common culture for each domain may not have been appropriate.For example, when we asked "Who were the major water users in the Crocodile River catchment?"we were assuming that people had mental models of who used the water over the entire catchment.It is possible that the individuals we talked with had mental models of users at different geographical scales when answering this question.In situations where the data show no consensus, such as in our research, an additional challenge is drawing the line between weak agreement and multicentric and fragmented domains (Caulkins and Hyatt 1999).This is complicated by the use of different diagnostic criteria by scholars.Given these limitations, Horowitz (2009) argues that consensus analysis is of most value as an analytical tool in cases where cultural or knowledge domains are not idiosyncratic and have clear boundaries.
These methodological challenges have significant implications for the use of consensus analysis by natural resource management practitioners.For time-and capacity-constrained organizations and individuals, using consensus analysis would initially require working with someone with the technical expertise to set up the field research and analyze and interpret the data.This may not be financially feasible in some cases.And while practitioners can be trained in consensus analysis, the technical complexity of the method can pose a significant barrier to its use as an applied tool for better understanding different groups' mental models and for developing strategies for action.Despite these challenges, consensus analysis has the potential to be a tool of great value to natural resource management practitioners in general, and to the catchment management agencies and supporting organizations in the Inkomati and in other water management areas in Africa.As noted by Wester et al. (2003), Mackay et al. (2003), anddu Toit andPollard (2008), among others, one of the biggest obstacles to achieving South Africa's water management vision is improved engagement processes with diverse stakeholders.Participatory tools, such as consensus analysis, that reveal the plurality of views and provide a vehicle for negotiation through the "interfacing and sharing of knowledge" (Roux et al. 2006) are consequently needed.Mistrust, misconceptions, and misunderstandings -fueled by a long history of social, economic, and cultural differences and injustices-have created significant barriers between South Africa's advantaged and disadvantaged populations, and between scientists and managers, among others (Roux et al. 2006).Consensus analysis can help build bridges between these groups by giving "equal voice" to all (through the qualitative phase 1 of the research) and identifying (through the quantitative phase 2) differences and convergences in mental models.In doing so, consensus analysis can contribute towards "resilient pathways" for achieving South Africa's "some, for all, forever" vision (Bohensky 2008).
Our experience using consensus analysis in the Crocodile River catchment showed the potential of the consensus analysis method.In phase 1 of the http://www.ecologyandsociety.org/vol16/iss1/art45/Fig. 9. Hierarchical cluster analysis of the consequences of the river stopping flowing ('Level' is a measure of the relative distance between concept clusters) research, the freelisting exercise provided a fairly easy and quick way of identifying key issues and their relative importance, thus providing powerful information if used constructively.For example, illegal water use was mentioned as the leading cause of the problems with the flows of the Crocodile River.Exploring this contentious issue further with different groups (e.g., large irrigators and rural poor; national-and catchment-level decision makers, etc.) would be a critical first step if agreement is to be reached on how to best resolve the problem of illegal use and, ultimately, to improve river flows.
Consensus analysis of the pilesorts (phase 2) revealed that the conservationists and irrigators grouped the consequences of the river not flowing similarly.The unambiguous recognition of undesirable social, economic, and ecological changes indicates that they did not like where they were headed.This provides a potentially useful starting point for negotiations around future wateruse priorities.Unfortunately, there was no agreement among the conservationists and irrigators we interviewed regarding future uses of the water (yes/no questionnaire).Under the National Water Act, the national government is legally obligated to "reserve" or ensure the provision of sufficient water for meeting basic human needs and protecting aquatic ecosystems.Allocation of remaining water has yet to be defined but preference will be given to uses that have high social, economic, or environmental value and which promote equity (National Water Act, Act No. 36 of 1998, section 27 (Republic of South Africa 1998)).Thus, building consensus among the diversity of water users is a critical task.It would be interesting to repeat the consensus analysis method after the catchment management agency in the Inkomati Table 7. Results of analysis of freelists of "priorities for future water use?", showing frequency with which interviewees identified priorities, the percentage of interviewees who identified that priority, the average ranking of that priority, and the salience (Smith's S) of the priority.Basin has been in operation, to identify if the actions they take will help foster agreement around future water-use priorities and actions.
Responses to this article can be read online at: http://www.ecologyandsociety.org/vol16/iss1/art45/responses/ http://www.ecologyandsociety.org/vol16/iss1/art45/After completing the questionnaire, each interviewee was asked to participate in two pile-sorting exercises, one focused on 'consequences of the Crocodile River not flowing' and the other on 'priorities for future water use'.For the pile-sorting exercises, the top consequences (those mentioned at least twice in the free lists generated in Phase 1) were written on separate index cards (total of 23 cards).A separate set of cards was created for the priorities for future water use that were mentioned at least twice (three of the 40 priorities listed in the first phase were merged to create a total of 37 cards).Interviewees were first asked to pile sort the 'consequences of the Crocodile River not flowing'.They were given the stack of cards, each containing a single word or phrase (with an identification number written on the back), and asked to organize them into groups or piles on the basis of similarity.They were instructed that there was no right or wrong way to sort the cards and that they could make as many piles as they wanted, with a minimum of two piles.Interviewers noted this information as well as recording separately the identification numbers on the back of each card for each pile that had been created.We repeated the exercise for the cards on 'priorities', but this time constrained them to three piles: priorities that are highly important, of medium importance, and of low importance.
Using Anthropac (Borgatti 1996a), the results of the questionnaire and pile-sorting exercises were converted to aggregate proximity matrices of interviewees and analyzed with consensus analysis (a module in Anthropac).The consensus analysis module applies minimum residuals factor analysis to interviewee responses to determine whether there is sufficient agreement among the persons interviewed to suggest that they share elements of their mental model on a particular issue.The estimates produced by the CA module in Anthropac, and the questions they inform, are summarized in Table 1 (in article).It is important to note that the 3 to 1 ratio between the first and second eigenvalues has been widely accepted in the consensus analysis literature as the appropriate threshold for determining consensus (Romney et al. 1987, Borgatti 1996b, Romney 1999).More recently, several scholars have extended the logic of consensus analysis to the interpretation of data that is below this threshold (for example, Caulkins and Hyatt 1999, Handwerker 2002, Caulkins 2004).The diagnostic criteria we used as a guide to interpret our data are presented in Table 1 (in article).
In addition to the consensus analysis module, we used non-metric multi-dimensional scaling (MDS) and Johnson's hierarchical clustering.
Although MDS is generally not necessary in consensus analysis as it may distort the data (Weller 2007), along with cluster analysis, it is a useful complementary visual tool that is commonly used with consensus analysis.Both MDS and cluster analysis facilitate visualizing the degree to which people share words or concepts.In the map produced by MDS, people who are in closer agreement appear closer together; people who have different understandings, or mental models, on the issue appear farther apart.MDS produces a Kruskal stress score, which reflects the degree to which the MDS model represents the data.The lower the score (closer to 0), the better the representation.It is recommended that 2-dimensional MDS be used but, if shown to reduce the stress, 3-dimensional MDS is acceptable.All of the MDS graphics presented in this paper had stress scores that were below than the cutoffs suggested by Sturrock and Rocha (2000).Cluster analysis is often used to interpret the groupings of people in the MDS map.It produces a schematic diagram of clusters of people in accordance to their similarity.While our main interest was to assess the level of consensus among people, we were also interested in identifying the similarities and differences among the items pile sorted or grouped (into yes/no categories) in the questionnaire for each of the domains (major waters users, problems, consequences, priorities).To this end, we also used non-metric MDS and Johnson's hierarchical clustering.

Fig. 3 .
Fig. 3.A three-dimensional nonmetric MDS map of similarities among respondents, with respect to their identification of major users of the Crocodile River catchment (stress = 0.121).

Fig. 4 .
Fig. 4. A three-dimensional MDS map of similarities among major users of the Crocodile River catchment, as identified by irrigators and conservationists (stress = 0.082).

Fig. 7 .
Fig. 7.A two-dimensional MDS map of conservationists, according to similarity of pile-sorting consequences if the Crocodile River stops flowing (stress = 0.128).

Fig. 8 .
Fig. 8.A two-dimensional MDS map of consequences, based on how similarly they were perceived by irrigators and conservationists (stress = 0.100).
It forms the southern boundary of Kruger National Park, and is the largest tributary of the Komati River.

Table 3
Map of the Crocodile River within the Inkomati Water Management Area, South Africa.

Table 2 .
Results of analysis of freelists of "major users of water", showing frequency with which interviewees identified users, the percentage of interviewees who identified that user, the average ranking of that user and the salience (Smith's S) of the user.

Table 3 .
Results of the consensus analysis of the yes/no questionnaire "Who are the major users of water?" (18 items).

Table 4 .
Results of analysis of freelists of "causes of the problems with current flows in the Crocodile River", showing frequency with which interviewees identified causes, the percentage of interviewees who identified that cause, the average ranking of that cause, and the salience (Smith's S) of the cause.

Table 5 .
Results of the consensus analysis of the yes/no questionnaire "causes of the problems with current flows in the Crocodile River?" (31 items).
of the consequence.

Table 8 .
Results of the consensus analysis of the pile sort for "priorities for future water use in the Crocodile River not flowing?"(37 items).