Ecology and Society
Home | Archives | About | Login | Submissions | Subscribe | Contact | Search
 E&S Home > Vol. 16, No. 1 > Art. 45

Copyright © 2011 by the author(s). Published here under license by The Resilience Alliance.
Go to the pdf version of this article

The following is the established format for referencing this article:
Stone-Jovicich, S. S., T. Lynam, A. Leitch, and N. A. Jones. 2011. Using Consensus Analysis to Assess Mental Models about Water Use and Management in the Crocodile River Catchment, South Africa. Ecology and Society 16(1): 45. [online] URL:

Research, part of Special Feature on Mental models in human - environment interactions: Theory, policy implications, and methodological explorations

Using Consensus Analysis to Assess Mental Models about Water Use and Management in the Crocodile River Catchment, South Africa

Samantha S. Stone-Jovicich 1, Timothy Lynam 1, Anne Leitch 2,3 and Natalie A. Jones 4

1CSIRO Ecosystem Sciences, Townsville, 2CSIRO Ecosystem Sciences, Brisbane, 3ARC CoE Coral Reef Studies, James Cook University, 4University of Queensland, School of Rural and Natural Systems Management


The content, structure, and distribution of mental models can be elicited and measured using a variety of methods. In this article we explore a method for eliciting mental models within the context of water use and management in South Africa. This method is consensus analysis, a technique developed in cognitive anthropology. We used it to analyze qualitative data from semistructured interviews, pilesorts, and questionnaires to test quantitatively the degree of sharing and diversity of mental models within and across social groups. The consensus analysis method focused on comparing the mental models of two key stakeholder groups in the Crocodile River catchment in South Africa, i.e., conservationists and irrigators, to better understand the level of consensus between these groups. We specifically investigated the level of agreement regarding: (1) major water users of the Crocodile River, (2) causes of the current problems with flows in the river, (3) consequences of the river not flowing, and 4) priorities for future use. We discuss the results and examine the strengths and challenges of consensus analysis for eliciting and measuring mental models. We also evaluated the usefulness of this method in assisting natural resource managers to identify strategies for improving integrated management of water resources.

Key words: consensus analysis; mental models; South Africa; water management


It has long been recognized in the field of natural resource management that creating shared understanding among people is critical if collective decision-making processes and actions aimed at resolving social-ecological problems are to be reached (Röling 2002, Adams et al. 2003). This is particularly true in contested arenas where a plurality of views may hinder the development of solutions. As such, it is vital that natural resource management practitioners, and other key partners, have access to approaches and tools that enable them to better comprehend different social groups’ (and individuals’) beliefs, opinions, and knowledge. It is particularly important that they gain an understanding of the similarities and differences in perspectives within and among different groups regarding the natural resource problem at hand, including its causes and solutions. With this knowledge, natural resource management practitioners are better placed to identify the most appropriate strategies for facilitating negotiation and fostering collective action for improved social and ecological outcomes.

One approach to “getting into people’s heads” is the notion of mental models. A mental model refers to the internal representations of external reality that people carry around with them, and it forms a cognitive basis for reasoning, decision making, and behavior (Jones et al. 2011). According to Quinn (2005), mental models have a social cognition component. Over time people internalize experiences they share with the group(s) to which they belong to create cultural meaning, or a shared mental model. They use these collective mental models to perceive and relate to the world around them. Mental models draw on a broad set of theoretical and applied fields which include cognitive psychology (Johnson-Laird 1983), organizational studies (Walsh and Ungson 1991, Langan-Fox et al. 2000), business management (Axelrod 1976, Senge 1990), human decision making in high reliability systems (Endsley 1995), system dynamics modeling (Doyle and Ford 1998), and knowledge management (Davison and Blackman 2005). A review of the theories underpinning the concept of mental models, at the scale of individuals and groups, and of techniques that have been used to elicit and represent them is presented by Jones et al. (2011).

A range of techniques has been experimented with to elicit and measure the content, structure, and distribution of mental models; each technique has strengths and weaknesses (Jones et al. 2011). In this current paper we present an application of consensus analysis, a method that is associated with the cultural consensus model and developed by A.K. Romney and colleagues (Romney et al. 1986). Commonly used in anthropology to analyze cultural domains, this method has been used to examine a diverse range of topics including illness-related knowledge (Dressler et al. 1998, Curry et al. 2002, Nyamongo 2002, Daniunlaiyte 2004), fisheries management (Miller et al. 2004), organizational change (Caulkins and Hyatt 1999), and traffic safety (Kim et al. 2008). The advantage of the consensus analysis method is that it applies rigorous statistical analyses to structured interview data (typically collected in multiple stages with relatively few participants) to assess the extent of knowledge (Caulkins and Hyatt 1999) or the degree of shared knowledge (Curry et al. 2002) within and between groups.

This current paper describes the use of consensus analysis to capture and measure the mental models of stakeholder groups associated with water use and management in the Crocodile River catchment in South Africa. A first objective of the research, and this paper, is to explore the scientific merits and challenges of consensus analysis as a technique for eliciting, analyzing, and documenting elements of mental models. Drawing on a large body of work on consensus analysis, including Caulkins and Hyatt (1999) and Caulkins (2004), we were interested in understanding the following points:

1. Are the mental models regarding water use and management in the Crocodile River catchment:
  • Shared? (Is there consensus?)
  • Weak? (Is there “weak agreement”, or tendency towards agreement but with considerable variability?)
  • Multicentric? (Are there multiple group mental models, either nonopposing/overlapping or diametrically opposed/contested?)
  • Fragmented? (Is there no agreement? Does each individual have a different mental model?)
2. What is the content and structure (i.e., similarities and differences in how components are organized) of the mental models?

A second objective of the paper is to evaluate the usefulness of the consensus analysis method for decision makers and practitioners working towards sustainable, equitable, and efficient water management under South Africa's National Water Act, Act No. 36 of 1998 (Republic of South Africa 1998). The central objective of the National Water Act, popularly paraphrased as “some, for all, forever”, is the decentralization of water management to the catchment (river basin) scale, primarily through the establishment of water management areas and catchment management agencies. Highly progressive and ambitious at the time it was passed into law, the National Water Act calls for, among other things, ecological integrity of water resources; equity in access to water resources, benefits, and services; and participation of stakeholders in decision-making processes about water resources (see Republic of South Africa 1998, Wester et al. 2003). However, more than ten years later, the implementation of the National Water Act, and of the catchment management agencies more specifically, has proven to be a difficult and slow process. Among the problems encountered is the stronghold of old, primarily bureaucratic, and top-down institutional structures and processes (Rogers et al. 2000). These appear to have further institutionalized the power of well-organized groups (primarily mining and industry sectors, water suppliers in larger towns, and commercial farmers) in the catchment management agency establishment processes (Wester et al. 2003, Goldin 2010). By contrast, the rural poor, the small-scale irrigation sector, and the local domestic water entities have tended to be marginalized (Wester et al. 2003, Goldin 2010). This has been compounded by confusing public engagement processes (du Toit and Pollard 2008). Combined, these have led to disagreements, conflicts, and delays in the development of catchment management agency proposals and to growing disillusionment and participation fatigue among stakeholders (Wester et al. 2003, du Toit and Pollard 2008). As highlighted by Wester et al. (2003:809), “there are many conflicting views—and real conflicts—among stakeholders over water issues, which need to be articulated clearly as part of the CMA [catchment management agency] establishment process” (italics added). MacKay et al. (2003:353) also emphasize the need for “new institutions, new tools, a new mindset and a robust implementation plan” (italics added) if South Africa’s “some, for all, forever” vision for water management is to be achieved. Consensus analysis may be one practical tool that can assist water resources decision makers, managers, users, and other key partners, in the development and establishment of the catchment management agencies in the Crocodile River catchment and other catchments in South Africa.


Consensus analysis is a technique used in association with the cultural consensus model in anthropology (Romney et al. 1986, Romney et al. 1987, Romney 1999). While the model, and the consensus analysis method, do not use the term mental model, they use concepts that are similar such as schema, knowledge domains, and cultural domains. The model defines culture as knowledge shared by a group (Romney 1999). The assumption is that cultural domains—comprised of systems of knowledge which jointly refer to a single conceptual sphere that individuals use to interpret and respond to the world of experience (Weller and Romney 1988, Handwerker 1998)—are learned and shared. The extent of sharing on the particular items comprising a domain can vary and consensus analysis allows investigation of the content and distribution (or sharing) of words and concepts among individuals. Consensus analysis thus enables one to “infer whether there is a culturally central or a more diversified understanding of the domain within the local knowledge system” (Caulkins and Hyatt 1999).

The cultural consensus model and consensus analysis have their origins in cognitive and mathematical anthropology and in psychometrics (Romney and Weller 1984, Romney et al. 1986, Romney et al. 1987, Romney 1999). Consensus analysis consists of an analysis of “informant responses to systematic interview questions” (Romney et al. 1986) which statistically assesses the amount of agreement among a group of people about a domain of culture (or knowledge). More specifically, consensus analysis involves a factor analysis of an intersubject agreement matrix that has been corrected for guessing and for Bayes’ theorem to produce statistical estimates that address three questions (Romney et al. 1986):

  1. To what extent does a group of respondents share a single cultural model (i.e., a single shared mental model)?
  2. If there is consensus, to what extent does each respondent agree with the group’s cultural beliefs (i.e., the group’s mental model)?
  3. What are the “culturally appropriate answers” to each item mentioned by the group?
The model, and by extension consensus analysis, makes three critical assumptions (Romney et al. 1986, Weller 2007):

  1. Respondents share a common culture (i.e., there exists a single set of answers to the question(s) being asked).
  2. Answers are provided by each respondent independently of all other respondents (not by a group nor in consultation with others).
  3. The questions asked should all be on a single topic and at the same level of difficulty.

If consensus is not found, one can assume that one or more of these assumptions was not met.


Study area: the Crocodile River catchment

This study was conducted in the Crocodile River catchment in South Africa (Fig. 1). The Crocodile River drains a catchment of about 10 400 km2, with a total main-stem length of approximately 320 km. It forms the southern boundary of Kruger National Park, and is the largest tributary of the Komati River.

The Crocodile River catchment is located within the Inkomati Catchment Water Management Area, the first water management area in the country to be declared under South Africa's new National Water Act, Act No. 36 of 1998 (Republic of South Africa 1998). At the time of this study in 2007, the governing body—the Inkomati Catchment Management Agency (ICMA)—had been in existence for approximately one year. As the first of 19 catchment management agencies (CMAs) to be established under the National Water Act, the Inkomati Catchment Management Agency was being delegated and assigned new responsibilities and powers, with the Department of Water Affairs and Forestry (the previous management authority) acting as an interim transitional authority. Poor compliance of the National Water Act, overallocation of water, and weak long-term planning for sustainable water management were some of the challenges facing the Inkomati Catchment Management Agency. (For more information regarding the study area and the National Water Act, see Republic of South Africa 1998, Rogers et al. 2000, MacKay et al. 2003, Wester et al. 2003, Goldin 2010; and D. du Toit, H. Biggs, and S. Pollard, unpublished manuscript).

Data collection and analysis: consensus analysis

Consensus analysis can be conducted through one or more phases of data collection and analysis. Typically, when there is little information available about a given domain, or when a domain is potentially not very coherent (i.e., heterogeneous or complex), two phases of data collection and analysis are undertaken. The objective of the first phase is to elicit more knowledge about the domain under inquiry and this generally entails open-ended and semistructured questions such as freelisting. The second phase uses the responses given in the first phase to create a set of structured questions that are analyzed using consensus analysis. These include dichotomous, ordering, and similarity tasks such as pilesorts, and questionnaires (multiple choice, yes/no and true/false, paired-comparison, and triadic) (see Weller and Romney 1988, Borgatti 1996b).

Unlike most conventional survey methods, consensus analysis can produce a high level of statistical confidence with small sample sizes (4 to 30 individuals) (Romney et al. 1986, Weller and Romney 1988:77). In our study, two phases of data collection and analysis were undertaken with 33 individuals and 43 individuals, respectively (Fig. 2). We first carried out semistructured interviews. The data from these interviews were used to construct yes/no questionnaires and pile-sorting exercises which generated the data that were then analyzed using consensus analysis (as well as multidimensional scaling and hierarchical cluster analyses). The diagnostic criteria we used as a guide to interpret our data are presented in Table 1. The data collection and analysis for phases 1 and 2 are described more fully in the appendix.


Theme 1: water users

From the first round of interviews with 33 people, a wide and diverse range of water users were identified. Interviewees listed 25 different water users in the Crocodile River catchment which ranged from diverse social (human) groups to plants and animals (Table 2).

The most frequently cited water user was commercial farmers, who were mentioned by 97% (i.e., 32) of the interviewees. Commercial farmers were also perceived as the most salient water users—i.e., interviewees listed them near the top of the list, perceiving them as the “most important” water users or water users with which they were the most familiar.

The yes/no questionnaire, comprised of a list of 18 water users (all water users that had been mentioned more than once in the first phase of interviews), was filled out by 43 people (16 irrigators and 27 conservationists). The data were analyzed in Anthropac (Borgatti 1996a) with the consensus analysis module. The results of the analyses (Table 3) indicate that there was no consensus among the whole group (irrigators and conservationists combined) about the major water users in the Crocodile River catchment. There also was no agreement among conservationists or irrigators as separate groups.

As indicated in Table 3, for all three groups, the ratio between the first and second eigenvalues was below the 3 to 1 ratio, thus indicating that there was not a single-factor solution or a shared mental model regarding the major water users of the Crocodile River catchment. For example, the ratio for the group as a whole (irrigators and conservationists) was 2.16 to 1. At first glance, this ratio suggests that the group as a whole mildly agreed with each other or shared a “weak” mental model (see Table 1 for guiding diagnostic criteria). However, there was high variability in individual competence scores (SD±0.53) and a very low average (group) competence score of 0.20, the latter indicating that conservationists and irrigators as a group shared only 20% of the beliefs regarding who are the major water users. In addition there were 17 individuals with negative scores. People with negative scores have a very different or unusual understanding of the issue under inquiry, indicating the potential presence of more than one group with different beliefs or mental models (Weller 2007). A closer inspection revealed that seven of the individuals with negative scores were irrigators and the remaining ten were conservationists. As we did not collect any sociodemographic information about our interviewees, it was not possible to explore this difference further. However, the absence of two or more factors of similar strength suggests that there were not subgroups (e.g., men and women, irrigators and conservationists, etc.) holding distinct mental models. Moreover, consensus analysis performed separately on the subgroups of irrigators and conservationists also indicated a lack of consensus (more specifically, “little to no agreement”, see Table 1) with an eigenvalue ratio between the first and second factor of less than 2.0 to 1 for each group.

The lack of agreement among the group as a whole (and within each group) was checked, visually, with multidimensional scaling (MDS) (Fig. 3 for whole group only) which showed irrigators and conservationists scattered across the map. This was confirmed with hierarchical clustering (not shown) which showed no distinctive clustering of people.

We also were interested in assessing if irrigators and conservationists, as a whole group and as separate groups, conceptualized water users in similar categories. A MDS map of the similarities among water users according to the whole group is shown in Fig. 4. The stress score of 0.082 was below the 0.172 stress value cutoff suggested by Sturrock and Rocha (2000) for an 18-item representation in three dimensions. Looking at the MDS map, in the upper center and right corner, rural (rural populations) and urban groups (urban areas, municipal and city authorities, domestic users, black townships, and housing and residential development sector) appeared close together, meaning that conservationists and irrigators perceived them as being more or less similar. Another grouping appears in the upper left corner, with the tourism industry, recreational users (trout fishers), aquaculture, environmental stakeholders (the environment and Kruger Park), and international groups (Mozambique) clustered together. In order to further explore why these groups were perceived to be similar, additional interviews with conservationists and irrigators would be needed.

Theme 2: causes of problems with current flows

A total of 33 factors were mentioned as responsible for the causes of the problems with the current flows in the Crocodile River (Table 4). Of these, the most frequently mentioned was illegal use of water, which was mentioned by 31% (10 people) of the 33 people interviewed. This was closely followed by low rainfall (25%), forest plantations (22%), commercial agriculture (22%), and Kwena dam (19%). If the frequency of times Kwena dam was mentioned was combined with the frequency of mentions of dams in general (16%), then dams were mentioned the most frequently (35%). However, although dams were most frequently mentioned, they were not perceived as the most important or salient cause of problems with current flows. Illegal use of water was identified as the most salient cause of the problems.

A consensus analysis of the yes/no responses to the question “What are the causes of the problems with current flows in the Crocodile River?” showed similar findings to those found for “major water users”. Irrigators and conservationists as a whole group did not hold a shared understanding, or mental model, about the underlying factors causing the problems with the current flows in the Crocodile River (Table 5; ratio between first and second eigenvalues 2.14 to 1, average competence score 0.44 ± 0.43). Similar results were found for conservationists and irrigators as separate groups. Irrigators agreed with each other only 25% of the time (average competence score 0.25 ± 0.47). Among conservationists, the average level of shared beliefs regarding underlying causes was also low and the answers were highly variable (0.46 ± 0.45). For all three groups, the first to second eigenvalue ratios were greater than 2.0 to 1 (but below the 3.0 to 1 threshold for consensus), suggesting that there may be some tendency towards agreement (“weak agreement”). However, again, the presence of several negative scorers (eight for the group as a whole, three for irrigators, and five for conservationists) indicate that there are considerable differences in beliefs regarding the factors that have led to the Crocodile River’s current flow problems. This is also evident in the low average competence scores and in the high standard deviation in individual competence scores.

A MDS map (Fig. 5) confirmed that irrigators and conservationists as a group did not have a shared mental model regarding the causes of the river flow problems. The group of irrigators (represented by blue squares) and conservationists (red circles) formed clusters indicating that there may be consensus within each of these groups. We ran MDS maps and hierarchical cluster analyses (not shown) for the two groups separately which showed individuals scattered across rather than in close proximity to each other, indicating a lack of agreement.

We also ran MDS maps and hierarchical cluster analyses of the causes of the flow problems in the Crocodile River (not shown) to see if conservationists and irrigators organized problems into separate clusters. Neither conservationists nor irrigators—as separate groups or as a whole group—grouped the list of causes of the river flow problems into any distinct, coherent categories.

Theme 3: consequences of the river not flowing

A total of 34 different negative impacts were mentioned in response to the question “What are the consequences of the Crocodile River not flowing?”. The consequences mentioned covered a broad range of ecological, social, and economic impacts. The impact on biodiversity was mentioned most frequently, i.e., by 30% of the people interviewed (see Table 6). It was also perceived as the most salient or important of consequences. This was followed by impact on the whole economy (24%), on animals (18%), and on Mozambicans (18%).

Because it is not possible to run consensus analysis on unconstrained pilesorts, a MDS map of the 43 people who participated in the pile-sorting exercise suggests that there was no agreement among irrigators and conservationists with respect to the consequences of the river not flowing (see Fig. 6; stress score = 0.132). This is the classic “fried egg” pattern of disagreement noted by Romney and Weller (1984) which shows that individuals in the center of the MDS map had the highest competence scores, whereas those toward the periphery had the lowest. Additional background information about each of the interviewees would have been useful to reveal if those conservationists and irrigators clustered more closely together in the center of the MDS map shared some common characteristic (e.g., higher education levels, lived longer period of time in the region, etc.).

We also ran a MDS map and cluster analysis on conservationists and irrigators separately. The MDS map of the conservationists interviewed suggests that they generally agreed with each other with the exception of two individuals (indicated by numbers 24 and 27 in Fig. 7) who had different perspectives from the rest of the group regarding what would happen if the river stopped flowing. The MDS map of irrigators (not shown) seemed to show a weaker level of agreement among irrigators.

We also wanted to know if people had pile sorted the consequences into similar piles. Fig. 8 shows the MDS map of the pilesorts of consequences in two dimensions (stress score = 0.100). The consequences of the Crocodile not flowing were clearly separable into social, economic, and ecological categories (red and black, blue, and green, respectively).

The consequences circled are those that were found to be grouped together in the cluster analysis (Fig. 9). Looking at the x-axis of the MDS map from left to right, it appears that the consequences clustered on the left side are ecological impacts and the consequences clustered on the right side are socioeconomic impacts. The y-axis suggests that the cluster of socioeconomic consequences is further broken down into social impacts (top right-hand corner) to more economic impacts (bottom left-hand corner). Within the cluster of social impacts, there also appears to be a distinction between social impacts that will occur at the level of the catchment and those that will occur at the international scale.

Two items—“less water to use” and “everyone impacted”—did not group well within the social category. Compared to the other consequences, which were very specific, both of these are very broad and potentially vague terms, which may have been open to different interpretations during the pile-sorting exercise.

Similar findings were revealed for irrigators and conservationists separately (not shown, MDS stress scores in two dimensions were 0.090 for irrigators and 0.136 for conservationists). In other words, both groups generally pile sorted the consequences into ecological categories and subcategories of socioeconomic impacts. The only difference between the two was that irrigators tended not to see “negative impacts on tourism” as an economic impact; rather, they saw this as a separate item.

Theme 4: priorities for future use

Over 70 different priorities were listed. The top 40 are presented in Table 7. At the top end, over 40% listed “basic human needs” as a priority for future water use, followed by equity balance which was mentioned by 34%. “Water to meet ecological and environmental needs” was mentioned by 28%.

For the three groups, the low ratios (below the 3 to 1 threshold) between the first and second eigenvalue factors, combined with extremely low average competence scores, indicate that there is not a single, shared mental model regarding the priorities for future water use (Table 8). The eigenvalue ratio of 2.21 to 1 for the conservationists interviewed suggests that they may potentially weakly agree with each other. However, the average competence score of 0.065 (i.e., shared only 6.5% of priorities listed) clearly indicates that there is very little agreement among them. For all three groups, the lack of consensus was confirmed with MDS maps and hierarchical cluster analysis (not shown).

We also mapped the priorities, using MDS and cluster analysis, to see if the individuals we had interviewed had pile sorted them similarly, but these analyses did not enable us to identify any clear patterns among the group as a whole or for irrigators as a separate group. For conservationists, however, the MDS map (not shown; 3 dimensions; stress = 0.139) and the cluster analysis results suggest that conservationists distinguished among priorities. They tended to pile sort the priorities focused on water management for conservation or sustainability together, and to pile sort the priorities for water management for human activities in another group.



We were interested in eliciting and comparing the mental models of conservationists and irrigators with respect to: (1) who were the major water users of the Crocodile River catchment, (2) what were the causes of the current problems with flows in the Crocodile River, (3) what would be the consequences of the river not flowing, and (4) what should be the priorities for future use. Were the mental models regarding water use and management in the Crocodile River catchment shared, weak, multicentric, or fragmented? Among the irrigators and conservationists we interviewed there was no consensus, as a whole group or as separate groups, regarding major water users, the causes of the river flow problems, and future priorities for the Crocodile River catchment. With eigenvalue ratios of less than the 3 to 1 recommended cutoff for consensus, the groups did not have a shared mental model regarding these issues. While there may not have been consensus, the data suggested that there may be “weak agreement”, i.e., a tendency towards agreement but with great variability in beliefs or knowledge (as evidenced by eigenvalue ratios of 2.0 to 2.9 to 1 (Caulkins 2004, on (1) major water users for the whole group, (2) causes of the flow problems for all three groups, and (3) priorities for conservationists. However, in almost all of these cases, the group and individual competence scores were too low and variable, respectively, to conclude that they shared a “weak” mental model. Rather, the mental models in these instances would be considered “fragmented” (Caulkins 2004) or “turbulent” (Caulkins and Hyatt 1999). The exception was the whole group and conservationists with regard to the causes of the problems with the river flows. In both cases, approximately 45% of the group shared similar beliefs regarding the causes. Over time, this weak agreement may increase into consensus or a shared mental model or, alternatively, may weaken further (Caulkins 2004).

Given that we used unconstrained pilesorts to assess the level of agreement regarding the consequences of the Crocodile River not flowing, we were unable to analyze the data with consensus analysis. However, MDS and hierarchical cluster analysis indicated that there seemed to be little agreement within the whole group, some agreement among irrigators, and stronger agreement among conservationists.

What was the content and structure of these mental models? I.e., were there similarities or differences in how items were grouped together by interviewees? With respect to major water users, causes of river flow problems, and future priorities, some items were perceived by irrigators and conservationists as being similar and grouped together. However, some of the groupings of items were difficult to interpret, and additional discussions with interviewees are needed to further explore these clusters. With regards to the consequences of the Crocodile River not flowing, irrigators and conservationists clearly separated them into three broad categories of social, economic, and ecological impacts.


As an analytical tool for eliciting and measuring mental models, consensus analysis has both strengths and weaknesses. Its strength lies in its quantitative analysis of people’s knowledge and beliefs (qualitative data). It allows one to assess, in quantitative terms, who agrees with whom, about what, and to what degree. Moreover, because it is not an invasive method, i.e., it minimizes the involvement of the researcher via use of prompt questions and the like, it also permits the elicitation of mental models with minimal interference, and therefore minimal potential influence, by the researcher.

On the other hand, consensus analysis poses significant challenges. Careful preparation is needed in designing and carrying out the two phases of fieldwork. The wording of the questions must be well thought out in order to capture people’s beliefs and knowledge regarding conventional truths or judgments and not their personal preferences or histories (Gatewood 1999, Horowitz 2009). If the domain is potentially complex and comprises a high diversity of issues, the first phase of data collection may necessitate time-consuming methods such as open-ended or semistructured interviews. While the data collected from such interviews are rich, it entails content analysis which may bring in bias unless multiple coders are used to cross-check and validate the codes produced (see Guest et al. 2006).

In both phases of data collection, sampling must also be carefully considered. In phase 1, it is recommended to collect data until one reaches either “saturation of issues”, i.e., no new issues appear, or an identification of a core set of issues that people seem to agree on. In phase 2, sample size is dependent on the average (group) competence score which is often not known beforehand (see appendix for further details). Moreover, it is important to sample either across differences in life experiences and/or sociodemographic variables such as gender, age, and ethnicity, or to focus on a very specific group (Handwerker 2005). In our research, we cannot exclude sampling problems as a potential explanation for the lack of consensus. Our samples were quite modest by survey research standards but, as previously mentioned, consensus analysis can produce very reliable results from small samples (4 to 30 individuals) (Weller and Romney 1988). However, when sample sizes are small, the average level of competence must be 0.4 or above to obtain reliable results (Weller 2007). While our sample sizes for the whole group (43) and conservationists (27) were on the higher end, our sample of irrigators was small (17). Moreover, the average competence scores were consistently very low (below 0.4) for irrigators, as well as for conservationists and for the whole group in certain domains (major water users and priorities). This suggests that either we violated one or more of the three assumptions of the cultural consensus model (discussed further below) or that our sample sizes were too small.

Other challenges for applying consensus analysis revolve around the analysis and interpretation of the results. As pointed out by Garro (2000) and Handwerker (2002), the three assumptions of the cultural consensus model are highly restrictive. One must ensure that: (1) there is a common culture, or domain of knowledge, that people in the group share; (2) the responses of individuals in a group are given independently of others; and (3) all the questions asked are on a single topic and are at the same level of difficulty. These are often not met in field research (Garro 2000, Handwerker 2002). Our findings (lack of consensus across all four domains—water users, causes of problems, consequences, and priorities) suggest that one or more of these assumptions were not met or that “something else [went] wrong” (Romney et al. 1986:166). Given that the data were collected from independent interviews, it is likely that the lack of consensus was due to the absence of a common culture regarding water users, causes of problems, consequences, and priorities; and/or because questions that varied in difficulty or were conceptually too broad; and/or due to some other factor. Upon further reflection, our assumption of a common culture for each domain may not have been appropriate. For example, when we asked “Who were the major water users in the Crocodile River catchment?” we were assuming that people had mental models of who used the water over the entire catchment. It is possible that the individuals we talked with had mental models of users at different geographical scales when answering this question. In situations where the data show no consensus, such as in our research, an additional challenge is drawing the line between weak agreement and multicentric and fragmented domains (Caulkins and Hyatt 1999). This is complicated by the use of different diagnostic criteria by scholars. Given these limitations, Horowitz (2009) argues that consensus analysis is of most value as an analytical tool in cases where cultural or knowledge domains are not idiosyncratic and have clear boundaries.

These methodological challenges have significant implications for the use of consensus analysis by natural resource management practitioners. For time- and capacity-constrained organizations and individuals, using consensus analysis would initially require working with someone with the technical expertise to set up the field research and analyze and interpret the data. This may not be financially feasible in some cases. And while practitioners can be trained in consensus analysis, the technical complexity of the method can pose a significant barrier to its use as an applied tool for better understanding different groups’ mental models and for developing strategies for action.

Despite these challenges, consensus analysis has the potential to be a tool of great value to natural resource management practitioners in general, and to the catchment management agencies and supporting organizations in the Inkomati and in other water management areas in Africa. As noted by Wester et al. (2003), Mackay et al. (2003), and du Toit and Pollard (2008), among others, one of the biggest obstacles to achieving South Africa’s water management vision is improved engagement processes with diverse stakeholders. Participatory tools, such as consensus analysis, that reveal the plurality of views and provide a vehicle for negotiation through the “interfacing and sharing of knowledge” (Roux et al. 2006) are consequently needed. Mistrust, misconceptions, and misunderstandings—fueled by a long history of social, economic, and cultural differences and injustices—have created significant barriers between South Africa’s advantaged and disadvantaged populations, and between scientists and managers, among others (Roux et al. 2006). Consensus analysis can help build bridges between these groups by giving “equal voice” to all (through the qualitative phase 1 of the research) and identifying (through the quantitative phase 2) differences and convergences in mental models. In doing so, consensus analysis can contribute towards “resilient pathways” for achieving South Africa’s “some, for all, forever” vision (Bohensky 2008).

Our experience using consensus analysis in the Crocodile River catchment showed the potential of the consensus analysis method. In phase 1 of the research, the freelisting exercise provided a fairly easy and quick way of identifying key issues and their relative importance, thus providing powerful information if used constructively. For example, illegal water use was mentioned as the leading cause of the problems with the flows of the Crocodile River. Exploring this contentious issue further with different groups (e.g., large irrigators and rural poor; national- and catchment-level decision makers, etc.) would be a critical first step if agreement is to be reached on how to best resolve the problem of illegal use and, ultimately, to improve river flows.

Consensus analysis of the pilesorts (phase 2) revealed that the conservationists and irrigators grouped the consequences of the river not flowing similarly. The unambiguous recognition of undesirable social, economic, and ecological changes indicates that they did not like where they were headed. This provides a potentially useful starting point for negotiations around future water-use priorities. Unfortunately, there was no agreement among the conservationists and irrigators we interviewed regarding future uses of the water (yes/no questionnaire). Under the National Water Act, the national government is legally obligated to “reserve” or ensure the provision of sufficient water for meeting basic human needs and protecting aquatic ecosystems. Allocation of remaining water has yet to be defined but preference will be given to uses that have high social, economic, or environmental value and which promote equity (National Water Act, Act No. 36 of 1998, section 27 (Republic of South Africa 1998)). Thus, building consensus among the diversity of water users is a critical task. It would be interesting to repeat the consensus analysis method after the catchment management agency in the Inkomati Basin has been in operation, to identify if the actions they take will help foster agreement around future water-use priorities and actions.


Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.


Thanks are given to Dr. Steve Mitchell and Ms. Una Wium from the Water Research Commission, and to Dirk Roux and Nikki Funke from the Council of Scientific and Industrial Research (CSIR) who took a particular interest in this initiative and especially in the public meetings associated with it; and to Michele Hofmeyr, Sarah Webb, and Thomas Gyedu-Ababio of SANParks who assisted with interview scheduling and logistical arrangements. Thanks are also given to the Inkomati Catchment Management Agency (especially Sizile Ndlovu, Joseph Mabunda, and Gugu Sandleni) who assisted with logistics and venues for most of the interviews. Thanks also to Ramatshediso Mphuti for information compiled on the Crocodile River catchment. Special thanks are extended to the almost 100 people with an interest in water and related issues of the region and who participated in our interviews, discussion groups, and dialogue sessions. Thanks are also given to Susan Weller from the University of Texas Medical Branch, Russell Bernard from the University of Florida, and Alexander Herr from the Commonwealth Scientific and Industrial Research Organization (CSIRO) for their invaluable assistance with the interpretation of the consensus analysis results. This research would not have been possible without funding and/or support from the following organizations: CSIRO’s Complex Systems Science; French National Institute for Agricultural Research (INRA); Resilience Alliance; South African National Parks; and the Water Research Commission (South Africa): Workshop39/2007.


Adams, W., M. D. Brockington, J. Dyson, and B. Vira. 2003. Managing tragedies: understanding conflict over common pool resources. Science 302:1915-1916.

Axelrod, R. 1976. Structure of decision. Princeton University Press, Princeton, New Jersey, USA.

Bohensky, E. L. 2008. Discovering resilient pathways for South African water management: two frameworks for a vision. Ecology and Society 13 (1): 19 [online] URL:

Borgatti. S. P. 1996a. ANTHROPAC 4.0. Analytic technologies, Natick, Massachusetts, USA.

Borgatti. S. P. 1996b. ANTHROPAC 4.0 methods guide. Analytic Technologies, Natick, Massachusetts, USA.

Caulkins, D. D. 2004. Identifying culture as a threshold of shared knowledge: a consensus analysis method. International Journal of Cross Cultural Management 4:317-333.

Caulkins, D., and S. B. Hyatt. 1999. Using consensus analysis to measure cultural diversity in organizations and social movements. Field Methods 11:5-26.

Curry, M. D., H. F. Mathews, H. J. Daniel, J. C. Johnson, and C. J. Mansfield. 2002. Beliefs about and responses to childhood ear infections: a study of parents in eastern North Carolina. Social Science and Medicine 54:1153-1165.

Daniulaityte, R. 2004. Making sense of diabetes: cultural models, gender and individual adjustment to Type 2 diabetes in a Mexican community. Social Science and Medicine 59:1899-1912.

Davison, G., and D. Blackman 2005. The role of mental models in the development of knowledge management systems. International Journal of Organisational Behaviour 10:757-769.

Doyle, J. K., and D. N. Ford. 1998. Mental models concepts for system dynamics research. System Dynamics Review 14:3-29.

Dressler, W. W., M. C. Baliero, and J. E. dos Santos. 1998. Culture, socioeconomic status, and physical and mental health in Brazil. Medical Anthropology Quarterly 12:424-446.

du Toit, D., and S. Pollard. 2008. Updating public participation in IWRM: a proposal for a focused and structured engagement with Catchment Management Strategies. Water SA 34:707-713.

Endsley, M. R. 1995. Towards a theory of situation awareness in dynamic systems. Human Factors 37:32-64.

Garro, L. C. 2000. Remembering what one knows and the construction of the past: a comparison of cultural consensus theory and cultural schema theory. Ethos 28:275-319.

Gatewood, J. B. 1999. Culture . . . one step at a time, Part 1. The Behavioral Measurement Letter 6:5-10.

Goldin, J. A. 2010. Water policy in South Africa: trust and knowledge as obstacles to reform. Review of Radical Political Economics 42:195-212.

Guest, G., A. Bunce, and L. Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field Methods 18:59-82.

Handwerker, W. P. 1998. Consensus analysis: sampling frames for valid, generalizable research findings. Pages 165–178 in V. C. de Munck and E. J. Sobo, editors. Using methods in the field: a practical introduction and casebook. AltaMira Press, Walnut Creek, California, USA.

Handwerker, W. P. 2002. The construct validity of cultures: cultural diversity, culture theory, and a method for ethnography. American Anthropologist 104:106-122.

Handwerker, W. P. 2005. Sample design. Pages 429–436 in K. Kempf-Leonard, editor. Encyclopedia of social measurement, Volume 3. Elsevier, San Diego, California, USA.

Horowitz, D. M. 2009. A review of consensus analysis methods in consumer culture, organizational culture and national culture research. Consumption Markets & Culture 12:47-64.

Johnson-Laird, P. N. 1983. Mental models. Harvard University Press, Cambridge, Massachusetts, USA.

Jones, N. A., H. Ross, T. Lynam, P. Perez, and A. Leitch. 2011. Mental models: an interdisciplinary synthesis of theory and methods. Ecology and Society 16(1):46. [online] URL:

Kim, T. G., E. T. Donnell, and D. Lee. 2008. Use of cultural consensus analysis to evaluate expert feedback of median safety. Accident Analysis and Prevention 40:1458-1467.

Langan-Fox, J., S. Code, and K. Langfield-Smith. 2000. Team mental models: techniques, methods and analytic approaches. Human Factors 42:242-271.

MacKay, H. M., K. H. Rogers, and D. J. Roux. 2003. Implementing the South African water policy: holding the vision while exploring an unchartered mountain. Water SA 29:353-358.

Miller, M. L., J. Kaneko, P. Bartram, J. Marks, and D. D. Brewer. 2004. Cultural consensus analysis and environmental anthropology: yellowfin tuna fishery management in Hawaii. Cross-Cultural Research 38:299-314.

Nyamongo, I. K. 2002. Assessing intracultural variability statistically using data on malaria perceptions in Gusii, Kenya. Field Methods 14:148-160.

Quinlan, M. 2005. Considerations for collecting freelists in the field: examples from ethnobotany. Field Methods 17:219-234.

Quinn, N. 2005. How to reconstruct schemas people share, from what they say. Pages 33–81 in N. Quinn, editor. Finding culture in talk: a collection of methods. Palgrave Miller, New York, New York, USA.

Republic of South Africa. 1998. National Water Act, Act No. 36 of 1998. Republic of South Africa Government Gazette, 26 August 1998, No. 19182. Pretoria, Gauteng Province, South Africa. URL:

Rogers, K., D. Roux, and H. Biggs. 2000. Challenges for catchment management agencies: lessons from bureaucracies, business and resource management. Water SA 26:505-511.

Röling, N. 2002. Beyond the aggregation of individual preferences. Pages 25–47 in C. Leeuwis and R. Pyburn, editors. Wheelbarrows full of frogs: social learning in resource management. Koninklijke Van Gorcum, Assen, The Netherlands.

Romney, A. K. 1999. Cultural consensus as a statistical model. Current Anthropology 40:S103-115.

Romney, A. K., W. H. Batchelder, and S. C. Weller. 1987. Recent applications of cultural consensus theory. American Behavioral Scientist 31:163-177.

Romney, A. K., and S. C. Weller. 1984. Predicting informant accuracy from patterns of recall among informants. Social Networks 6:59-77.

Romney, A. K., S. C. Weller, and W. H. Batchelder. 1986. Culture as consensus: a theory of culture and informant accuracy. American Anthropologists 88:313-338.

Roux, D. J., K. H. Rogers, H. C. Biggs, P. J. Ashton, and A. Sergeant. 2006. Bridging the science–management divide: moving from unidirectional knowledge transfer to knowledge interfacing and sharing. Ecology and Society 11 (1):4 [online] URL:

Senge, P. 1990. The fifth discipline: the art and practice of the learning organization. Doubleday, New York, New York, USA.

Smith, J. J. 1993. Using ANTHROPAC 3.5 and a spreadsheet to compute a freelist salience index. Cultural Anthropology Methodology Newsletter 5:1-3.

Sturrock, K., and J. Rocha. 2000. A multidimensional scaling stress evaluation table. Field Methods 12:49-60.

Walsh, J. P., and G. R. Ungson. 1991. Organizational memory. The Academy of Management Review 16:57-91.

Weller, S. C. 2007. Cultural consensus theory: applications and frequently asked questions. Field Methods 19:339-368.

Weller, S. C., and A. K. Romney. 1988. Systematic data collection. Sage Publications, Newbury Park, California, USA.

Wester, P., D. J. Merrey, and M. de Lange. 2003. Boundaries of consent: stakeholder representation in river basin management in Mexico and South Africa. World Development 31:797-812.

Address of Correspondent:
Samantha S. Stone-Jovicich
Australian Tropical Science & Innovation Precinct (ATSIP)
Building 145 James Cook Drive
James Cook University, Douglas Campus


Home | Archives | About | Login | Submissions | Subscribe | Contact | Search