Phase 1

In the first phase of our research (see Fig. 2 in article), we conducted separate interviews with 33 individuals. These individuals were identified through purposive and snowball sampling methods. They were affiliated with a diversity of organizations and social groups living or working in the Crocodile River Catchment, including government water management officials, members of irrigation associations, and conservationists.

The interviews were part of a larger study and the interview schedule (comprised of twelve questions; not shown) included four semi-structured, open-ended questions on water use and management in the Crocodile Catchment that were of interest to us for consensus analysis. These were:

The first question was a free listing technique which asked interviewees to provide an answer in the form of a list words or concepts (Weller and Romney 1988). The remaining three questions involved eliciting responses in a narrative form. The interviews were audio-recorded with the interviewee’s permission.

Analysis of the responses to the four questions was a two-step process. For the first question – water users – we listed the water users in the order that they were mentioned by each interviewee. For the remaining three questions, we did a content analysis of the responses. Themes emerging from the interviewee responses were coded and, for each question, we kept a separate running list of codes, creating new codes as new themes emerged. To eliminate conceptual redundancies, the same codes were used for similar themes or issues. This coding process allowed the answers given in narrative form to be listed in the form of one word or a short phrase, in the order that they were mentioned, hence resulting in free lists. Free listing makes two assumptions: 1) people tend to list things that they are most familiar with or believe are most important before they list things that are less familiar or less important, and 2) people who know a lot about a subject will list more things than people who know less (therefore, they will have longer free lists) (Quinlan 2005).

The resulting free lists for the four questions were subsequently analyzed with Anthropac software (Borgatti 1996a). Analysis of free lists in Anthropac generates four pieces of information: 1) frequency (the number of times each item was listed), 2) response percentage (the percentage of interviewees who mentioned each item), 3) rank (the aggregate average rank for each item), and 4) salience [a measure based on the frequency and rank of each item which indicates how much knowledge informants share and how important that knowledge is to them (Smith 1993)].

A total of 43 people who had not participated in the first phase were selected, also using purposive and snowball sampling (see Fig 2 in article). For data analyzed with consensus analysis [i.e. consensus analysis module in Anthropac software(Borgatti 1996a)], determining the sample size needed before the research begins is often not possible, as it depends on the average (group) level of competence (i.e. level of agreement among respondents) which is not known in advance. The general rule of thumb is that the average competency should be at least 0.5 (i.e. the group of respondents interviewed share 50% of the beliefs or knowledge) to obtain reliable results (Romney et al 1986) but it can be as low as 0.4 (Weller 2007). With an average competency of 0.5, a sample size of 23 is needed to correctly classify 95% of the answers at the .99 confidence level (see Weller and Romney 1988: 77, Weller 2007: 354). Given that the average competency is not known beforehand, the researcher has two options: (1) prior to collecting data, adopt stringent criteria (i.e. assume a low level of agreement, i.e. 50% sharing of beliefs or average competency score of .5, and aim for a high accuracy of answers, i.e. .95 validity) and, thus, a minimum sample size of 23, and/or (2) analyze the data at multiple stages of data collection to obtain the average competence score and stop interviewing when a minimum of .5 average competence score is obtained (for further details, see Romney et al 1986, Weller 2007).

In phase 2 of our research, we decided to narrow our focus on irrigators and conservationists as we were interested in knowing if these very different groups shared mental models about water use and management in the Crocodile River Catchment. Our total sample size was 43 people, comprised of 27 conservationists and 16 irrigators. While we aimed to increase the number of participants (particularly irrigators) in our study, we were unable to due to logistical constraints. The 43 people we interviewed were members of Irrigation Boards/Water User Associations, the Department of Agriculture and Land Administration (including the Directorate of Environmental Affairs), Ecolink (a local environmental NGO) and SANParks. Separate interviews were carried out with each person. They were asked to complete a yes/no questionnaire followed by two pile-sorting exercises.

The questionnaire consisted of two sections, one focused on ‘major water users’ and the other on ‘causes of problems with current flows’. A total of 25 ‘major water users’ were listed in the questionnaire (those mentioned by more than one person in Phase 1). The list of ‘causes of problems’ included all the causes that had been brought up more than once in the first phase (25 causes) and an additional 6 that had been mentioned only once but were deemed to be potentially important based on the research team’s knowledge of the topic. This resulted in a total of 31 ‘causes of problems’ being listed in the questionnaire. Interviewees were first asked to respond to the question ‘of the following list of water users in the Crocodile River Catchment, which are major water users?’ They had to check ‘yes’ if they thought a particular water user listed was a major user of water and ‘no’ if they thought otherwise. They repeated the exercise with the question ‘Do these things contribute to problems with current flows in the Crocodile River?’.

After completing the questionnaire, each interviewee was asked to participate in two pile-sorting exercises, one focused on ‘consequences of the Crocodile River not flowing’ and the other on ‘priorities for future water use’. For the pile-sorting exercises, the top consequences (those mentioned at least twice in the free lists generated in Phase 1) were written on separate index cards (total of 23 cards). A separate set of cards was created for the priorities for future water use that were mentioned at least twice (three of the 40 priorities listed in the first phase were merged to create a total of 37 cards). Interviewees were first asked to pile sort the ‘consequences of the Crocodile River not flowing’. They were given the stack of cards, each containing a single word or phrase (with an identification number written on the back), and asked to organize them into groups or piles on the basis of similarity. They were instructed that there was no right or wrong way to sort the cards and that they could make as many piles as they wanted, with a minimum of two piles. Interviewers noted this information as well as recording separately the identification numbers on the back of each card for each pile that had been created. We repeated the exercise for the cards on ‘priorities’, but this time constrained them to three piles: priorities that are highly important, of medium importance, and of low importance.

Using Anthropac (Borgatti 1996a), the results of the questionnaire and pile-sorting exercises were converted to aggregate proximity matrices of interviewees and analyzed with consensus analysis (a module in Anthropac). The consensus analysis module applies minimum residuals factor analysis to interviewee responses to determine whether there is sufficient agreement among the persons interviewed to suggest that they share elements of their mental model on a particular issue. The estimates produced by the CA module in Anthropac, and the questions they inform, are summarized in Table 1 (in article). It is important to note that the 3 to 1 ratio between the first and second eigenvalues has been widely accepted in the consensus analysis literature as the appropriate threshold for determining consensus (Romney et al. 1987, Borgatti 1996b, Romney 1999). More recently, several scholars have extended the logic of consensus analysis to the interpretation of data that is below this threshold (for example, Caulkins and Hyatt 1999, Handwerker 2002, Caulkins 2004). The diagnostic criteria we used as a guide to interpret our data are presented in Table 1 (in article).

In addition to the consensus analysis module, we used non-metric multi-dimensional scaling (MDS) and Johnson’s hierarchical clustering. Although MDS is generally not necessary in consensus analysis as it may distort the data (Weller 2007), along with cluster analysis, it is a useful complementary visual tool that is commonly used with consensus analysis. Both MDS and cluster analysis facilitate visualizing the degree to which people share words or concepts. In the map produced by MDS, people who are in closer agreement appear closer together; people who have different understandings, or mental models, on the issue appear farther apart. MDS produces a Kruskal stress score, which reflects the degree to which the MDS model represents the data. The lower the score (closer to 0), the better the representation. It is recommended that 2-dimensional MDS be used but, if shown to reduce the stress, 3-dimensional MDS is acceptable. All of the MDS graphics presented in this paper had stress scores that were below than the cutoffs suggested by Sturrock and Rocha (2000). Cluster analysis is often used to interpret the groupings of people in the MDS map. It produces a schematic diagram of clusters of people in accordance to their similarity.
While our main interest was to assess the level of consensus among people, we were also interested in identifying the similarities and differences among the items pile sorted or grouped (into yes/no categories) in the questionnaire for each of the domains (major waters users, problems, consequences, priorities). To this end, we also used non-metric MDS and Johnson’s hierarchical clustering.