Resilience and Higher Order Thinking

. To appreciate, understand, and tackle chronic global social and environmental problems, greater appreciation of the importance of higher order thinking is required. Such thinking includes personal epistemological beliefs (PEBs), i.e., the beliefs people hold about the nature of knowledge and how something is known. These beliefs have profound implications for the way individuals relate to each other and the world, such as how people understand complex social-ecological systems. Resilience thinking is an approach to environmental stewardship that includes a number of interrelated concepts and has strong foundations in systemic ways of thinking. This paper (1) summarizes a review of educational psychology literature on PEBs, (2) explains why resilience thinking has potential to facilitate development of more sophisticated PEBs, (3) describes an example of a module designed to teach resilience thinking to undergraduate students in ways conducive to influencing PEBs, and (4) discusses a pilot study that evaluates the module's impact. Theoretical and preliminary evidence from the pilot evaluation suggests that resilience thinking which is underpinned by systems thinking has considerable potential to influence the development of more sophisticated PEBs. To be effective, however, careful consideration of how resilience thinking is taught is required. Finding ways to encourage students to take greater responsibility for their own learning and ensuring close alignment between assessment and desired learning outcomes are particularly important.


INTRODUCTION
There is increasing recognition of the influence of cognitive and psychological factors on the way people adapt and respond to the world's chronic social and environmental problems (Grothmann andPatt 2005, Fazey et al. 2007). Adaptive responses to change are clearly influenced and shaped by the context, culture, and society in which the individuals are embedded (Eakin 2005, McLeman and Smit 2006, Cliggett et al. 2007), but are also affected by perceptions of the nature of the change and the associated risks (Grothmann and Patt 2005, McLeman and Smit 2006, Marshall and Marshall 2007, Shisanya and Khayesi 2007. Learning interventions that promote the careful analysis of problems and empower people to play an active part in their resolution can have significant effects on responses to social and environmental change problems (Stave 2002, Hagmann and Chuma 2000, Fraser et al. 2006, Reed et al. 2006, Ison et al. 2007, van Aalst et al. 2008. Such interventions, however, will be more effective if individuals are already disposed to thinking critically and have a greater capacity to learn in complex and dynamic systems (Fazey et al. 2005, Fazey et al. 2007. Dispositions for critical thinking and capacity for problem solving can be both learned and taught (Perkins et al. 2000, Perkins 2007. This raises questions about the processes and effective learning environments that not only contribute to change in understanding but which also facilitate the development of higher order thinking. Understanding the kinds of instructional practices, contexts, and subjects of focus that create more effective thinkers, and understanding how these approaches can be incorporated into development programs is a crucial aspect of developing more pluralistic, equitable, and informed global societies (Sterling 1996, UNESCO 2005. One subject of focus that has potential to help facilitate the development of higher order cognition is resilience thinking (see Appendix 1 for definitions To help make decisions about the likely outcome of a course of action (e.g. using concepts of adaptive cycles to help understand when to target interventions to transform the trajectory of a social-ecological system).

Explanation
To help suggest why something occurs (e.g. understanding regime shifts or crossing thresholds).
Identify key questions For research and/or practice.

Facilitate design
To help form a plan or scheme to reduce vulnerability of a system to externally induced change.

Communicate
To convey knowledge and ideas between researchers and/or practitioners.

Motivate/Engage
To increase broad interest in social and ecological issues.
Guide practice To help develop new policies.

Conceptual change
To help facilitate the way people view social-ecological systems (e.g. through the lens of resilience that sees social-ecological systems as dynamically complex and where, in addition to efficiency, innovation and flexibility are important).

Development of higher order thinking
Subject of focus for teaching that facilitates development of more sophisticated cognition.
of resilience thinking and other interrelated terms used in this paper). This approach to the stewardship of sustainability includes a range of inter-related concepts such as social-ecological resilience, panarchy, adaptive co-management, transformation, and adaptive governance (Walker and Salt 2006). The approach arose from recognition of the need for continuous learning for understanding complex, dynamic, and interacting human and ecological systems Meffe 1996, Folke 2006). Resilience thinking has deep roots in systems thinking which emphasizes the importance of the dynamic feedback effects between system components and the overall behavior that emerges from these processes Holling 2002, Walker et al. 2004). Thus resilience thinking, underpinned by systems thinking, gives credence to an integrative and holistic approach to research and practice that bridges many disciplines. Overall, the resilience framework assists thinking about the dynamic and often debilitating complexity of social and environmental problems (Carpenter and Folke 2006), and is increasingly being applied globally to a wide range of contexts (e.g., Allison and Hobbs 2004, Armitage and Johnson 2006, McDaniels et al. 2008, and has been suggested to be a useful lens for formulating more effective social and environmental policies (Ison et al. 2007, Boyd et al. 2008. Questions remain about the extent to which resilience thinking is useful as a generalized set of concepts that can predict and explain real-world processes (Carpenter et al. 2001, Gallopin 2006. Explanation and prediction are, however, only some of the many potential uses of a conceptual tool (Table 1). This paper focuses on one of these uses: the potential value of the teaching of resilience thinking as a way to influence the higher order cognition of students. It asks whether teaching resilience thinking is likely to influence personal epistemological beliefs (PEBs)-i.e., the beliefs people hold about the nature of knowledge and how they come to know something (Hofer and http://www.ecologyandsociety.org/vol15/iss3/art9/ Pintrich 1997, Buehl and Alexander 2001)-and thus whether resilience thinking is likely to be a useful subject of focus that can contribute to broader shifts in peoples' perceptions of and relationships to complex real-world problems. To do this, the paper (1) summarizes a review of the educational psychology literature on PEBs; (2) explains why the teaching of resilience thinking may have potential to influence the development pf PEBs; (2) provides an example of how teaching resilience thinking can be taught to affect the development of more sophisticated PEBs; and (3) discusses a pilot study that evaluated the teaching module's impact on students. The pilot study did not intend to provide conclusive evidence for the effects of teaching resilience thinking, but instead aimed to draw out additional issues regarding the design of effective teaching programs to affect the way students perceive ill-structured problems.

Personal epistemological beliefs
There are many different ideas about what constitutes higher order thinking and how greater ability to analyze and tackle complex problems can be developed. An important area of educational psychology that is highly relevant to understanding effective thinking is the study of personal epistemological beliefs (PEBs). These are the beliefs people hold about the nature of knowledge and how a person comes to know something. Studies from educational psychology typically aim to determine how these personal beliefs affect perception, learning, and behavior (Hofer and Pintrich 1997, Buehl and Alexander 2001, Hofer 2001. Such beliefs are considered to operate at higher levels than many other forms of thinking and have a major impact on how people tackle illstructured problems (Kitchener 1983, Kuhn 2000).
An example of how PEBs operate at higher levels to impact other levels of cognition are the thinking processes associated with data analysis and interpretation. Interpretation and analysis require cognitive processes for operation and immediate interpretation of physical actions, e.g., operating a computer or statistical package, or writing. Interpretation and analysis also involve higher levels of thinking, including the strategies and monitoring involved in analysis. Both of these levels of cognition are, however, significantly influenced by an individual's beliefs about knowledge and knowing; when knowledge is viewed as being tentative, evolving, or context dependent, then there will be greater tendencies to dig deeper into the data, look for hidden relationships, and consider the multiple possible interpretations of the results than if knowledge is considered to be fixed, certain, or made of concrete facts. Studies have therefore found strong associations between students' epistemological beliefs, how they learn, and academic performance (Ryan 1984, Schommer 1993, Rukavina and Daneman 1996, Buehl and Alexander 2001, Hofer 2000, Rodriguez and Cano 2006. Students who believe learning occurs quickly or not at all tend to overestimate how much they understand, leading to oversimplified conclusions (Schommer et al. 1992, Hofer 2000. Similarly, students who believe knowledge is "right or wrong" (dualistic thinking) and consider themselves to have reached understanding when they can recite "the facts" tend to have lower grades than students who believe that knowledge is context dependent (relativistic thinking) or that real understanding is only achieved when they can apply that knowledge to another situation Pintrich 1997, Ryan 1984).
Personal beliefs about the nature of knowledge and knowing have significant implications for understanding the ways in which people interact with and learn about complex social-ecological systems. They influence use of strategies (Schommer et al. 1992, Kitchener 1983, thinking processes (Kardash and Howell 2000), the extent to which mental models and deeper conceptual change occurs during learning (Qian and Alvermann 2000), and abilities to make reasoned judgments (King 1992). These personal beliefs are also highly relevant to understanding conflict, which often revolves around different parties making competing claims about what constitutes knowledge and differences in how people evaluate new information (Hofer and Pintrich 1997 Overall, PEBs have profound impacts on the way people make fundamental decisions that affect themselves and the lives of others (Kuhn 1991, King and Kitchener 1994, Hofer 2001). More sophisticated PEBs are therefore essential for achieving broad sustainability goals and a more pluralistic, involved and politically aware society (Hofer and Pintrich 1997). Unfortunately, while PEBs have a major impact on behavior, most research suggests that a sophisticated view of knowledge and knowing is relatively rare (Kuhn 1991, King and Kitchener 1994, Hofer and Pintrich 1997, and that higher education contributes much less to its development than is often claimed Pintrich 1997, Tsui 1999). Understanding the nature of epistemological beliefs and how they change in relation to different learning environments, subjects of focus, and instructional practices is therefore an important area of educational research that has direct relevance to sustainable development.

Models of epistemological beliefs
Educational psychology approaches the notion of epistemology in a different way than it approaches discussions about cultural interpretations of knowledge, such as the beliefs held by different indigenous peoples (Gegeo and Watson-Gegeo 2001) or differences between general epistemological approaches used in different research disciplines that pose challenges to the integration of knowledge (Van Eijck andRoth 2007, Miller et al. 2008). While these approaches tend to focus on generalized notions of epistemological views, educational psychology has studied the beliefs held by individuals about knowledge and knowing within particular cultural groupings (e.g., a cohort of students) and almost entirely in western education contexts. In these studies PEBs are typically assessed using qualitative interviews or semiquantitative questionnaires. These often use likert items using statements like "truth is unchanging in my subject" or "in my subject, most work has only one right answer". Factor analysis (e.g., principle component analysis) then determines the range and dimensions of beliefs.
A number of different generalized models have emerged from research on PEBs. These models have two important similarities Pintrich 1997, Hofer 2000). First, they suggest a person's beliefs can develop over time along a continuum of increasing sophistication (Perry 1970, Baxter Magolda 1992, Hofer and Pintrich 1997, Hofer 2001, Schommer-Aikins 2002. For example, seminal longitudinal studies in the 1950s and 1960s found that epistemological beliefs of Harvard students changed in a particular directional pattern (Perry 1970). Individuals started with a dualistic belief that knowledge is black or white and could be known. They then increasingly took relativistic perspectives, where the existence of multiple and diverse views and uncertainty were acknowledged to be possible. This was followed by increasing recognition that not all perspectives on a problem were necessarily equal, and that some were better than others. Finally, individuals developed greater ability to commit themselves to a particular view through careful judgment and evaluation of evidence and arguments (Perry 1970). PEBs are therefore not static, and educational programs can be designed to accelerate their development Pintrich 1997, Miri et al. 2007).
The notion that individuals develop more sophisticated PEBs as they mature is closely related to other models of human development. These include a range of stage models that explain how individuals and societies develop from less sophisticated ways of interacting with others and the world (e.g. being driven by desire) towards more sophisticated stages of higher consciousness (e.g., the transcendent self where there is awareness and concern for how one's mind, perceptions etc. influence one's behavior and interactions) (Cook-Greuter 2000). These stage models relate to development of higher states of consciousness, morals, epistemological beliefs, societal and institutional processes, and governance (Cook-Greuter 2000, Nidich et al. 2000, Orme-Johnson 2000, West 2004, Commons et al. 2007) and have been used to understand a range of issues such as terrorism and environmental leadership (Commons et al. 2007, Boiral et al. 2009). As with more sophisticated PEBs, however, attainment of higher stages of consciousness in individuals is rare (Cook-Greuter 2000, Boiral et al. 2009) and there are no societies that have achieved the highest levels of development (Commons et al. 2007). http://www.ecologyandsociety.org/vol15/iss3/art9/ The second key similarity between models of PEBs is that epistemological beliefs are multidimensional (Hofer and Pintrich 1997, Hofer 2000, Hofer 2001). For example, surveys of students, using a wide range of questions, find that beliefs cluster into two main areas: (1) the nature of knowledge (beliefs about what knowledge is); and (2) the nature or process of knowing (beliefs about how a person comes to know something). These areas each consist of two dimensions that can be expressed along a continuum (Table 2). Some models suggest different dimensions develop in tandem (e.g., Perry 1970, Baxter Magolda 1992) but increasingly it is believed that they can develop independently (Schommer-Aikins 2002). Again, these dimensions are often expressed along continua that move towards more sophisticated beliefs which are relatively consistent with other models of human development.

PEBs and interdisciplinary research
While PEBs can change over time, people also appear to flexibly apply epistemological beliefs to different contexts (Elby and Hammer 2001). An example would be a student who applies different beliefs to meet assessment requirements of different instructors. This suggests scope for enhancing skills for interdisciplinary research and practice which rely on integration of a wide range of different epistemological views (Kates et al. 2001, Evely et al. 2008, Miller et al. 2008). Much of interdisciplinary research, however, is rarely truly integrative, and usually involves a particular epistemological or philosophical perspective that is given prominence over those which provide a more secondary supportive role, such as an ecologist framing the research and later inviting a social scientist to contribute to "add value" to the work (Evely et al. 2008, Miller et al. 2008). To achieve more complete integration, epistemological pluralism is advocated where researchers work together to find ways to accommodate each others' beliefs through a continuous process of negotiation (Miller et al. 2008, Healy 2003. Those who have developed greater flexibility in understanding and applying different PEBs could act as knowledge Table 3. Some of the kinds of instructional practices and conditions that are likely to stimulate development of more sophisticated epistemological beliefs (From: Miri et al. 2007, Macllellan and Soden 2004, King and Kitchener 2002, Hofer 2001, Southerland et al. 2001, Tsui 1999, Dennick and Exley 1998, Dole and Sinatra 1998, Brown 1997, Terenzini et al 1995, King and Kitchener 1994.

Actively engage students in higher level thinking
• Raising doubt about the adequacy of understanding about an issue to promote reflective thinking e.g. through analysis of ill-structured and real-life problems. Problem-based learning also increases relevance of the work to students increasing motivation, and assists development of skills for dealing with messy real-life issues.
• Explicit requirements for students to think about knowledge, evidence, reliability and validity etc. (e.g. through practicing and being taught skills of gathering and evaluating data).
• Promote reflection and active examination of assumptions about knowledge through extensive participation of students in class and requiring students to work collaboratively, especially when they are given opportunities to reflect on how their knowledge is constructed. Participation and cooperative learning can occur in focused discussions, student-led seminars, problem based learning, role play etc. Discussion around controversial issues may be particularly useful.
• Students need to be actively engaged in the classroom, not just seen as recipients of transmitted information.
• Discussion of what counts as evidence needs to be a feature of course design.
• Students should be required to give reasons for the content of their work as this can reveal the level of sophistication of thinking e.g. by requiring students to relate to the evidence or the source of the information they provide.
Provision of an appropriate environment for promoting engagement with higher level thinking.
• Teaching needs to stimulate students to ascertain for themselves what is epistemologically confusing or problematic.
• Instructors need to show respect for student assumptions regardless of developmental level.
• Students require feedback and support, e.g. at both cognitive and emotional levels.
• Individuals need to be motivated to process new information.
• Students need to comprehend information and find it plausible in order to feel some comfort with the task, e.g. use of generally familiar knowledge.
• Students need to be accountable for their own goal achievements to get beyond surface level engagement e.g. through provision of appropriate rewards and environments that reinforce student expectations that they need to be accountable for their own learning.
brokers in interdisciplinary research teams to assist the negotiation process.

Factors influencing change in PEBs
A key question in educational psychology is how the development of more sophisticated PEBs can be accelerated. While there are relatively few studies of the influence of the teaching of specific instructional practices on epistemological beliefs (Valanides and Angeli 2005), there is considerable understanding about the general kinds of practices and environments that are most likely to affect them (Table 3). In general, students must be encouraged in using appropriate processes, incentives, and environments to actively engage in reflective and critical thinking about their work and the work of others. In particular, teaching needs to stimulate students to ascertain and resolve for themselves what is epistemologically confusing or problematic Ecology and Society 15(3): 9 http://www.ecologyandsociety.org/vol15/iss3/art9/ (Valanides and Angeli 2005). That is, students need to be encouraged to engage in thinking about their thinking, especially regarding their beliefs about knowledge and knowing (Maclellan and Soden 2004).
The approach taken to teaching is also important. Traditionally, theories of teaching and learning assume that knowledge is a sort of commodity which can be passed from person to person in inert form. Learning, however, is increasingly being understood as something where people learn by being active in the process, often through working in collaboration, discussion, and deliberation with others (Maclellan and Soden 2004). Traditional modes of teaching, where knowledge is "delivered" or "transmitted" from the instructor to the student are especially unhelpful because they reinforce perceptions that instructors are the authority on correct knowledge and because they reduce tendencies for students to make sense of ideas, reflect, and analyze issues for themselves. Appropriate facilitative styles of teaching, rewards, and incentives are therefore needed to ensure that students take ownership and responsibility for their own learning (Valanides and Angeli 2005).
Instructional practices aimed at facilitating the development of more sophisticated forms of higher order thinking are, however, more effective if they are taught within subject matter, rather than when the principles of critical thinking are taught separately (Valanides and Angeli 2005). Differences have also been observed in the PEBs of students in different courses, such as science students compared to psychology students (Hofer 2000), and between students of theology, pharmacology, and psychology (Kaartinen-Koutaniemi and Lindblom-Ylanne 2008). While the causes of these differences are not fully understood, different disciplinary contexts create certain expectations, espouse particular beliefs, and employ different types of assessments and rewards which then influence the PEBs of students. In addition to instructional practices, certain subjects of focus, with their underlying epistemological assumptions, are therefore likely to influence PEBs.

Resilience thinking and PEBs
The teaching of resilience thinking has considerable potential as a subject of focus for facilitating change in PEBs for three main reasons. First, resilience thinking directly encourages thinking in ways consistent with more sophisticated views of knowledge and knowing as they are described in educational psychology (Table 2). For example, to think in terms of resilience, knowledge must be viewed as being tentative and evolving in order to work appropriately with uncertainty and ignorance (thereby influencing beliefs about the dimensions of certainty and simplicity). Much of the potential influence is due to foundations in complexity science (Folke 2006), which requires thinking in terms of non-continuity, uncertainty, inseparability, and unpredictability (Ackoff 1996) and challenges traditional ways of mechanistic thinking (Wulun 2007). The process of developing an understanding of a system's dynamics also encourages inquirers to assess their own relationship with the system (Cavana and Mares 2004, Stave 2002, Vaill 1996, requires active application of critical thinking (Cavana and Mares 2004), and influences mental models and behavior (Cavaleri and Sterman 1997). Compared to those who do not think systemically, systems thinkers are also more likely to have beliefs associated with broader entities that have moral worth, such as ecosystems and living organisms (Luckett 2004), and they are more likely to be able to understand and deal with complex problems (Maani and Maharaj 2004). Resilience thinking also provides an additional layer to systems thinking, i. e., that of a broader framework and set of concepts to help analyze complex social-ecological systems. This framework requires learners to consider their own perceptions of where knowledge originates ("source" dimension) and how they justify their opinions about knowledge ("justification" dimension), such as by emphasizing the value of local and indigenous knowledge (Olsson et al. 2001, Fazey et al. 2006).
Second, because resilience thinking openly acknowledges and aims to deal with uncertainty, the best way to learn about and understand resilience thinking is by applying it to messy real-world problems. These are precisely the sorts of case studies trainee thinkers need to focus on to facilitate development of more sophisticated PEBs (King and Kitchener 1994). Practicing the tackling of illstructured problems can expose students to different perspectives about the issue under investigation. It can also provide the foundation and relevance for teaching that explicitly engages students in discussions about knowledge and knowing (Lattuca et al. 2004). Further, because teaching around illstructured problems emphasizes limits to By the end of the module, students were expected to be able to demonstrate competence in: Evaluating the underlying drivers of, and feedback processes within, changing systems; Evaluating the likely effects of management and policy interventions on a social or environmental system; Critically reviewing how human behaviour gives rise to unsustainable activity; Critically discussing the requirements for societies to be sustainable; Explaining some of the processes that enhance the sustainability of societies.
knowledge and that there are multiple ways of looking at a problem, expectations for the teacher to be the expert or external authority can also be challenged. This provides greater space for students to take control of their own learning and make sense of the problem for themselves, which has significant potential for facilitating epistemological development (Maclellan and Soden 2004).
Finally, resilience thinking is sufficiently flexible, relatively easy to grasp and understand, and relevant to a wide range of contexts (Allison and Hobbs 2004, Armitage and Johnson 2006, Carpenter and Folke 2006, Mumby et al. 2007, Boyd et al. 2008, McDaniels et al. 2008. This enables the epistemicrelated aspects of resilience thinking to be taught to a wide range of social, political, and environmental sectors and at different educational levels. This means it can be taught to people with different backgrounds simultaneously, thus encouraging greater exposure to different epistemological perspectives.

A MODEL FOR THE TEACHING OF RESILIENCE THINKING
This section provides an example of how resilience thinking can be taught in ways that specifically aim to facilitate the development of more sophisticated PEBs. It describes the rationale behind both the design and structure of a module taught to final-year undergraduate students. It is not intended that this model be adopted by all teachers of resilience, and some or all of the components can be replicated depending on specific requirements.

Aims and design rationale
Resilience thinking was taught through a module called "Sustainable Societies" which contributed to one-twelfth of final-year, science-based undergraduate courses. It was designed for 10 to 30 students from a wide range of undergraduate courses, including sustainable rural development, tourism management, and countryside management (which included some students focusing on broader human-environment issues, and others focussing more on ecological dimensions).
Overall, the module aimed to improve students' abilities to evaluate the interrelated aspects of development and sustainability. It set out to teach students to think in systemic ways through in-depth evaluation of feedback in case studies using conceptual modelling as an analytical tool and and resilience thinking as a theoretical base. Student performance was assessed in relation to five explicit learning outcomes (Table 4).
The module was also designed to encourage development of higher level thinking. It was based on the assumptions that the students' would be influenced by both teaching the subject of resilience/systems thinking and by using appropriate instructional practices and learning environments. Further, to enhance the development of analytical http://www.ecologyandsociety.org/vol15/iss3/art9/ skills, principles of variable practice and reflection were also applied. These principles are derived from the educational psychology literature and suggest how performance of a skill or ability can be improved in ways that enable the learner to more flexibly use these skills in novel situations. In simple terms, performance can be improved by practice and reflection. Varying the ways in which these aspects are conducted, however, also increases adaptability (see Fazey et al. 2005 and for details of the principles and how they relate to sustainability and adaptive management). The principles of variable and reflective practice provided a coherent structure for teaching different cognitive skills while ensuring that the principles were implemented together with consideration of appropriate subject matter (see below).

Structure of the module
The module was taught in eleven 2-hour lectures and three 3-hour workshops (Table 5). Throughout the module, students analyzed one of three case studies. Case studies consisted of 4-page summaries of a complex social-ecological system. They included: The impact of snowmobiles on reindeer herding communities in northern Scandinavia (Pelto 1973); the consequences of levees only policies for flood control in Nyngan, Australia (Newell and Wasson 2002); and lock-in effects of mass tourism in Kavos, Greece.
Lectures, which involved considerable active engagement between the students and the instructor, were broken up with discussions and short exercises. Students were regularly required to relate class material to their case studies during class exercises, workshop exercises, and their assignments. They were also encouraged to ask questions and discuss relevant issues as they arose. Small group work was a part of all class and workshop exercises. In most cases the group work was then related to the class as a whole either through discussion or presentation of the results, or by students examining the work of others. That is, wherever possible, attempts were made to facilitate reflection and the sharing of ideas and perspectives.
The learning outcomes were assessed through a 3000-word report that each student wrote about the analysis of the case study and the application of resilience thinking. The process of practicing the different steps that led to the completion of the final report included six subsidiary assignment components ( Table 6). Three of the assignment components required students to apply systems and resilience thinking to their case studies. In these components, students reflected on the adequacy of their own work by completing marksheets which were handed in with the assignments. The instructor provided simple comments on this work in two ways: (1) by working through these marksheets to indicate where there were discrepancies between the student's and instructor's evaluations of the quality of the work; and (2) by providing a simple coded indication of the quality of the work using 34 basic criteria which were then discussed in class. The other three subsidiary assignments required students to use the marksheets to assess the work of one of their peers for each stage in the development of the report.
In general, the subsidiary assignments, class exercises, and workshops varied the ways that different cognitive skills were practiced. The selfreflection, the feedback from peers, feedback from the instructor, class discussions, and group activities varied the way in which students reflected on the work and the subject of focus (Table 7).
Overall, the module enabled students to: (1) develop extensive understanding of the dynamics of their case studies; (2) understand how concepts from resilience thinking helped them to make judgments about the state of their system of investigation; (3) view and appreciate differences between systems and student perspectives; and (4) identify leverage points in space and time to achieve transformation of those systems. Thus students were ultimately required to conduct in-depth investigations of the uncertainties and complexities inherent in their own case studies and the case studies of others through the lens of resilience thinking, potentially exposing students to different epistemological perspectives, especially with regards to the uncertainty and complexity of sustainability related problems. The use of extensive and structured discussion, reflection and practice around these topics provided an appropriate subject of focus, method, and environment conducive to facilitating the development of PEBs. http://www.ecologyandsociety.org/vol15/iss3/art9/ Table 5. Lecture and workshops in the module. L = lecture and Ws = Workshop. All class exercises were collaborative with students working in small groups which usually included students working on the same case studies.

Session
Material and exercises

L1
Viewing the world as a complex system 1 Students filled in questionnaire on personal epistemological beliefs, were introduced to dynamic complexity, balancing and reinforcing feedback in systems and causal loop diagrams. Students required to study for class test in next lecture.

L2
Viewing the world as a complex system 2 Class test on understanding causal loop diagrams and identifying balancing and reinforcing feedback in systems. Causes of dynamic complexity and implications explained. Exercise to develop understanding of how simple system behaviour gives rise to complex emergent processes.

L3
Resilience 1: Determinants of ecological resilience Explanation of resilience, stable states, adaptive systems and 'ball in the bowl metaphor' and ecological determinants of resilience. Exercise for students to build hypothetical ball-in-bowl model for sustainable tourism. Homework set for identifying reinforcing and balancing feedback in causal loop diagrams.

Building influence diagrams
Exercise about the sustainability of grazing practices in the Sahel used for practice in identifying variables and making causal linkages between them. Concepts such as system boundaries and causal linkages introduced and the use of Vensim software for building influence maps explained. Students worked in small groups to identify component variables and started making linkages between them. First component of assignment set.

L4
Resilience 2: Determinants of social resilience Importance of adaptive capacity through individuals, institutions, communities and societies. Exercise for students to consider different adaptive strategies that people use to deal with flooding of a housing development on a floodplain.

Adaptive cycles and Panarchy
Adaptive cycles and Panarchy explained. Students are required to apply the theory to their case study and identify which phase their case study might be in, what the other phases might have looked like, the scale and the influence of other scales on their case study. Second component of assignment set.

Ws2
Building causal loop diagrams Using Sahel case study, students work collaboratively to build causal loop diagrams of the system. Third component of assignment set.

L6
Leverage Short introduction to concept of leverage points in systems. Account by expert on factors influencing malaria in Africa. Students then required to build a conceptual model of the system and identify the most appropriate points to influence the system.

Ws3
Application of resilience thinking With small groups, students were required to relate resilience thinking to their case studies. Fifth component of assignment set.

L8
Ecological feedback Explanation of the concept of bringing ecological feedback into sustainable development using case study from Australia. Local and Traditional ecological knowledge discussed, and exercise used on how the technological development influences how societies interact with their environment.

L9
Adaptive management Concept of adaptive management (active and passive explained) using case study on mink eradication in Scotland. Students required to relate the work to their own case studies. Sixth component of assignment set.

L10
Stretch goals, backcasting and scenario planning Introduction of some of the tools for transforming systems towards more sustainable trajectories. Exercise for students to apply them to their own case studies.

L11
Limitations to system's and resilience thinking and module review Post-test of questionnaire and review of the module, with discussion about the advantages and limitations of resilience thinking. Submission of the final assignment.

Methodology
To evaluate the module described above each student completed, both before and after delivery of the module, a 27-statement questionnaire from Hofer (2000). In the questionnaires, students indicated on a 1-5 scale the extent to which they agreed with the statements (see Table 8 for examples).
Given the low sample size (n = 23), use of factor analysis to determine the dimensions from the data collected was not possible. Instead, each statement was assigned to one of the four dimensions described by Hofer (2000) ( Table 2). Average scores for the statements relating to each dimension were calculated and used for analysis, with low scores being equivalent to more sophisticated PEBs (as defined by Hofer 2000). As there was very little difference in the way the teaching intervention was delivered between years, data from students from 2007 (n = 16) and 2008 (n = 8) were combined. Multivariate repeated measures was then used to determine differences between pretest and post-test PEB scores (within subject tests) and between male and female students (between subject tests). Academic performance was obtained by taking the average of marks for final assignments produced by each student as assessed by two independent instructors. Forward stepwise multiple regression was then used to determine relationships between the average scores for each dimension from the posttest questionnaire and the academic performance of students. All error bars are expressed below are standard error. SPSS 16.0 was used for statistical analysis.

Results
Overall, there was a significant difference in PEB scores before and after delivery of the teaching intervention with a small change towards more sophisticated PEBs (F 18 = 2.938, p = 0.049). There was no overall difference between males and females (F 18 = 1.879, p = 0.158) and no statistically significant interaction between pretest and post-test scores nor between males and females (F 18 = 1.695, p = 0.195). Univariate tests for each dimension found that the overall difference in pretest and posttest PEB scores was mostly due to a change in how students perceived the certainty of knowledge (scores of 2.62 +/-0.1 and 2.33 +/-0.09 for pretest and post-test respectively). That is, students had increasingly viewed their broad subjects of study (countryside management, tourism, etc.) as being characterized by more uncertainty than they had before the intervention (F 21 = 12.066, p = 0.002, Fig.  1a). This difference, however, was largely due to a change in the perceptions held by male students http://www.ecologyandsociety.org/vol15/iss3/art9/ Table 7. Sources of variation in the way elements were practiced and the way the practice was reflected on.

Element
Variable ways in which element was practiced Variable ways in which students reflected on their practice:

Identifying system components
• In different case studies in class exercises; • In different case studies in workshops; • During in-depth analysis of chosen case study.
• Formal self reflection on ability to apply theory in assignment component 1; • Formal feedback from peers on ability to apply theory in assignment component 2; • Formal feedback from tutor on ability to apply theory from assignment component 1; • Reflection in group work in class exercises (e.g. lecture 6) and workshops 1 and 2.

Linking system components
• In different case studies in class exercises; • In different case studies in workshops; • During in-depth analysis of chosen case study.
• Formal self reflection on ability to apply theory in assignment component 1 and 3; • Formal feedback from peers on ability to apply theory in assignment component 2 and 4; • Formal feedback from tutor on ability to apply theory from assignment component 1 and 3.

Building causal loop diagrams
• In different case studies in class exercises; • In different case studies in workshops; • During in-depth analysis of chosen case study.
• Formal self reflection on ability to apply theory in assignment component 3; • Formal feedback from peers on ability to apply theory in assignment component 4; • Formal feedback from tutor on ability to apply theory from assignment component 3.
• Reflection in group work in class exercises (e. • Formal self reflection on ability to understand and apply systems models in assignment components 1, 3 and 5; • Formal feedback from peers on ability to apply theory in assignment components 2, 4, and 6; • Formal feedback from tutor on ability to apply theory from assignment components 1, 3, and 6; • Reflection in class discussions and exercises on different components of theory with exposure to perspectives from tutor, peers working on same chosen case study, and peers working on different chosen case study (all lectures and workshops); • Formal reflection on the process in which students developed understanding in the final assignment. Fig. 1. (a) Average scores for the different dimensions of epistemological beliefs for all students before and after teaching Resilience Thinking (n = 24); (b) Average scores for the different dimensions for males and females before and after the teaching intervention (n = 24). Error bars are standard error. Low scores relate to more sophisticated PEBs as defined by Hofer (2000).
about the certainty of knowledge. That is, the views held by the male students after the intervention changed to levels that were similar to those held by the female students; the perceptions held by the female students did not change after delivery of the teaching intervention. This was demonstrated by a statistically significant interaction between pretest and post-test scores and gender for the dimension of certainty (F 21 = 6.232, p = 0.021, Fig. 1b, Table  9). There were no other significant differences between the four dimensions of knowledge and knowing for the univariate tests (Table 9).
One of the possible explanations for the change in epistemological beliefs was that students had formulated a clearer understanding of their own PEBs or had become more familiar with the questionnaire. This may have resulted in greater consistency in the average scores per student for each dimension in the post-test questionnaire and thus would have affected statistical results (Fig. 1a). Multivariate repeated measures tests using coefficient of variation in scores for each dimension for each student were, however, not different (pretest vs. post-test: F 18 = 1.100, p = 0.387; sex: F 18 = 0.341, p = 0.847; interaction between test and gender: F 18 = 0.955, p = 0.456), suggesting that the change in PEBs was genuine rather than being an artifact of lower variation around PEB scores in the post-test questionnaire.
Multiple regression found that two of the four dimensions were statistically related to academic performance (Table 10, Fig. 2). First, students who had a greater tendency to view knowledge as being justified by careful evaluation of evidence (low scores for the dimension of justification for knowing) performed better than those who had a stronger belief that knowing was justified through what felt right. Second, contrary to expectation, students who had a greater tendency to perceive the source of knowledge to be from external, expert sources performed better than those who had a greater tendency to see the source of knowledge as being constructed by interaction with their environment and others. There was no other statistically significant relationship between academic performance and beliefs about the certainty and simplicity of knowledge (Table 10, Fig. 2).

Implications of the results for the teaching of resilience thinking
Despite the relatively short duration of the teaching intervention, the results show that there was a small but statistically significant increase in the sophistication of students' PEBs during the period in which the module was delivered. Other modules taught at the same time may well have influenced the results, but this seems unlikely given that many of the students were studying different modules and that most of these modules were not directly designed to influence higher order thinking. Change in PEBs was mostly due to a change in beliefs about the certainty of knowledge (Fig. 1). This is not surprising given that the module focused heavily on analysis of the inherent uncertainty in complex systems and the relationships between system components. Nevertheless, given the small sample size and lack of robust controls, the results of the evaluation need to be interpreted with caution.
There are three main reasons that could explain why the changes in PEBs were relatively small. First, students had already been exposed to a wide range of views and perspectives through multidisciplinary teaching and a variety of practical and theoretically based assignments. The teaching of resilience thinking may therefore have been unable to significantly challenge their existing higher order thinking. Second, while the module did encourage students to wrestle with real-world problems, it did not directly include extended discussions or exercises that were explicitly designed with the intention of exposing them to different epistemological perspectives. More extensive exercises, such a role-plays where students are required to actively view an issue from different epistemological perspectives, may therefore be required. Finally, the results suggest that students who believed that knowledge came from external sources (experts, instructors, etc.) had higher grades than those who believed that they had to make sense of something for themselves, such as through careful evaluation of evidence. This may have been because students were too constrained by the assessment method and had a perception that they needed to work towards the instructor's ideas of Table 8. Examples of statements used in the PEB questionnaire against which students expressed the extent to which they disagreed or agreed on a 1-5 scale likert scale.

Dimension Example
Certainty of knowledge Truth is unchanging in my subject.
What is accepted as knowledge in my subject is based on objective reality.
Simplicity of knowledge Ideas in my subject are really complex.
All experts in my subject area understand the subject in the same way.

Source of knowledge
The most important part of work in my subject is coming up with original ideas.
If you read something in a textbook for my subject, you can be sure it is true.
Justification for knowing In my subject, it is good to question the ideas presented.
First-hand experience is the best way of knowing something in my subject area.
what was the "right answer" to get the best grades. This suggests that the module did not sufficiently employ facilitative styles of teaching, and in future the module needs to more carefully align assessments to the desired epistemologically related learning outcomes.

VALUE OF TEACHING RESILIENCE THINKING
This paper has demonstrated a strong theoretical basis and some preliminary empirical evidence that the teaching of resilience and systems thinking can influence the higher order thinking of students. This raises key issues about how the teaching of resilience thinking might contribute more generally to sustainability education. Importantly, the teaching of resilience thinking may be more useful for helping some people than it is for other people in developing analytical skills for dealing with complex problems. It might, for example, have greater impact on PEBs if it is taught much earlier than with final-year undergraduates or at preuniversity levels. Resilience related concepts could, for example, be taught at a variety of educational levels (Fazey et al. 2007) in similar ways to the teaching of other forms of critical thinking (Briggs 1999).
The teaching of resilience thinking may also have more effect on people from some backgrounds than others. This relates to the underlying epistemological assumptions of resilience thinking itself. A system's view is only one of many that are necessary for understanding complex processes (Kline 1995) and while both quantitative and qualitative methods can be used to understand systems, systems approaches still often tend to favor positivist views of the world. Consequently, resilience thinking does not necessarily cater for the multiple ways that people can experience the same event or circumstance, which is a fundamental premise of a large portion of social research (Evely et al. 2008). A further criticism of resilience thinking is that it can overemphasize the biophysical structure of a system and underrepresent the dynamic influence of the humans within it (McLaughlin and Dietz 2008). In other words, the ecological foundations of resilience thinking have resulted in a set of conceptual tools that may have limited capacity to explain the human social component of social-ecological systems.
These limitations have important implications.
Research relating to resilience is, for example, most often published in scientific journals and presented in a language and form that is more accepted by the scientific community than the many other important ideas and concepts related to sustainability. Its http://www.ecologyandsociety.org/vol15/iss3/art9/ Table 9. Results of univariate tests for differences in PEBs before and after delivery of the teaching intervention and for interactions between pre-and post-test scores and sex of the students. relative acceptability in the science domains may be partly because resilience thinking provides a convenient stepping stone between certain levels of epistemological thinking that are more in line with broader transitions in western worldviews of how humans relate to their environment (Metzner 1994). That is, in science domains, resilience thinking may be sufficiently comfortable to be accepted, but sufficiently challenging to instigate change in the beliefs about the processes and dynamics of socialecological systems. While its use in non-sciencebased fields is increasing, it could be having less impact either because relativistic thinking may already be more common or because it is not accepted due to its limited capacity to cater for human social processes.
The relationship of resilience thinking to other models of human development, however, suggests that resilience thinking may have a particularly important role to play outside of academia. The characteristics of a typical resilience thinker are most similar to those associated with the end of the "conventional" stages and the beginning of the "postconventional" stages of the human and societal development model (Cook-Greuter 2000, Commons andGoodheart 2007). So, even though there are many stages that go well beyond the level of sophistication of a resilience thinker (Cook-Greuter 2000 Nidich et al. 2000, Orme-Johnson 2000, Commons and Goodheart 2007, resilience thinking seems particularly well placed for helping facilitate transition at the current levels of human development. Thus, the extent to which resilience thinking will influence higher order thinking will depend on an individual's current level of development. Finally, the paper highlights that the way in which resilience thinking is taught probably has at least the same, if not greater, effect than the teaching of the subject itself. The teaching of resilience thinking will therefore have much greater impact if it is taught through modern, research-led instructional practices that are also known to promote development of more sophisticated epistemological beliefs. This paper has, for example, pointed to the importance of: (1) exposing students to epistemologically challenging situations in ways that require them to directly engage in thinking about their thinking; (2) developing assessment methods that are constructively aligned with the intended epistemologically related learning outcomes; and (3) teaching in ways that reduce traditional transmission modes of information delivery to increase students' ownership and responsibility for learning. This can be difficult in many higher education settings where students are already conditioned to be passive recipients of knowledge, and where facilitative teaching styles are less common or not valued. In conclusion, resilience thinking will not be a panacea that fully explains how the world works. It does, however, seem particularly well suited to being a conceptual tool that can help bridge different epistemological perspectives, assist people to think differently about how they view and interact with social-ecological systems, and influence how they perceive the world more generally. Provided resilience thinking is taught appropriately, it could be a useful tool for contributing to broader sustainability goals. To make the best use of the conceptual framework provided by resilience thinking, greater consideration and exploration of its impact on PEBs and other forms of higher level cognition is needed. This will inevitably require multimethod approaches that follow the principles of epistemological pluralism.