|Home | Archives | About | Login | Submissions | Notify | Contact | Search|
Copyright © 2005 by the author(s). Published here under license by The Resilience Alliance.
Go to the pdf version of this article The following is the established format for referencing this article:
Bammer, G. 2005. Integration and Implementation Sciences: building a new specialization. Ecology and Society 10(2): 6. [online] URL: http://www.ecologyandsociety.org/vol10/iss2/art6/
Synthesis Integration and Implementation Sciences: Building a New Specialization
1National Centre for Epidemiology & Population Health, Australian National University
Developing a new specialization—Integration and Implementation Sciences—may be an effective way to draw together and significantly strengthen the theory and methods necessary to tackle complex societal issues and problems. This paper presents an argument for such a specialization, beginning with a brief review of calls for new research approaches that combine disciplines and interact more closely with policy and practice. It posits that the core elements of Integration and Implementation Sciences already exist, but that the field is currently characterized by fragmentation and marginalization. The paper then outlines three sets of characteristics that will delineate Integration and Implementation Sciences. First is that the specialization will aim to find better ways to deal with the defining elements of many current societal issues and problems: namely complexity, uncertainty, change, and imperfection. Second is that there will be three theoretical and methodological pillars for doing this: 1) systems thinking and complexity science, 2) participatory methods, and 3) knowledge management, exchange, and implementation. Third, operationally, Integration and Implementation Sciences will be grounded in practical application, and generally involve large-scale collaboration. The paper concludes by examining where Integration and Implementation Sciences would sit in universities, and outlines a program for further development of the field. An appendix provides examples of Integration and Implementation Sciences in action.
Key words: complexity; integration; knowledge management; participatory methods; synthesis; systems thinking; uncertainty
Researchers, funders, and research end-users are increasingly appreciating that new research skills must be developed if human societies are to be more effective in tackling the complex problems that confront us and in sustaining the sort of world we wish to live in. Researchers must collaborate and integrate across traditional boundaries. They must bring together academic disciplines, and become more involved in the implementation of research in policy, product, and action.
There is now a critical mass of researchers who have turned their efforts to meeting these challenges. By working on real-world problems, they have made theoretical and methodological advances to help deal with complexity, uncertainty, change, and imperfection—the primary characteristics of the vital issues modern societies face. Developments have occurred in research on the environment, public health, business and management, national security, and other applied topics. Nevertheless, these advances are not embedded in mainstream academic activity. At best, they have led to issue-focused, cross-disciplinary research centers. At worst, individual researchers are isolated at the margins of their departments. Although some argue that such arrangements are desirable because they are flexible and adaptable, they also come at a great cost. The limited interaction and communication across different areas of application mean that, for example, researchers in environment are not exposed to developments in public health or business or security. Thus, there have been only low levels of intellectual cross-fertilization and learning, and limited exploitation of the significant synergies between approaches.
Mainstream research is progressively starting to embrace these new investigative imperatives, but is even more poorly connected to existing knowledge. This has led to considerable duplication of effort, reinventing, usually at less sophisticated levels, methods and frameworks that already exist.
The time is ripe for coalescence and coordination. An effective and efficient mechanism is to develop a new specialization: “Integration and Implementation Sciences.” This will involve bringing together and providing a clear identity and accepted place for a large “college” of peers, who can be both supportive and critical of each other’s work. The aim of such a specialization will be to buttress, not replace, existing centers, informal interdisciplinary programs, and individual efforts. As I outline below, just as the discipline of statistics supports research on myriad problems, the specialization of Integration and Implementation Sciences will strengthen an extensive range of research endeavors. The vision for Integration and Implementation Sciences is to provide solid theoretical and methodological foundations to allow complex societal issues to be systematically addressed using evidence-based approaches. The three pillars will be:
Like statistics and epidemiology, the specialization will advance through application to a diverse range of problems. Similarly, Integration and Implementation scientists will not necessarily have content expertise. As their work will complement, rather than replace, traditional disciplinary and specialist perspectives, collaboration will central to how Integration and Implementation Sciences operates. What Integration and Implementation scientists could contribute to these partnerships includes:
No Integration and Implementation scientist will be expert in all of these skills. However, they will have a broad framework of knowledge encompassing all these aspects, and deep knowledge of some of them. They will be able to bring in colleagues to fill skill gaps. And they will be able to recognize when leading-edge theory and methods are being used, when breakthroughs in thinking have been made, and when wheels are being reinvented. Developing the framework for these skills and allowing for concentration on a subset of them could provide the core of undergraduate and graduate curricula. This is identical to how other specializations and disciplines operate.
This paper begins by briefly revisiting calls for new research approaches to complex problems. It posits that the core elements of these new approaches already exist, but are fragmented because, apart from researchers working in the environmental sciences, the investigators who have developed them are marginalized. It then outlines the core elements of Integration and Implementation Sciences. It examines where Integration and Implementation Sciences would sit in universities and finally outlines a program for the further development of the field.
This paper aims to present a broad sweep of ideas about this proposed new specialization. I focus on the practicalities of what the new specialization would involve and how it would structurally fit, rather than building the case for its necessity or explicitly linking the arguments to the extensive discourse on the philosophy of science, the long-standing debate about the role of scholarship and universities in society, or discussions about the future of science. Readers interested in these issues might wish to refer to works on critical realism (e.g., Mingers 2000), post-normal science (e.g., Funtowicz and Ravetz 1993), and consilience (Costanza 2003, Wilson 1998), as well as seminal works by authors such as Kuhn (1970) and Ravetz (1996). The argument presented here has both commonalities with and differences from the practical aspects of post-normal science and consilience, but developing these is not within the scope of this paper. The specialization I propose has intersections with existing disciplines, particularly political science, which are not teased out here. Furthermore, I only touch lightly on the extensive literature underpinning each of the areas covered in this paper. My aim is not to write a definitive treatise, but to spark discussion and to rally like minds. If I am right that a critical mass of supporters for a new specialization exists, the task of building a comprehensive, consistent, and compelling argument is one for us to undertake together, along with stimulating action to build stronger links between the core methodologies and to embed them more firmly in academic structures.
Examples of Issues where the Key Deficiency is Lack of Integration and Implementation
There is growing appreciation that a major deficiency in our capacity to tackle key national and global problems lies in the inability to amalgamate knowledge created by different disciplines with the experience of key actors and interest groups, and then to effectively use that knowledge to bring about social improvement.
For example, there are ten risks described in the 2002 World Health Report (World Health Organization 2002) that account for one third of premature deaths worldwide, including tobacco smoking, unsafe sex, high cholesterol levels, being underweight, and iron deficiency. These are risks for which proven, cost-effective interventions are available. But human society seems unable to implement integrated solutions in a widespread, large-scale, and coherent manner (see also World Health Organization 2004).
Furthermore, despite some successes, in many areas concerned with sustainability, such as global climate change and biodiversity loss, research evidence and consensus among leading researchers about recommended actions has had little impact on government policy, business practice, or the actions of local communities in either rich or impoverished countries (Board on Sustainable Development 1999, Cash et al. 2003).
Many factors contribute to the inability to implement integrated interventions, including:
Calls for New Approaches
Researchers, research funders, policy makers, business, and civil society are grappling with how research can best meet pressing social, environmental, and technological challenges.
A 1999 United Nations Educational, Scientific and Cultural Organization (UNESCO) report stated:
... it must be recognized that the relationship between scientific research, education, technological innovation and practical benefits is much more diverse and complex today than in the past, and frequently involves many players other than researchers. The progress of science cannot be justified purely in terms of search for knowledge. In addition, it must be defended ... through its relevance and effectiveness in addressing the needs and expectations of our societies. (UNESCO 1999.)
Similarly, in the context of sustainable development, Agenda 21 (a key document generated at the 1992 Rio Earth Summit and being implemented worldwide) called for:
...supporting new scientific research programs, including their socio-economic and human aspects, at the community, national, subregional, regional and global levels, to complement and encourage synergies between traditional and conventional scientific knowledge and practices and strengthening interdisciplinary research related to environmental degradation and rehabilitation. (United Nations Conference on Environment and Development (UNCED) 1992, Article 35.9[a].)
The Organisation for Economic Co-operation and Development (OECD) (1996) has also made a similar point about the knowledge-based economy:
The science system, essentially public research laboratories and institutes of higher education, carries out key functions in the knowledge-based economy, including knowledge production, transmission and transfer. But the OECD science system is facing the challenge of reconciling its traditional functions of producing new knowledge through basic research and [education] ... with its newer role of collaborating with industry in the transfer of knowledge and technology. (OECD 1996.)
Gibbons, Nowotny, and colleagues (Gibbons et al. 1994, Nowotny et al. 2001) have called for recognition of “Mode 2” knowledge production, and have outlined what this entails. Here, problems are defined in the context of application rather than a disciplinary framework. The focus is on developing a transdisciplinary approach: the research is carried out by heterogeneous non-hierarchical groups that come together transiently, and are based outside universities. The researchers interact with the relevant social actors to ensure a greater degree of social accountability, and quality is judged by a wider range of criteria, using reflexive processes. They argue that Mode 2 knowledge production challenges the traditional role of universities. Funtowicz and Ravetz, (1993) make similar points in a different argument structure in describing post-normal science.
Implementing New Approaches
Grappling with the new role for research has not been confined to rhetoric or the isolated activities of committed groups. Policy makers, in particular, are using their funding clout to bring about change in the conduct of research.
For example, the U.S. National Science Foundation (NSF) (2001) has “Promote Partnerships” as one of its “three core strategies that guide the entire agency in establishing priorities, identifying opportunities, and designing new programs and activities,” namely:
Collaboration and partnerships between disciplines and institutions and among academe, industry and government enable the movement of people, ideas and tools throughout the public and private sectors. Furthermore, these partnerships optimize the impact of people, ideas and tools on the economy and on society. (U.S. NSF 2001.)
The 2001–2006 Strategic Plan of the NSF also recognizes that “The escalating complexity of science and engineering is moving research toward a collaborative mode with greater focus on intellectual integration.” (U.S. NSF 2001.)
Similarly, the European Commission’s 6th Framework Program funding mechanism allocates over 80% of its 16 billion Euro budget to projects that “integrate and focus” European research. It has further introduced “Integrated Projects” as a new funding mechanism within that program (European Commission 2002).
In Australia, a Cooperative Research Centres Programme (Australian Government Cooperative Research Centres Programme 2004) was established in 1991 to draw together a range of researchers and end users in six sectors: environment, manufacturing technology, agriculture and rural-based manufacturing, medical science and technology, information and communication technology, and mining and energy. Although the focus now is on commercially oriented centers, with business groups as the end users, earlier incarnations also included centers with “public good” orientation, with government departments, community groups, and similar public clients. In all cases, researchers are charged with crossing disciplinary and organizational boundaries, as well as extending beyond the scientific domain.
Another Australian example comes from Land and Water Australia, a leading government funding body for research and development in natural resource management. They have adopted integration as a key theme for their current and future development. Their 2001–2006 Strategic Plan (Land and Water Australia 2001) states that:
... to be useful and influential, ... R&D must be closely integrated with other policy instruments and with the objectives of key stakeholders at policy and management levels. This is a major priority of the Land & Water Australia Board and a critical emphasis within this Strategic R&D Plan.
On the whole, funders are not mandating how integrative activities should occur, but are giving researchers lee-way in developing new methods and frameworks. Some, such as Land and Water Australia, have funded reflection on how integration works (Land and Water Australia 2005), whereas others, such as some Cooperative Research Centres, have willingly participated in research into their processes (van Kerkhoff 2002).
The Task for Integration and Implementation Sciences—In Brief
The examples above show that the calls for improved integration and implementation are widespread and diverse. Nevertheless, although they broadly run along the same themes, the calls do not cohere into a single, easily definable problem or solution. One of the tasks for Integration and Implementation Sciences will be to define the similarities and differences across this range of contexts, and so build a more robust, sophisticated, and subtle approach to these issues.
Along with responding to these calls for change, Integration and Implementation Sciences will also need to protect and strengthen core elements of intellectual activity, such as independence, dispassion, and creativity. In the U.S. context, Bok (2003) focuses particularly on pressures on universities to commercialize, examines what universities can learn from business, and cautions against pursuits that undermine or distort the foundations of academic work. A broader set of reflections by Australian academics on “Why Universities Matter” (Coady 2000) highlights key values and ideals of university research and teaching.
There are increasing numbers of researchers developing skills in integration and implementation. Examples of projects that have built or applied theory and methods that come under the rubric of Integration and Implementation Sciences are presented in
There are also some notable global projects where skills in integration and implementation have been extensively applied, particularly the Intergovernmental Panel on Climate Change (2004) and the World Commission on Dams (2000). There has, however, been little attempt to synthesize the theory and methods used across such projects.
Furthermore, although it can be argued that there is the critical mass of researchers to provide the foundation for a new specialization, the field is far from cohesive. Instead, the field is characterized by:
Such marginalization and fragmentation has many consequences for the field. The preponderance of small groups that are not well networked leads to considerable duplication and reinventing of the wheel. Productive cross-fertilization of ideas is limited, which in turn means that the field does not reach its potential in terms of progress. The high demand for the approaches encompassed under Integration and Implementation Sciences by policy makers, business, and other practitioners, and the associated emphasis on consulting, often leaves little time for reflection, let alone for theory and methodology building.
Multiple groups of small size have costs associated with lack of economies of scale. For example, such groups often have no administrative support, putting a high extra load on research and teaching staff. A disproportionate amount of effort may also have to go into fundraising, especially for self-funded groups either inside or outside the academy. In time, the enthusiasm and energy of staff is ground down, limiting opportunities for networking, let alone innovation.
All this can also contribute to low standing within the academy and a perception that the field lacks rigor and attracts only low-quality staff and students. This perception is exacerbated by the lack of high-impact journals and the other accoutrements of established disciplines and specializations. It is worth noting that the environmental sciences are starting to overcome some of the marginalization issues discussed here, and are developing critical mass in their own right. Integration and Implementation Sciences would not replace existing growing environmental science programs, but their endeavors would be strengthened by providing a coherent framework that could be broadly applied, and that would expose environmental sciences to developments in other areas (such as public health and business management), as well as supply a critical forum for the evaluation of integrative ideas developed in the environmental sciences.
Even though developing a specialization could improve many of the problems outlined in this section, there are associated costs that should be noted, particularly the loss of inclusiveness. The current diffuse networks have the benefit of involving a large number of disparate players. Setting boundaries is likely to exclude some of them, and this will certainly lead to debate and dispute about the mission of the new specialization. On the other hand, these debates can also be helpful, especially if they are structured to sharpen thinking and to develop a greater sense of collegiality among researchers who are now only dimly aware of each other.
There are three aspects to Integration and Implementation Sciences. First is that the specialization will aim to find better ways to deal with the defining elements of many current societal issues and problems—namely complexity, uncertainty, change, and imperfection. Second is that there will be three theoretical and methodological pillars for doing this: 1) systems thinking and complexity science, 2) participatory methods, and 3) knowledge management, exchange, and implementation. Third, operationally, Integration and Implementation Sciences will be grounded in practical application and will generally involve large-scale collaboration.
Defining Elements of the Societal Issues Tackled—Complexity, Uncertainty, Change, and Imperfection
I propose four key elements for the sorts of issues Integration and Implementation Sciences will be designed to tackle, and flesh them out briefly below. They intersect, but do not completely overlap with the attributes of problems amenable to a post-normal science approach, namely that facts are uncertain, values are in dispute, stakes are high, and decisions are urgent (Funtowicz and Ravetz 1993). The four key elements Integration and Implementation Sciences will aim to tackle are complexity, uncertainty, change, and imperfection.
Complexity has many dimensions, including an extensive array of factors, with both linear and nonlinear connections and interdependencies and a range of relevant political, cultural, disciplinary, and sectoral perspectives. Geographical and temporal scales can be huge. An important dimension of complexity is identifying and understanding properties that disappear when a system is studied in disaggregated segments.
A necessary adjunct to complexity is uncertainty. In dealing with any complex issue or problem, there will always be many unknowns, including facts, causal and associative relationships, and effective interventions. Some unknowns result from resource limitations on research, some result from methodological limitations, and some are simply unknowable. There are epistemological, ethical, organizational, and functional aspects to dealing with uncertainty, ignorance, and risk.
The unknowns are compounded by constant change—occurring on many fronts, including biological evolution (e.g., the development of new communicable diseases); scientific, technological, and economic developments; international relations; and manifold intended and unintended consequences of local, national, and international policy and programs.
Perfect knowledge and solutions are impossible. Imperfection too has many dimensions. Dealing with complexity involves setting boundaries to the approach we take, and where we set boundaries is crucial in determining what is included, excluded, and marginalized. Uncertainty and change also necessarily lead to imperfection. Furthermore, social issues are deeply contextualized so that an excellent solution in one person’s eyes is anathema in another’s.
Theoretical and Methodological Pillars: Systems Thinking and Complexity Science, Participatory Methods, and Knowledge Management, Exchange, and Implementation
The key theoretical and methodological foundations to Integration and Implementation Science will be:
These will provide a range of conceptual and methodological tools for dealing with complexity, uncertainty, change, and imperfection, including modeling, decision and risk analyses, deliberative democracy processes, principled negotiation processes, and so on.
Systems thinking and complexity science
Although both systems thinking and complexity science concern themselves with looking at wholes, they encompass several schools of thought, which are noted, at best, for their indifference and, at worst, animosity, to each other. There is not only a gulf between systems thinking and complexity science, but also within different branches of, particularly, systems thinking.
For example, schools of thought encompassed by systems thinking and some of their key practitioners include (adapted from Troncale 2000):
Complexity science has spun off from systems thinking, and specifically tackles systems with self-organizing, emergent properties and nonlinear dynamics. In particular, complexity science deals with systems that share some or all of the following characteristics (CSIRO Australia 2002):
Much of complexity science is highly mathematical, and other conceptual methods are poorly developed. There are three primary approaches (CSIRO 2002):
A large number of methodologies have been developed. In systems thinking, they include clustering theory, comparative systems analysis, computer modeling and simulation tools, control theory, critical path methods, decision analysis, divergence mapping, flowcharting, game theory techniques, input–output analysis, lifecycle analysis, linkage proposition analysis, network theory, optimization theory, relational database analysis, scenario building, viable system diagnosis, strategic assumption surfacing and testing, interactive planning, and critical systems heuristics (Flood and Jackson 1991, Troncale 2000). Complexity science analysis tools include fractals, chaos theory, lattice models, renormalization group theory, and agent-based modeling (CSIRO 2002)
Despite these advances, there seems to be little progress in reaching agreement on key overarching theoretical concepts. As the following three examples illustrate, there does seem to be considerable scope for such unification.
One suggestion for developing common elements in systems thinking has been made by Checkland (1984) who proposes two sets: emergence and hierarchy, and communication and control.
For Integration and Implementation Sciences, hierarchy is valuable in terms of providing a structured way of thinking across scale, showing that systems are not closed, providing a big-picture view, including interactions between local and global, and showing linkages, including between sectors and stakeholders. Hierarchy also sets the context for emergent properties, in other words, properties that exist at one scale, but not at others. For example, wetness is an emergent property of water, a property that cannot be predicted from its component gaseous elements, hydrogen and oxygen. Emergent properties appear when a system is examined as a whole instead of as separate parts, or when separate parts of a system are examined as a coupled framework. Communication and control are important in terms of understanding vicious (reinforcing) and virtuous (balancing) cycles, effective points of intervention, and sources of unintended effects.
A second example comes from Troncale (2000), in his development of a broader range of common elements to unify the physical and life sciences:
Finally, Higginbotham et al. (2001) have described a set of key concepts in complexity science:
These authors have applied complexity science to thinking about health, but the principles they have developed have broader relevance to Integration and Implementation Sciences, namely that:
Even these brief lists indicate apparent similarities and overlaps between the principles for systems thinking and complexity science, so that refining them would seem to be a worthwhile and achievable task. There is likely to be considerable value to Integration and Implementation Sciences in the application of common principles to complex social, environmental, and technical issues. For example, there has already been considerable work on boundaries as social and personal constructs that determine what is included, excluded, and marginalized, and the intimate link between boundary judgments and value judgments (Midgley 2000). Defining the commonalities will also throw the real differences into sharper relief and highlight the particular strengths and limitations of different systems and complexity-based approaches.
In terms of the defining elements of the societal problems of concern to Integration and Implementation Sciences, systems thinking and complexity science will be particularly valuable in allowing complexity to be productively embraced. There is a definitional and semantic issue around the word complexity that will eventually have to be resolved. Complexity science uses the word quite precisely, as indicating the characteristics outlined above, and would prefer the more general social term to be “complicated” rather than “complex.”
Systems thinking and complexity science also provide some insights into uncertainty and, through risk analysis and similar methods, ways of dealing with uncertainty. As this sketchy overview suggests, there is still considerable scope for development of systems and complexity theory and methods.
In systems- and complexity-based approaches, there is growing appreciation of the need to work with affected stakeholders to understand the full range of aspects of any particular system. Nevertheless, considerable development in linking systems thinking and complexity science with participatory methods is still warranted.
Much systems thinking and complexity science occurs within a positivist framework, but there is growing appreciation of different ways of knowing. Thus, some forms of systems thinking have embraced different epistemologies; for example, critical systems thinking is located in a critical social science perspective (Midgley 2000). But the value of marrying systems thinking and complexity science with different epistemologies still remains largely unexplored.
Finally, there is still relatively little formal intersection between systems thinking and complexity science, on the one hand, and other aspects of knowledge management, exchange, and implementation, on the other. Much interaction with decision makers, for example, is still premised on providing them with information, as the major contribution of intellectual effort. Soft systems methods tend to take this further by involving decision makers, and other stakeholders, in developing the evidence through a range of participatory methods.
The importance of participatory methods is based on recognition that the various stakeholders think about the same issue differently, and that exploring, sharing, and synthesizing these different understandings enriches our knowledge about an issue, and can often trigger a new way to look at and contend with it. In addition, for some issues, an appropriate way of dealing with uncertainty and imperfection is to give the stakeholders a more direct role in making decisions.
Integration and Implementation Sciences will focus on participatory methods in a research context; it therefore involves various forms of structured engagement between researchers and relevant social actors or “practitioners,” such as community representatives, business groups, and policy makers. The new specialization will aim to draw on the significant insights of political theory to enhance both thinking about and the practice of participatory engagement.
There are many participatory methods that have been used in research projects and programs, including dialogue (Yankelovich 1999), Delphi methods (Linstone and Turoff 1975), consensus building (Fisher et al. 1991, Susskind et al. 1999), search conferences (Emery 1999), and executive sessions (Hough 2002). In addition, action research is a particular style of highly engaged investigation that generally incorporates systems thinking and an epistemology that is not based on positivism (Reason and Bradbury 2001).
In general, participatory methods encompass a wide range of engagements, and can involve two or more parties, encompassing few to many disciplines and practitioners; they can be short or long term; they can challenge elites or be controlled by them; and they can vary in the degree to which they empower marginalized groups. Participatory methods can enable practitioners and researchers to learn together about problems of common interest in a way that provides reciprocal benefits. They can combine their perspectives to build new concepts, insights, and/or practical innovations, which they could not produce alone.
Brown et al. (2003) have developed a framework for thinking about participatory methods, which can provide an underpinning for Integration and Implementation Sciences. The framework has four key elements:
Thus, engagement between researchers and practitioners must take into account different social, political, and ethical paradigms, different engagement goals and interests, and different expectations about accountability. Furthermore, the relationships and organization must be able to accommodate power differences, build trust, and develop effective control, ownership, work division, and decision-making processes.
Methods and technologies can be divided into four types: a) focused puzzle solving, b) issue exploration and agenda setting, c) intervention and assessment, and d) long-term domain development (Brown 2001). Participatory focused puzzle-solving methods are appropriate when answers are needed to well-defined problems. They make efficient use of the comparative advantages of each party, and do not require expensive, ongoing relations. Issue exploration and agenda-setting methods are appropriate when multiple views are needed for understanding complex, ill-structured problems. They allow many voices to be involved in identifying issue patterns and implications, and set the stage for wide participation in problem solving. Participatory intervention and assessment methods document, analyze, and improve the quality of interventions and best practices. They focus on existing programs and activities, and are particularly useful for identifying costs and benefits of possible solutions. Finally, participatory long-term domain development methods involve ongoing co-inquiry to build perspectives, theory, and practice in new domains. These methods are particularly useful in providing in-depth analysis over the longer term of poorly understood problems. They can produce new paradigms for understanding intractable problems and lead to fundamental changes in theory and practice.
The final element of the framework comprises contextual forces and institutions, and involves taking into account the broad range of political, social, economic, and other influences at play, on global, national, and local scales, at the time of the engagement. It also takes into account the impact of the auspices under which the participatory methods are conducted, and of the institutional bases of the researchers and practitioners.
In terms of enhancing the ability of research to deal with complexity, uncertainty, imperfection, and change in societal issues, participatory methods can make significant contributions. Bringing stakeholders into the picture gives voice to the complexity; as well as being part of the complexity, stakeholders’ involvement should heighten their awareness of other dimensions and perspectives, and their awareness that the solutions are unlikely to be as straightforward as they might at first have thought. Ideally, it will also make stakeholders more conscious of significant unknowns and of the impossibility of finding perfect solutions. It could also make them more aware of the past changes that have led to the current situation, and of future possibilities in the short and long term.
Thus, one function of participatory methods is to raise the level of knowledge of both the researchers and the stakeholder practitioners. Participatory methods can also be used to give stakeholders more say over the focus, conduct, and interpretation of the research, and therefore, invite them to share responsibility for how complexity, uncertainty, imperfection, and change are dealt with in the investigation. As outlined above, action research is a methodology that aims to make stakeholders central in the research, and casts the researchers as facilitators. This intensity of engagement is often not necessary, possible, or desirable. At this stage, there is little to guide researchers on the appropriateness of different levels and types of participation and their strengths and weaknesses. There has also been little formalization of key methodological issues in participatory research, such as building trust.
Although participatory methods have been discussed here in terms of the engagement between researchers and practitioners, they are also relevant to collaborations between researchers from different disciplines. Different disciplines also contribute to both shaping and understanding complexity. They address different knowledge gaps, and approach and tackle uncertainty in a range of ways. Research involving various disciplines can attempt to involve them all in the design, performance, and conclusions drawn from the investigation, or can allocate more limited and defined roles. Again, there is little to guide us on the strengths, weaknesses, and appropriateness of diverse approaches.
Knowledge management, exchange, and implementation
“Knowledge management, exchange, and implementation” is a way of characterizing a number of interrelated issues, particularly:
1. Diverse epistemologies
As outlined above, appreciating diverse epistemologies involves valuing different ways of understanding the world, both in terms of recognizing that everyone has a different world view, and that there are a range of approaches to conducting research.
Along with growing appreciation that people view the world differently, and that there can be significant variations between distinct groups, such as historians and police, there is increasing emphasis on the development of tools for understanding different world views and for enabling diverse groups to work together. For example, Senge (1990) is one researcher who has developed such tools, including looking for leaps of abstraction; mismatches between espoused theories and theories in use; and scenarios and computer simulations. Such tools have often focused on diverse groups within the same cultural context, but there is now also growing interest in the development of transcultural competence (Koehn and Rosenau 2002).
Much has been written about different epistemologies that guide the conduct of research. A highly accessible description is Neuman’s (2003) comparison of positivism, interpretive social science, and critical social science. He compares these epistemologies on a number of dimensions of research, such as the reason for undertaking the research, how each epistemology views social reality, what characterizes good evidence, and the place of values. For example, in terms of the reason for conducting research:
In terms of the place for values:
Although many researchers are clear about the epistemological approach they use in their work, for others the importance of understanding their epistemology has never been raised. They are likely to conduct research as they have been taught, without questioning the underlying world view. In the past, this was largely true of researchers using positivist epistemology, but as other approaches have become more common, it is now also true of researchers using other epistemologies. It is also becoming more common for researchers to use a mix of elements from different epistemologies.
One of the challenges for Integration and Implementation Sciences will be the development of hybrid epistemologies, especially in encouraging social actors to broaden their world views. In terms of research, there will also be the challenge of integrating across different epistemologies.
In terms of facilitating research that crosses disciplines, a distinction is often made between multi-, inter-, and transdisciplinarity.
Multidisciplinarity does not require researchers to leave their disciplines. Such research seeks to tackle an issue by getting researchers in a range of disciplines to use their traditional theories and methods to present an understanding of the issue, and then attempts to integrate these different understandings. The challenge is in the integration, and the methods for this have been poorly developed to date. As a consequence, multidisciplinarity has a rather poor reputation.
In an interdisciplinary approach, researchers are required to look beyond their own discipline, and work with the other relevant disciplines to find areas of overlap that are likely to yield new understandings. Klein (1990) suggests that there are five phases in the interdisciplinary process:
She also argues that there are four models of integration:
Most interdisciplinary work involves small clusters of neighboring disciplines, such as physics, mathematics, and chemistry, or sociology and anthropology. Occasionally, an area of overlap will be so productive as to spawn a whole new discipline or specialization, such as biochemistry or mathematical psychology.
A transdisciplinary approach aims to develop a new common conceptual framework that provides a new level of coherence for the different disciplines. It is analogous to, and perhaps even identical with, emergence in complexity science.
Higginbotham et al. (2001) suggest that there are two general ways of developing a common conceptual framework, namely: having an individual synthesize findings from a multitude of disciplines to provide a comprehensive explanation of a complex issue, or constructing a team whose members bring their combined resources to focus on problem solving.
2. Accessing knowledge
Librarianship has evolved and expanded into the new field of Information Science and Technology, and tackles the issue of providing enhanced methods for accessing both the large volume and the diverse forms of knowledge. For example, the American Society for Information Science and Technology (2004) brings together “individuals who share a common interest in improving the ways society stores, retrieves, analyzes, manages, archives and disseminates information.” The Journal of Information Science (2004) expands this scope by also including the creation and use of knowledge resources, and how policy and practice can “achieve a greater impact on the world economy.”
The American Society for Information Science and Technology webpage (2004) provides a brief history of the evolution of information science:
Another advance in accessing knowledge has been the development of systematic reviews. Two well-known examples are the Cochrane (2004) and Campbell (2004) collaborations, which methodically assess the evidence for healthcare (Cochrane) and social, behavioral, and educational (Campbell) interventions.
3. Understanding how action occurs and is influenced by evidence
Developing better understanding of how policy is made, how business operates, and how activism succeeds, in addition to how action is and can be influenced by evidence, intersects with a number of established disciplines and specializations, including science and technology studies, political science and policy studies, sociology, and business studies. There is also overlap here with a growing literature on knowledge translation (e.g., Lamb et al. 1998) and diffusion of innovation (Rogers 1995). What Integration and Implementation Sciences seeks to do is to aggregate the individual learnings from these areas, again so that productive synergies can occur. In essence, this involves research on research and its effectiveness.
I present a few examples of the sort of research that Integration and Implementation Sciences will seek to encompass and foster.
In a study of integration within two Australian Cooperative Research Centres, van Kerkhoff (2002) has characterized four dimensions of research, the last two of which are particularly relevant here. The first dimension is the individual creativity that is the core building block of research. This can be directed to generating new pieces of knowledge or to integrating new and/or existing pieces. The second dimension involves interaction among researchers, which is necessary for the communication of ideas and quality control, as well as the illumination of a research problem by different perspectives. These can be different theories and methods within a discipline, or can involve integrating, more or less, the perspectives of different disciplines. The third dimension involves the interaction of the research with the larger social system within which it sits. It involves taking stakeholder perspectives into account through a range of participatory processes and other formal and informal mechanisms. The fourth dimension is change through time. This requires researchers to add a new dimension to their concerns, namely the way in which research makes a difference through time. Researchers, policy makers, and other practitioners grapple together with the uncertainties of how the action and research contexts may change over time, and how these contexts can be strategically shaped by research.
A second example comes from the work of Gibson (2003), who argues that neither translation nor transfer adequately capture the complexity of the interaction between research and policy. He posits that transformation is likely to be a more appropriate concept. Gibson investigated how research influenced policy on four public health issues and concluded that success requires more than changing the timing and format of communication about research or even creating joint projects. Instead, he argues that research must be transformed into knowledge that is invested with meaning and power that binds government to a particular view or course of action. It also needs advocacy coalitions that are inspired to see policy reflect their beliefs and values. He contends that research is either “minted” into valuable currency for policy arguments or “muted,” depending on the social context of the justification, the irrefutability of the data, and the immutability of the policy. Finally, he makes the case that research is transformed into knowledge and power when it becomes part of a policy discourse that simultaneously shapes what is being governed and provides the reason and authority for government.
The theoretical insights of van Kerkhoff and Gibson present some important new perspectives for research in Integration and Implementation Sciences, and provide a context in which to embed a range of other insights.
For example, Gibson’s research raises the question of whether research transformation is also necessary to effectively change practice, other than policy. Certainly Moore (1995) has traced the increasing complexity of the research–practice interface in the U.S. Agricultural Extension Service. Extension started with written pamphlets, but farmers found these neither useful nor convincing. The next phase was to hire extension workers to disseminate knowledge, and this has evolved into a third phase where farmers are in a better position to use researchers for their own purposes. Furthermore, Moore has started to catalogue the factors that made the U.S. Agricultural Extension Service a success, including the sheer numbers of researchers and outreach workers, the training of both researchers and farmers to be reflective practitioners, and the thick, strong connections between the land grant universities, experimental stations, and farms.
Other important research insights that can be brought to bear on this aspect of Integration and Implementation Sciences include understanding about the importance of how issues are framed, the building of alliances, and the role of research and researchers in advocacy (Chapman 2001). It also provides opportunities to think about and systematize new roles, such as knowledge brokers and boundary spanners (Cash et al. 2003, Williams 2002). It also helps invigorate appreciation that research (or professional enquiry) is a scarce resource, particularly in relation to the vast number of problems confronting human societies (Lindblom 1990). Thus, there is considerable value in maximizing the use of research as an aid to lay enquiry, so that the bulk of the population is better equipped to appreciate nuance, and deal with uncertainty and imperfection.
Grounding in Practical Application and Collaboration
The third, operational, aspect of Integration and Implementation Sciences will be that it has a firm footing in practical application and generally involves large-scale collaboration. As outlined above this makes Integration and Implementation Sciences similar to disciplines such as statistics and epidemiology. The analogy with statistics, in particular, is drawn out further in examination of where this new specialization sits in universities.
Although the theory and methods of Integration and Implementation Sciences will be developed through engagement with practical problems, there is no home base to which breakthroughs can be reported, and where they can be critically assessed. This is an important difference from disciplines like statistics, where such home departments play a critical role. The development of the specialization of Integration and Implementation Sciences will be a way of establishing such a home base.
The lack of a specialist or disciplinary core also means that those engaged in Integration and Implementation Sciences do not have a unifying identity. As a consequence, researchers mainly characterize themselves either through their area of application, e.g., as human ecologists, environmental scientists, or management specialists, or through a key approach or method, such as action researcher or system dynamics specialist.
Identity as a specialist in Integration and Implementation Sciences will complement, rather than replace, these existing identities. The difference that a specialization will make is that specialists in Integration and Implementation Sciences will be able to identify with a broader cadre of researchers and develop better-rounded skill sets. For example, although there is considerable overlap in their modes of operation between researchers using soft systems methods and action researchers, there is little crossover between these groups in terms of university coursework, professional associations, or even research collaboration. Soft systems researchers often have very polished systems methods, but underdeveloped participatory skills, with the opposite holding true for action researchers. Bringing these two groups together under a unifying umbrella will increase the chances that both will bring a more highly developed set of theory and methods to bear on the problems they deal with.
A specialization will also provide a one-stop shop for researchers newly seeking access to integration and implementation skills. This will meet a growing demand, as appreciation of the need for these skills grows. Where new researchers gain a foothold currently tends to be arbitrary, as it is extremely difficult to obtain a comprehensive overview of the Integration and Implementation Sciences field, existing knowledge, and key players. Thus, researchers new to the area often spend considerable time searching for resources and key contacts, and their early work often involves significant duplication of existing knowledge.
The same holds for policy makers and other practitioners seeking to link with researchers having Integration and Implementation Sciences skills. There is nowhere for such practitioners to go to receive an overview of what Integration and Implementation Sciences can offer, and to match needs with available approaches. If practitioners contact universities or other public good research organizations, the aspect of Integration and Implementation Sciences they link with, and whether they indeed manage to link with any form of Integration and Implementation Sciences, is largely a matter of chance. Outside universities, there are now a large number of commercial, consultant-based packages available, but most are limited in the approaches they offer and there are no mechanisms for quality control.
The latter is not intended as a criticism of consultants practicing approaches that are part of Integration and Implementation Sciences. Indeed, they have largely been responsible for the development of this field. Many have left universities to set up their own businesses because this has given them more freedom to undertake the practice-based research they care about. Furthermore, researchers who survive in universities and other research organizations are often required to be wholly or partially self-funded, often through consultancy work. Commercially based researchers are not in a position to develop colleges of critical peers, overarching associations, robust and comprehensive theoretical and methodological bases, or curricula for undergraduate and postgraduate education—in other words to develop a specialization. That is the role of universities. Thus, the development of a specialization will also provide a solid underpinning for commercial consultancy practice, a place where consultants can learn new skills or update existing skills, and where they can contribute lessons from their practice-based experience to invigorate and move forward the development of theory and methods. Given that consultants rely on the methods and other intellectual property they develop to make their living, incorporating these into the academy will also be a challenge.
Statistics as a Useful Analogy
So far, I have dealt with the importance of a home base for Integration and Implementation Sciences. Here, I will expand on this idea, using analogies between statistics and Integration and Implementation Sciences.
Statistics is embedded in the academy at three levels. First, there are home-base departments where theory and methods of statistics are developed and advanced. Second, other significant academic departments incorporate statistical training into their core curriculum and have at least some staff with a strong statistical bent. For example, disciplines such as biology, psychology, sociology, and geography provide core training in statistics, particularly as relevant to the discipline, and have staff and research programs with a strong quantitative orientation. In addition, multidisciplinary departments, such as public health, often employ statisticians who are willing to work on public health problems. Third, there is an expectation that a large proportion of staff and students throughout the academy will have a basic level of statistical competence.
Like statistics, some elements of Integration and Implementation Sciences are already embedded in other significant academic areas. For example, many departments and centers dealing with environmental issues incorporate integrated assessment, other systems approaches, and participatory approaches in their teaching and research. Public health departments often have a strong orientation to participation and implementation. However, the incorporation of Integration and Implementation Sciences is largely idiosyncratic, and there is generally little interaction between departments with different content area expertise about core or best methods. Some approaches that are key elements of Integration and Implementation Sciences have become standard in some established academic areas. For example, most law schools now include principled negotiation (alternative dispute resolution) in their teaching, if not research.
As I have already pointed out, unlike statistics, Integration and Implementation Sciences has no home base or shared understanding of what this area encompasses. There is also not the same level of individual competence among researchers in Integration and Implementation Sciences as there is in statistics. Although many staff and students throughout the academy have basic competencies, such as building trust, thinking laterally, and seeing interconnections (and some have very advanced competencies), these tend to be seen as personal attributes rather than academic skills. Furthermore, staff and students tend to be left to their own devices in the development of these competencies.
Certainly, the building blocks for a solid home base for Integration and Implementation Sciences exist and establishing home-base departments would have positive spin-offs for established disciplines and specializations and for individual staff and students.
The field of statistics provides another useful analogy, namely the comfortable coexistence of diversity in statistics, where some statisticians are trained predominantly in statistics and work on a variety of problems, whereas others have training in statistics and another discipline and work largely in a particular area, e.g., health. It is easily conceivable that some of those trained in Integration and Implementation Sciences would work on a wide range of problems, whereas others would work in more depth in areas such as environmental sciences or security.
The relationship between Integration and Implementation Sciences and traditional disciplines might be somewhat different, however, from the relationship of statistics and other traditional disciplines. Those trained in Integration and Implementation Sciences plus a traditional discipline might be expected to focus particularly on bringing that disciplinary perspective to the understanding of a complex problem rather than (or in addition to) advancing the discipline. Certainly, a key task of Integration and Implementation Sciences will be to harness and build on disciplinary strengths. The disciplines have developed and continue to elaborate a wealth of theoretical, methodological, and content knowledge. Furthermore, the disciplines themselves recognize the importance of building effective ways to draw together their individual strengths. There may be a case here for reinvigorating multidisciplinary as well as interdisciplinary and transdisciplinary approaches, with a particular focus on different methods for integrating diverse discipline-based knowledge and methods.
Statistics does not, however, provide a complete analogy. Statistics is obviously a well-developed and defined academic area. There are a range of widely adopted standard techniques, and an array of known challenges that stimulate ongoing research. Integration and Implementation Sciences is poorly defined, with no widespread agreement about what the field does and does not encompass. As outlined above, some methods, such as principled negotiation, are relatively well defined and accepted, but others are idiosyncratically developed and applied. Even without a clear framework, however, the scope of Integration and Implementation Sciences is likely to be considerably broader than that of statistics. Furthermore, it seems unlikely that one core concept will lie at the heart of Integration and Implementation Sciences, in the same way that probability forms the nucleus for statistics. This is where the real developmental challenges for Integration and Implementation Sciences lie.
Challenges to Developing a Specialization
There are a number of key challenges in developing a specialization of Integration and Implementation Sciences, including:
For the specialization of Integration and Implementation Sciences to reach its potential, considerable developmental work is required, and many of the outstanding challenges have been presented earlier. The challenges are both intellectual and practical, and essentially fall into three areas:
An established academic specialization can offer: a more clearly defined scope for Integration and Implementation Sciences, and complementarities with existing disciplines and specializations; a more robust theoretical base which will be a wellspring of innovation; and a large and critical “college” of peers to evaluate current and future research and practice.
These allow for both the cross-fertilization of ideas and advancement of knowledge, as well as opportunities for quality control. Care must be taken to ensure that the specialization does not become too narrowly defined and lose its richness, and that it does not develop in a lopsided way, for example, with mathematical modeling taking precedence over participatory techniques.
Developing the specialization includes:
Integration and Implementation Sciences are critical if we are serious about integration, policy relevance, evidence-based practice, and innovation, which are key concepts now driving research. The challenges are substantial, but the critical mass of researchers and approaches means that rapid development is possible. This promises intellectual excitement and fulfillment, as well as effective practical outcomes in tackling the complex social, environmental, and technological issues we confront.
A network has been established at www.anu.edu.au/iisn—we invite you to join us!
Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link
ACKNOWLEDGMENTSLorrae van Kerkhoff made a number of astute observations that have shaped the text in various places, and designed Figure 1. Caryn Anderson alerted me to the importance of the fields of knowledge management and information sciences. Useful comments were also received from Yoland Wadsworth, Susan Goff, Lesley Treleaven, and Steve Dovers, as well as the anonymous reviewers. Louis Lebel introduced me to the idea of hybrid epistemologies. Michael Smithson expanded my thinking about uncertainty, ignorance, and risk. Olivia Harkin provided valuable assistance with the formatting. As an inaugural Fulbright New Century Scholar, I had the opportunity to reflect on and develop Integration and Implementation Sciences in relation to improving global health. Colleagues at NCEPH, the Hauser Center, and Harvard’s Center for International Development have provided stimulating discussions and feedback. The development of these ideas has also been facilitated by the following: a National Health and Medical Research Council Population Health Research Capacity Building Grant “Environment and population health: research development from local to global,” a Colonial Foundation Trust grant for the “Drug Policy Modelling Project,” and the U.S. National Cancer Institute-funded project “Initiative on the Study and Implementation of Systems.”
American Society for Information Science and Technology. 2004. (Online.) URL: http://www.asis.org/.
Australian Government Cooperative Research Centres Programme. 2004. (Online.) URL: www.crc.gov.au.
Batliwala, S. 2003. Bridging divides for social change: practice-research interactions in South Asia. Organization 10:595–615. (Also see URL: www.sparcindia.org.)
Board on Sustainable Development, Policy Division, National Research Council. 1999. Our common journey: a transition toward sustainability, National Academy Press, Washington, D.C., USA. (Online.) URL: http://www.nap.edu/books/0309067839/html/.
Bok, D. 2003. Universities in the marketplace. The commercialization of higher education. Princeton University Press, Princeton, New Jersey, USA and Oxford, UK.
Brown, L. D., editor. 2001. Practice-research engagement for civil society in a globalizing world. CIVICUS and Hauser Center for Nonprofit Organizations, Washington, D.C., USA.
Brown, L. D., G. Bammer, S. Batliwala, and F. Kunreuther. 2003. Framing practice-research engagement for democratizing knowledge. Action Research 1:81–102.
Campbell Collaboration. 2004. The Cambell Collaboration homepage. (Online.) URL: http://www.campbellcollaboration.org/.
Cash, D. W., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, D. H. Guston, J. Jaeger, and R. B. Mitchell. 2003. Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences 100:8086–8091.
Chapman, S. 2001. Advocacy in public health: roles and challenges. International Journal of Epidemiology 30:1226–1232.
Checkland, P. 1984. Systems thinking, systems practice. John Wiley, Chichester, UK.
Clark, W. C., and N. M. Dickson. 2003. Sustainability science: the emerging research program Proceedings of the National Academy of Science. 100:8059–8061
Coady, T., editor. 2000. Why universities matter. A conversation about values, means and directions. Allen and Unwin, St Leonards, New South Wales, Australia.
Cochrane Collaboration. 2004. The Cochrane Collaboration homepage. (Online.) URL: http://www.cochrane.org/.
Costanza, R. 2003. A vision of the future of science: reintegrating the study of humans and the rest of nature. Futures 35:651–671.
CSIRO Australia. 2002. Science Investment Focus Group on Complex Systems Science. Draft report. [Online.] URL: http://www.dar.csiro.au/css/documents/fin057_0202_sifg.doc.
Dance, P., R. Brown, G. Bammer, and B. Sibthorpe. 2004. Aged care services for indigenous people in the Australian Capital Territory and surrounds: analysing needs and implementing change. Australian and New Zealand Journal of Public Health 28:579–583.
Emery, M. 1999. Searching. The theory and practice of making cultural change. Dialogues on work and innovation. Volume 4. John Benjamins Publishing Co., Amsterdam, the Netherlands.
European Commission. 2002. The Sixth Framework Program in brief. [Online.] URL: http://europa.eu.int/comm/research/fp6/pdf/fp6-in-brief_en.pdf.
Fisher, R., W. Ury, and B. Patton. 1991. Getting to yes. Negotiating an agreement without giving in. Random House Business Books, London, UK.
Flood, R. L., and M. C. Jackson. 1991. Creative problem solving: total systems intervention. John Wiley, Chichester, UK.
Funtowicz, S. O., and J. R. Ravetz. 1993. Science for the post-normal age. Futures 25:739–755.
Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P. Cott, and M. Trow. 1994. The new production of knowledge. The dynamics of science and research in contemporary societies. Sage Publications, Thousand Oaks, California, USA.
Gibson, B. 2003 From transfer to transformation: rethinking the relationship between research and policy. Dissertation, National Centre for Epidemiology and Population Health, The Australian National University. [Online.} URL: http://thesis.anu.edu.au/public/adt-ANU20040528.165124/index.html.
Higginbotham, N., G. Albrecht, and L. Connor. 2001. Health social science. A transdisciplinary and complexity perspective. Oxford University Press, Melbourne, Australia.
Hough, L. 2002 A meeting of the minds. What happens when the Kennedy School’s executive sessions unite practitioners and academics together? Harvard University John F. Kennedy School of Government Bulletin, Spring:32–37. (Online.) URL: http://www.ksg.harvard.edu/ksgpress/bulletin/spring2002/features/meeting.html.
Intergovernmental Panel on Climate Change. 2004 http://www.ipcc.ch. Accessed 6 December 2004.
Journal of Information Science. 2004. (Online.) URL: http://jis.sagepub.com/.
Klein, J. T. 1990. Interdisciplinarity: history, theory and practice. Wayne State University Press, Detroit, Illinois, USA.
Koehn, P. H., and J. N. Rosenau. 2002. Transnational competence in an emergent epoch. International Studies Perspectives 3:105–127.
Kuhn, T. S. 1970. The structure of scientific revolutions. Second edition. University of Chicago Press, Chicago, Illinois, USA.
Lamb, S., M. R. Greenlick, D. McCarty, and the Committee on Community-based Drug Treatment, Division of Neuroscience and Behavioral Health, Institute of Medicine, editors. 1998. Bridging the gap between practice and research. Forging partnerships with community-based drug and alcohol treatment. National Academy Press, Washington, D.C., USA.
Land and Water Australia. 2001. Strategic R&D plan 2001–2006. Land and Water Australia, Canberra, Australia. (Online.) URL: http://www.lwa.gov.au/downloads/publications_pdf/PR010102.pdf.
Land and Water Australia. 2005. Integration Symposium. Land and Water Australia, Canberra, Australia. CD-ROM.
Lindblom, C. E. 1990. Inquiry and change. The troubled attempt to understand and shape society. Yale University Press and Russell Sage Foundation, New Haven, Connecticut, USA.
Linstone, H. A., and M. Turoff, editors. 1975. The Delphi method. Techniques and applications. Addison-Wesley Publishing Company, Reading, Massachusetts, USA.
Martens, P., R. S. Kovats, S. Nijhof, P. De Vries, M. T. J. Livermore, D. J Bradley, J. Cox, and A. J. McMichael. 1999. Climate change and future populations at risk of malaria. Global Environmental Change S9:89–107.
Midgley, G. 2000. Systemic intervention: philosophy, methodology, and practice. Kluwer Academic/Plenum Publishers, New York, New York, USA.
Mingers, J. 2000. The contribution of critical realism as an underpinning philosophy for OR/MS and systems. Journal of the Operational Research Society 51:1256–1270.
Moore, M. H. 1995. Learning while doing: linking knowledge to policy in the development of community policing and violence prevention in the United States. Pages 301–331 in P.-O. Wikstrom and R. V. Clarke, editors. Integrating crime prevention strategies: propensity and opportunity. Swedish National Council for Crime Prevention, Stockholm, Sweden.
Neuman, W. L. 2003. Social research methods. Qualitative and quantitative approaches. Fifth Edition. Allyn and Bacon, Boston, Massachussetts, USA.
Nowotny, H., P. Scott, and M. Gibbons. 2001. Rethinking science. Knowledge and the public in an age of uncertainty. Polity Press in association with Blackwell Publishers, Cambridge, UK.
Organisation for Economic Co-operation and Development (OECD). 1996. The knowledge-based economy. OECD, Paris, France.
Ravetz, J. R. 1996 Scientific knowledge and its social problems. Transaction Publishers, New Brunswick, New Jersey, USA.
Reason, P., and H. Bradbury, editors. 2001. Handbook of action research: participative inquiry and practice. Sage Publications, Thousand Oaks, California, USA.
Reich, M. R., editor. 2002. Public-private partnerships for public health. Harvard Center for Population and Development Studies, Cambridge, Massachussetts, USA.
Reich M. R., and V. M. Nantulya. 2002. Road traffic injuries and health equity. Harvard Center for Population and Development Studies, Cambridge, Massachussetts, USA. (Online.) URL: http://www.hsph.harvard.edu/traffic/.
Rogers, E. M. 1995. Diffusion of innovations. Free Press (Division of Simon and Schuster), New York, New York, USA.
Senge, P. M. 1990. The fifth discipline. The art and practice of the learning organization. Century Business, London, UK.
Susskind, L., S. McKearnan, and J. Thomas-Larmer, editors. 1999. The consensus building handbook. A comprehensive guide to reaching agreement. Sage Publications, Thousand Oaks, California, USA.
Trochim, W. 1989. An introduction to concept mapping for planning and evaluation. Pages 1–12 in W. Trochim, editor. Evaluation and Program Planning Special issue 12. (Online.) URL: http://www.conceptsystems.com/papers/intro_article.cfm.
Troncale, L. 2000. A comprehensive introduction to the systems sciences. Workshop handout, World Congress of the Systems Sciences, Toronto, Ontario, Canada.
United Nations Conference on Environment and Development (UNCED). 1992. Agenda 21. [Online.] URL: http://www.un.org/esa/sustdev/agenda21text.htm.
United Nations Educational, Scientific and Cultural Organisation (UNESCO). 1999. Introductory note to the Science Agenda—a framework for action. Page 469 in UNESCO World Conference on Science. Science for the 21st Century: a new commitment. UNESCO, Budapest, Hungary.
U. S. National Science Foundation. 2001. NSF GPRA strategic plan FY 2001–2006. [Online.] URL http://www.nsf.gov/pubs/2001/nsf0104/nsf0104.doc.
van Kerkhoff, L. 2002. Making a difference: science, action and integrated environmental research. Unpublished dissertation. School of Resources, Environment and Society, the Australian National University, Canberra, Australia.
Williams, P. 2002. The competent boundary spanner. Public Administration 80(1):103–124.
Wilson, E. O. 1998. Consilience. The unity of knowledge. Abacus (Little, Brown and Company), London, UK.
World Commission on Dams. 2000. Dams and development: a new framework for decision-making. http://www.dams.org/.
World Health Organization. 2002. Reducing risks, Promoting healthy life. World Health Report 2002. World Health Organization, Geneva, Switzerland. [Online.] URL: http://www.who.int/whr.
World Health Organization. 2004. World Report on Knowledge for Better Health. Strengthening health systems. World Health Organization, Geneva, Switzerland. (Online.) URL: http://www.who.int/rpc/meetings/pub1/en/.
Yankelovich, D. 1999. The magic of dialogue. Transforming conflict into cooperation. Simon and Schuster, New York, New York, USA.
|Home | Archives | About | Login | Submissions | Notify | Contact | Search|