Strategic Framework for Evaluation that is Useful and Utilized

Health and the Information Highway Division, Health Canada
July 2003

Table of Contents

• Introduction
• Methodology
• Findings
• Strategic Framework
• Conclusion
• References

Introduction

There is growing interest in the computerization of the health care system. Over the last ten years, a number of projects introducing New Information and Communications Technologies (NICT) have been conducted. The innovative nature of these projects, in an era of transformation in the organization of health care and of the rapid and constant change in the technologies used, underscores the importance of an integrated evaluation of such projects. Since these projects involve different types of decision-makers (local, regional, and central) as well as a variety of public and private partners, evaluation needs to be both useful and utilized if it is to support decision-making.

Although the importance of an integrated evaluation of New Information and Communications Technologies (NICT) projects has been acknowledged [1], very little has been written about project evaluation that is useful and used in supporting decision-making. Consequently, the purpose of this study is to help develop strategies for practical evaluation that is actually used by project leaders and other decision-makers who play a part in introducing and disseminating new information and communications technologies in the health sector.

This study includes two phases. The objectives of the first phase, which is the subject of this report, are as follows:

• To identify and summarize appropriate strategies and mechanisms to provide evaluation that supports decision-making at various stages of NICT projects in the health sector.
• To develop a relevant strategic framework to guide evaluation activities in the development, emergence, implementation, and diffusion phases of innovative NICT projects.
• To validate the strategic framework through consultations with experts.

The purpose of the second phase is to:

• Apply and validate the strategic framework developed in NICT project evaluation strategies.
• Assess the relevance of the proposed strategies and their effects.
• Develop a strategic evaluation guide.

Before describing the methodology, it is important to define some of the concepts used in the study:

Evaluation: Evaluation is more an act of judgement than a technical act. It is also an act of political regulation in which the production of knowledge helps to support decision-making processes, and an act of social practice, allowing for debate and accountability [2]. As such, it constitutes one of the research methods used to influence decisions about social and health problems [3]. For the purposes of this project, evaluation helps to initiate, implement, and disseminate the diffusion of innovations where NICT are concerned.

Decision-making in health: Decision-making in health is influenced by crises, current issues, and the interests of various groups and organizations [4].

Innovation: An innovation is an idea, practice or object that is perceived as new by an individual or other unit of adoption [5]. Social innovation is innovation that enables better adaptation of services and stimulates the development of practice environments. It may, but does not always, stem from research. Its success requires the presence of many other conditions [6].

Diffusion of innovations: Process by which new ideas and products spread from the source to the end user [5].

Knowledge transfer: The transfer of knowledge from research findings is a communication activity designed to bridge the gap between researchers and decision-makers in the health sector [7].

Knowledge sharing: This concept means that every player in the decision-making process has knowledge, expertise or experiences that can be pooled. Sharing knowledge is essential to the translation of research findings into data, products, and procedures to improve human health [8].

Knowledge application: The ethical exchange, summarization, and application of knowledge in a complex system of relationships between researchers and users [3].

Methodology

To meet the aforementioned objectives, a multiple case study [9] was selected as the methodological approach. Different evaluations of information and communications technology projects in Quebec were analysed. A total of four projects were selected for this purpose: 1) the Rimouski health card project [10], 2) the Eastern Quebec telecardiology and teleradiology project (RTEQ) [11], 3) the Oncology Network Demonstration Project (DRIOQ) [12], and 4) Telemedicine for Remote Regions of Quebec: A Pilot Project In the Magdalen Islands [13]. These projects looked at the three different types of information and communications technologies (the health card, computerized patient records, and telemedicine) and they were led and / or evaluated by a single research team who developed and adapted evaluation strategies, methods, and activities to take the characteristics of each project into account. The projects were chosen for their interest to decision-makers and for their potential contribution to different phases of innovation projects, particularly in terms of knowledge sharing. The study was conducted using various methods, including project evaluation reports and interviews with key informants.

The analysis first looked at the nature and phases of the projects, stakeholders, perceived issues, methodological and theoretical approaches, strategies, data collection activities, and information sources used in the project evaluations. This work of comparing and summarizing the evaluation mechanisms helped to create a reference framework to ensure that evaluation results were integrated into the decisions made at various stages of the projects. This reference framework was validated by experts at various stages of its development through presentations to the Département informatique médicale du CHU de Lille II (Dim) et Centre de recherche informatique médicale (Cerim) Faculté de médecine de Lille on September 26, 2002; the Third Telehealth Symposium in Quebec City on October 17, 2002; the American Evaluation Association (AEA) in Washington on November 9, 2002; the Quebec public health research unit XDO team on March 5, 2003; and the Fourth Annual Telehealth Summer Institute in Calgary in June, 2003 (see Appendix A).

Findings

Nature of the Projects

The four projects shared some common characteristics, which were taken into account in the evaluations. They were all innovative, unique, complex, open-ended, and had an unforeseen evolution. They were all pilot projects with dissemination potential. All four projects were conducted in an environment marked by major socio-political dynamics between stakeholders in the Quebec health care system and a revolution in medical practices and health organizations.

Stages in the Projects Chosen for Evaluation

All the evaluations looked at the emergence and implementation stages, as well as the sustainability and dissemination potential of the selected projects (Appendix C). Findings (study on use and effects) were also taken into consideration in three of the four projects; that is to say, in all except the Oncology Network Demonstration Project (DRIOQ), which was confined to the study of emergence, development, and implementation.

Stakeholders

Monnier's nomenclature [14] was used to categorize the stakeholders in the different projects. The three environments described by Monnier [14] are action, reaction, and legitimization. The action environment includes stakeholders who conduct the operations required for healthy development of innovation and for its implementation. Stakeholders in the reaction environment are the recipients of the action--opinion leaders, the media, academics, etc. Finally, the legitimization environment includes decision-makers whose role is to endorse the emerging compromise and to authorize action. Evaluation was added as an environment to Monnier's nomenclature [14], given that stakeholders in the evaluation played both an important role in a project and a role that is different from that of stakeholders' in other environments. The results of the analysis conducted using this nomenclature appear in schematic form in Appendix D.

Monnier's adapted nomenclature [14] revealed that the project team may be operating in the action environment and the evaluation environment at the same time, depending on whether or not the members of the evaluation team are also on the project team. For instance, the Rimouski health card project and the Oncology Network Demonstration Project both had members of the evaluation team on their project teams. Consequently, project team members found themselves working in an environment that straddled both the action and evaluation environments. This overlapping meant that more experts were affiliated with one another and used complementary strategies, thereby ensuring the scientific quality and credibility of the evaluation.

Another distinctive feature that also emerged from the stakeholder analysis is that three of the four projects have an advisory committee operating in all four of the environments of the adapted model. In addition, in all four projects there were champions with roles in both the action and reaction environments.

Issues

The four evaluations looked at a number and a range of issues in order to represent, as faithfully as possible, the complexity of the projects. These were professional, ethical, legal, economic, organizational, technological, and socio-political issues which helped provide a clearer understanding of how the projects evolved as well as helping to meet the needs of decision-makers.

Methodological Approaches

The selected project evaluations were all treated as case studies [15, 16], as they were difficult to consider in isolation from their context. The only exception was the evaluation of the Eastern Quebec telecardiology and teleradiology project, which was actually more of a multiple case study. Given the nature of the projects, their complexity, and the vast quantity of information present, the evaluations integrated both qualitative and quantitative data. Finally, inductive reasoning was the approach taken to account for users' needs in all scenarios.

Theoretical Approaches

In the theoretical approaches, there are certain constants from project to project, but there are also variations in the frameworks used by evaluators. One unavoidable theoretical approach - because it is connected to the very nature of projects - was the 'clinical decision-making process' described by Eisenberg [17]. This approach contains three major phases: 1) access to the information, based on a more or less systematic collection of clinical data; 2) processing of the information, which calls for the exercising of professional judgement; and finally, 3) making the actual decision. Each of the phases is influenced by the patient's characteristics, by professional characteristics, and by the organizational characteristics of the practice environment. This use of this approach allows evaluators to estimate the "value-added" of the introduction of technological innovation in practice environments. Another theoretical framework, that of Donabedian [18], was used to in order to take into account the characteristics of health services organizations.

With regards to the dynamics of innovation, the theories used most often in the selected evaluations are in the tradition of work done by Rogers [5] on the adoption of technological innovations. They aim to elucidate a series of factors or conditions likely to have an impact on the adoption and use of technology, (potentially) in terms of expected outcomes and effects [10].

According to Rogers [5], the adoption of an innovation is influenced by five factors: 1) period of trial or experimentation, 2) relative advantage, 3) compatibility, 4) complexity, and 5) transparency. The period of trial allows the potential user to try out the innovation before deciding whether or not to adopt it. Relative advantage refers to the degree to which an innovation is perceived as better than the idea it supersedes. Compatibility refers to the quality of the innovation and how it is consistent with the values, culture, and practices of the user. Complexity corresponds to the degree to which an innovation is perceived as difficult to understand and use. Finally, transparency refers to the degree to which the innovation is physically visible and the results are easy to visualize, for both the user and all of his / her peers or community [10]. This theoretical approach enables evaluators to bring out not only the influence of the technical attributes of the innovation, which foster its adoption, but also, and especially, the perceptions of the innovation's adopters.

Rogers [5] is not the only author whose work has been used in these evaluations to explain the conditions for the adoption of an innovation [19]. On an individual level, the factors that induce adoption / resistance behaviour are identified on the basis of subjective intentions, attitudes, and standards [20]. On a group level, the political interaction model [21] and the system of professions [22] enable evaluators to identify mechanisms of adoption and resistance, and provide invaluable information about the results of effecting change. These theoretical approaches are also used for complementary studies on telemedicine [23]. Ajzen's theory of planned behaviour [24, 25] was also used to highlight the salient modal beliefs of health professionals regarding the use of telehealth in their practice, thus bringing into focus the main individual beliefs that are generally agreed upon in a given social group.

We noted the use of a socio-political approach, namely the theory of coalitions. Many researchers have described this theory, namely Gamson [26], Hinckley [27], Lemieux [28], and Fortin et al. [29]. The theory of coalitions presents different parameters such as assets, advantages, links, mode of decision-making, and mode of organization. Assets represent the initial distribution of means and resources among members of the coalition. The advantages indicate the share of benefits a member can hope to draw from each of the coalitions. Links are strategic preferences of a non-utilitarian nature between participants adhering to their mutual inclinations. The decision-making mode signifies the sum of all the methods required to cross the threshold to effective control of the decision. Finally, the organization mode refers to the rules of the game affecting the situation and behaviour as well as the conditions under which the coalition has operated.

Literature on the theory of coalitions was used in these evaluations right from the outset to formalize the coalitions. This theory provides an interesting conceptual basis for an understanding of the formation and longevity of coalitions. NICT innovations are tied in with the initiative of individuals who have professional ties and who have come together to work. On the basis of these stakeholders' coalitions, the challenge of the evaluations was to factor in the diverse interests associated with the innovation, at both the local and provincial level.

Publications from the field of administrative sciences and, more specifically, the field of management, were also used. The information selected was that which pertains to work done on change management [30-32]. The great complexity surrounding the definition of the object of change (in this case, innovation) and its implementation process call for a type of management that lifts many barriers. It is therefore important to study the strategies used, in order to manage change appropriately.

In the Eastern Quebec telediagnosis project, a different approach from that of Rogers', in terms of the conditions of adoption of an innovation, was also used; namely, that of enabling and restricting factors proposed by Green et al. [33]. This approach allows for the evaluation to make nuances and for improvements upon the factors or conditions established in the Rogers theory.

Finally, the Magdalen Islands project evaluation also drew on the experience gained through the RTEQ project evaluation [11]. It helped to identify more clearly and understand the strategic choices made by stakeholders involved in the emergence and implementation of telediagnostics; some of the major issues raised by the introduction of telemedicine in our health system; the impact of associated medical and professional dynamics; related costs and benefits; and the project's limitations and diffusion potential [13].

Strategies

All project evaluations applied strategies from different schools of thought in the field of evaluation, notably those of Patton, Guba and Rossi. Of particular note are the comprehensive, constructivist, pluralistic, and participatory approaches. The comprehensive strategy consists of describing and understanding the relative facts and issues related to the development, implementation, and progress of the project [34, 35]. The pluralistic strategy [14, 36] was also used in all the evaluations as a way of involving stakeholder groups in the evaluation process and gaining a better understanding of the variations between them.

The constructivist strategy [14, 35] was common to all project evaluations. This approach is based primarily on the observation, description, and understanding of phenomena. Its relevance stems from the actual nature of the projects themselves, where processes are of crucial importance and evolution is unforeseen.

A participatory approach was also used at every stage of all four projects to define, in cooperation with stakeholders, the needs, challenges, terms of use, evaluation issues, and criteria. This strategy helps to instil trust between the evaluation team and the stakeholders involved in the project. In so doing, it contributes to the actual development as well as to the evaluation of the project.

The project evaluations also used partnership strategies including, more specifically, the seeking of champions. This strategy helps to actively involve practitioners, viewed as leaders in their field by colleagues, so as to promote adoption of the project in targeted environments. Another partnership strategy is to seek support from management in the institutions concerned, in order to secure agreements with the appropriate departments or services.

Field's results-based approach [35] was used to evaluate the RTEQ project and the telehealth project in the Magdalen Islands. The objective of this approach is to compare to the previous situation results associated with the introduction of the innovation in terms of the advantages, risks, and associated costs. Both these evaluations based their strategy on other evaluations such as those of the Joint working group on telemedicine [38] and on some of the work conducted by Fortin et al. [10, 11, 29, 39].

It should be added that the strategy for the Magdalen Islands project evaluation was to describe the cost effectiveness of innovative projects [38]. This strategy allows for focusing on the way in which the costs and benefits of telemedicine are distributed among the institutions in the network.

Data Collection Activities and Sources of Information

All the project evaluations collected data through participation in the various project committees, field observations, and individual interviews with users and professionals. Literature reviews and summaries were carried out for both the telemedicine projects and DRIOQ. Questionnaires were used in the health card projects and in both telemedicine projects. Transmission slips were analysed in the case of the telemedicine project evaluations. Finally, discussion groups were organized for the health card project and the DRIOQ evaluations.

Essentially, all four project evaluations used strategies, approaches, and tools with formative and summative finalities. The methodology seems to indicate that their ultimate objective was knowledge sharing. In addition to sharing knowledge within the projects, these evaluations used knowledge dissemination strategies, such as writing reports and articles and participation in expert committees and conferences.

Strategic Framework

A strategic framework was designed in order to summarize and illustrate the findings of the analysis (see Figure 1).

Figure 1: Strategic Framework for Evaluation that is Useful and Utilized

The key elements in this framework (the stakeholders and the issues) are to be considered through the three components in the centre, i.e., theoretical approaches, strategies, and methodological and data collection approaches. The evolution of all the elements listed above depends on the stage at which projects are examined. As a result, stakeholders and issues at different levels of intervention (local, regional, and central) will be studied at all the various stages, from emergence to project diffusion. The sharing and transfer of knowledge and lessons learned are the outcome of all the dynamics at work between various components during project evaluation. This outcome is a determining factor for the future use of evaluation and its application to decision-making. Finally, the environment, which includes both the nature of the projects and the political context in which they are conducted, will affect all components of the strategic framework.

Conclusion

The work conducted thus far has helped to develop a strategic framework, which has been well received by several evaluation experts and project leaders and, particularly, to pave the way for new methods of evaluating new information and communications technologies in the health sector. The advantage of this framework is that it does not restrict the evaluation to only one method, but rather it describes a series of components that must be taken into consideration in order to promote use of the evaluation and its application in decision-making. The following steps are required in order to improve the strategic framework presented here: 1) addition of detailed descriptions of the activities that correspond to the framework components; 2) validation, through interviews, with decision-makers in the identified projects; and 3) application of the framework to new experimentation. In fact, the essential parts of the framework are currently being validated in the future CLSC project, which is funded under the Canada Health Infostructure Partnerships Program (CHIPP).

References

1. Table ministérielle en télésanté, Telehealth in Quebec: vision and orientations. 2001, Quebec Ministry of Health and Social Services: Quebec.
2. Bégin, C., P. Joubert, and J. Turgeon, L'Évaluation dans le domaine de la santé : conceptions, courants de pensée et mise en oeuvre, in Le Système de santé québécois: Un modèle en transformation, C. Bégin, et al., Editors. 1999, Les Presses de l'Université de Montréal: Montreal. p. 439.
3. Canadian Health Services Research Foundation, Knowledge Transfer: Looking Beyond Health. 2000.
4. Division of International Health - Epidemiology Program Office, Data for decision making. 2003, Center for Disease Control and Prevention.
5. Rogers, E. M., Diffusion of Innovations. 4th Edition ed. 1995, New York: The Free Press.
6. Duchesne, L., et al., Quebec Policy on Science and Innovation-Knowledge to Change the World. 2001, Quebec Ministry of Research, Science and Technology: Quebec. p. 169.
7. Health Research Transfer Network of Alberta, What is Research Transfer? 2001, Health Research Transfer Network of Alberta.
8. National Institute of Health, NIH Guide: Final NIH statement on sharing research data. February 2003.
9. Yin, R. K., Enhancing the Quality of Case Studies in Health Services Research. HSR: Health Services Research, 1999. 34(5): p. 1209-1224.
10. Fortin, J.-P., et al., Évaluation du projet québécois d'expérimentation de la carte santé à microprocesseur. 1996, Régie de l'assurance maladie du Québec: Quebec.
11. Fortin, J.-P. and C. Banville, Rapport d'évaluation des projets pilotes en télécardiologie et en téléradiologie. 1998, Centre francophone d'informatisation des organisations (CEFRIO): Quebec. p. 149.
12. Fortin, J.-P., et al., Oncology Network Demonstration Project (DRIOQ). 2001, Université Laval: Sainte-Foy.
13. Cloutier, A., et al., Telemedicine for Remote Regions of Quebec: A Pilot Project in the Magdalen Islands. 2001, Health Transition Fund, Health Canada: Sainte-Foy. p. 46.
14. Monnier, É. Évaluations de l'action des pouvoirs publics. 2^nd edition, revised and enlarged. 1992, Paris: Économica.
15. Yin, R. K., Case study research: design and methods. 1989, Newbury Park (CA): Sage Publications.
16. Stake, R. E., The Art of Case Study Research. 1995, Thousand Oaks: Sage Publication.
17. Eisenberg, J. M., Doctor's decisions and the cost of medical care. 1986, Ann Arbor, Michigan: Health Administration Press Perspectives.
18. Donabedian, A., et al., Volume II : Medical Care Appraisal. A Guide to Medical Care Administration. Vol. 2. 1980: American Public Health Association. 221.
19. Aubert, B. B. and G. Hamel, L'évaluation du potentiel de diffusion de la Carte Santé. 1995, Université Laval: Quebec. p. 104.
20. Fishbein, M. and I. Azjen, Belief, attitude, intentions and behavior: an introduction to theory and research. 1975, Boston: Addison-Westley.
21. Markus, M. L. and D. Robey, Information Technology and Organizational Change: Causal Structure in Theory and Research. Management Science, 1988. 34(5): p. 583-598.
22. Abbott, A., The system of profession. 1988, Chicago: University of Chicago Press.
23. Gagnon, M.-P., G. Godin, and J.-P. Fortin. Déterminants de l'adoption de la télémédecine par les professionnels de la santé. Symposium 2001 - La télésanté : une solution accessible. 2001. Québec.
24. Ajzen, I., The theory of planned behavior. Organ Behav Hum Decis Process, 1991. 50: p. 179-211.
25. Ajzen, I., From intentions to actions : A theory of planned behavior, in Action-control : from cognition to behavior, J. Kulh and J. Beckman, Editors. 1985, Springer: Heildelberg. p. 11-39.
26. Gamson, W., A theory of coalition formation. American Sociological Review, 1961. 26: p. 373-382.
27. Hinckley, B., Coalitions and Politics. 1981, New York: Harcourt Brace Jovanovich.
28. Lemieux, V., Les politiques publiques et l'exercice du pouvoir. 1991, Groupe de recherche sur les interventions gouvernementales (GRIG): Ste-Foy.
29. Fortin, J.-P., et al., L'action intersectorielle en santé - Rapport synthèse. 1994, Université Laval, XDO, Quebec Public Health Branch: Quebec. p. 44.
30. Fortin, J.-P., L. Lamothe, and L. Lapointe, La mise en place d'un dossier réseau informatisé en oncologie: la technologie au service d'un réseau de services intégrés. Ruptures, Interdisciplinary Health Journal, 2001. 8(2): p. 21-34.
31. Lapointe, L., L'adaptation de systèmes d'information cliniques par les médecins et les infirmières: Une étude des variables individuelles, socio-politiques et organisationnelles. 1999, Université de Montréal: Montréal.
32. Denis, J.-L. and F. Champagne, Understanding Organizational Change. Gestion, 1990. 15(1): p. 44-56.
33. Green, L. W., M. P. Eriksen, and E. L. Schor, Preventive practices by physicians : behavioral determinants and potential interventions. American Journal of Preventive Medicine, 1988. Suppl 4(4): p. 101-111.
34. Rossi, P. H. and H. E. Freeman, Evaluation : A systematic approach (5e edition). 1993, Newbury, (CA): Sage Publications.
35. Guba, E. and Y. Lincoln, Effective evaluation : Improving the usefulness of evaluation results through responsive and naturalistic approaches. 1981, San Francisco: Jossey-Boss.
36. Patton, M. Q., Utilization-focused evaluation. The new Century Text (3e edition). 1997, Newbury Park (CA): Sage Publications.
37. Field, M. J., ed. Telemedicine : A Guide to Assessing Telecommunications in Health Care. ed. D.o.H.C.a.S. Institute of Medicine. 1996, National Academy Press: Washington.
38. Joint Working Group on Telemedicine, Telemedicine report to congress : Evaluation. 1997.
39. Fortin, J.-P., F. Labbé, and P. A. Lamarche, Rapport d'évaluation du projet pilote en télépsychiatrie. 1999, Université Laval, Department of Social and Preventive Medicine, Quebec Public Health Centre: Quebec.
40. Barriault, M. and M. Gosselin, Évaluation des coûts des projets pilotes en télécardiologie et en téléradiologie. 1997, Faculty of Administrative Sciences - Department of Accounting Sciences, Université Laval, under the direction of CEFRIO: Quebec. p. 137.