Toward an Evaluation Framework for Electronic Health Records Initiatives

Health and the Information Highway Division, Health Canada 2003

Table of Contents

• Toward an Evaluation Framework for Electronic Health Records Initiatives: A Proposal For an Evaluation Framework
  • Background
  • Process Used to Develop The Framework
  • Nature of The Intervention Under Study
  • Synopsis of the Literature
    • Perspectives on Evaluation
    • Models and Frameworks Commonly Used to Guide Evaluation Projects
    • Key Messages Regarding Future Efforts To Evaluate Complex Health Information Systems
  • Proposed Approach to Planning an Evaluation Of An E.H.R. Initiative
  • Steps For Framework Development
    • Step 1: Identify Key Stakeholders in Each Jurisdiction
    • Step 2: Orient Key Stakeholders To the E.H.R. Initiative and Reach Agreement on Why an Evaluation Is Needed
    • Step 3: Agree on When To Evaluate
    • Step 4: Agree on What To Evaluate
    • Step 5: Agree on How To Evaluate
    • Step 6: Analyze and Report
    • Step 7: Agree on Recommendations and Forward Them to Key Stakeholders
  • Proposed Framework: Time Frames, Core Questions, Indicators, Data Sources and Study Design
    • Evaluation For Accountability: Measurement of Results
    • Evaluation For Performance Enhancement
    • Evaluation For Knowledge Development
  • Concluding Remarks
  • Bibliography
  • Appendices
    • Evaluation of Complex Health Information Systems: Sample Evaluation Questions
    • Evaluation Of Complex Health Information Systems Sample Indicators
• Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects
  • Executive Summary
  • Introduction
    • Background
    • Project Goals and Objectives
    • Research Team
    • Scope of Project
    • Target Audience
    • Participants
    • Deliverables
  • Inventory of Health Information Systems Across Canada
    • Methods
    • Key Findings
  • An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports Relevant to the Electronic Health Record
    • Methods
    • Key Findings
  • A Proposed Evaluation Framework for Electronic Health Records Initiatives Across Canada
    • Methods
    • Key Findings
  • Recommendations
  • Bibliography
• Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada
  • Introduction
    • Background
    • Process
  • Summary of Major Electronic Health Record (EHR) Initiatives across Canada
    • Unique Personal Identifier/Client Registry
    • Pharmacy Networks
    • Laboratory Networks
    • Diagnostic Imaging Networks
  • Overview of Major Initiatives
    • Newfoundland and Labrador
    • Nova Scotia
    • New Brunswick
    • Prince Edward Island
    • Quebec
    • Ontario
    • Manitoba
    • Commentaires
    • Saskatchewan
    • Alberta
    • British Columbia
    • Impact sur les soins au patient
    • Collaboration/intervenants
    • Yukon
    • North West Territories
    • Nunavut
  • Contact Persons for each Jurisdiction
  • Related Documents and Reports

Toward an Evaluation Framework for Electronic Health Records Initiatives: A Proposal For an Evaluation Framework

See also:

• Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects
• Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada

Prepared as a component of the study:

Towards an Evaluation Framework for Electronic Health Record Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects

Doreen Neville, ScD
Kayla Gates, B.Sc
Donald MacDonald, B.A.
Mike Barron, MBA
Shelia Tucker, B.A. (Conj). B.A. (Hons). B.Ed., MLIS, CPAD
Sandra Cotton, B.A.
Gerard Farrell, M.D.
Theodore Hoekman, PhD
Stephen Bornstein, PhD
Stephen O'Reilly, MBA

Background

An electronic health record (EHR) provides each individual with a secure and private lifetime record of their key health history and care within a health system. The record is available electronically to authorized health care providers and the individual anywhere, anytime, in support of high quality care. Recognizing the importance of the EHR in improving the quality and efficiency of health care, the federal government of Canada, in 2001, established Canada Health Infoway to support and accelerate the development and adoption of interoperable electronic health records solutions across the country. Four core components have been identified as the key building blocks of an EHR by Infoway and the Newfoundland and Labrador Centre for Health Information (NLCHI): (1) a unique personal identifier/client registry; (2) a pharmacy network; (3) a laboratory network; and (4) a diagnostic imaging network.

Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects, a project funded by Health Canada, Office of Health and the Information Highway, was carried out between May 2002 and December 2003. The goals of the project were to: (a) review current approaches to evaluating the impact of health information systems (particularly those leading to an EHR); and (b) develop an evaluation framework which addresses the information needs of key stakeholders and the identified best practices in the evaluation of such initiatives. Three deliverables were produced from the project and released as separate (but complementary) documents:

1. Towards an Evaluation Framework for Electronic Health Records: An Inventory of Health Electronic Health Records Initiatives Across Canada;
2. Towards an Evaluation Framework for Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports;
3. Towards an Evaluation Framework for Electronic Health Records: A Proposal for an Evaluation Framework.

This report presents the Proposal For An Evaluation Framework. The project was guided by an advisory committee comprised of key personnel who are leading the work of NLCHI around the development of EHRs, including the Chief Executive Officer , the Health Information Network Project Leader, the Director of Research and Development, the Director of Standards Development, the Director of Communications and Privacy, and the project's principal research investigator.

Process Used to Develop the Framework

The following process was used to develop the proposed evaluation framework.

1. A comprehensive search of the literature concerning the evaluation of complex health information systems, particularly those most closely related to the development of an Electronic Health Record, was conducted and used to generate a synthesis of the literature around evaluation efforts in this field and to outline a preliminary draft of an evaluation framework.
2. A summary of the current (as of May 2003) Electronic Health Records Initiatives across Canada was produced and provided to key informants in each jurisdiction for verification and revision as required.
3. During the course of this project, the Principal Investigator, Dr. Doreen Neville, was a Canadian Associate in the Commonwealth Harkness Program in International Health Policy. Dr. Neville presented the work in progress on the evaluation framework in bimonthly forums attended by a number international health policy experts and sought additional feedback from experts familiar with information system development throughout the year. At the final reporting seminar in Nashville, Tennessee in June 2003, a preliminary evaluation framework was presented and feedback received was incorporated into the framework.
4. In July 2003, participants were provided with both the inventory of current Electronic Health Record Initiatives Across Canada and the preliminary evaluation framework and asked to rank their priority evaluation questions within the 3 time frames proposed by the model (pre-implementation, implementation process and implementation impact). A total of 20 participants across Canada provided feedback on the proposed evaluation framework.
5. The feedback received from participants was used to further refine the proposed evaluation framework.
6. The framework document was then prefaced by a synopsis of the literature, which provides an overview of approaches to evaluation of electronic health records related projects, including perspectives on evaluation, evaluation models and frameworks commonly used, and a summary of the key messages regarding future initiatives in evaluation in this field. In addition, appendices were attached which provide a sample of evaluation questions and a menu of indicators used in previous studies or recommended for use in future evaluations.

Nature of the Intervention Under Study

As noted by Alvarez and Zelmer (1998), health information system initiatives across Canada share common goals, including: (1) integration of information systems to achieve a client focus and health services integration; (2) support for epidemiological research and health systems management; and (3) elimination of duplication and waste, with subsequent improvements of quality of care and reductions in cost. However, our review of the EHR initiatives in Canada (see the companion document Towards an Evaluation Framework For Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada) indicates that there is little uniformity in the design and planned implementation of the identified core components of an EHR (Unique Personal Identifier/Client Registry; Pharmacy Network, Laboratory Network and Diagnostic Services Network). Each jurisdiction has a differently configured legacy system upon which it is building its EHR and the form of the intervention under study is not consistent across the country. While this situation is not unique to Canadian health information systems initiatives (Healthfield 1999), the nature of the Canadian EHR interventions has implications for the types of evaluation approaches which will be most appropriate, as noted throughout the discussion of the literature which follows and the design of the proposed evaluation framework presented in this document.

Synopsis of the Literature

Critical appraisal of the published literature and project reports (see the companion document Towards an Evaluation Framework for Electronic Health Records Initiatives: An Annotated Bibliography and Systematic Assessment of The Published Literature and Program Reports) revealed that there is a dearth of information regarding evaluation of geographically dispersed health information systems. Most evaluations of information systems in health care have dealt with relatively small scale initiatives, wherein new technologies replace the existing (usually paper-based) system. The setting for most evaluation studies is within a hospital or a limited hospital to physician office interface (for example, enabling access to lab test results). Search of the literature did not detect a single study that describes the evaluation of a system with all four core components of an Electronic Health Record (EHR), although several large scale Health Information Systems Initiatives were very similar in their goals and level of technical complexity. We identified a total of 93 articles/reports which were germane to the development of an evaluation framework in this field, and these were described in the Annotated Bibliography. In addition, several seminal texts in the field were reviewed and considered in the development of the proposed evaluation framework.

Burkle, Ammenwerth, Prokosch and Dudeck (2001) concluded, following a review of evaluations of clinical information systems, that a generic approach to evaluation does not exist; the evaluation approach depends on available resources, goals of the evaluation and the type of technology that is being examined. While we concur with this assessment, we feel that there is value in highlighting several aspects of evaluation of health information systems which inform the task at hand - development of an evaluation framework for Electronic Health Records Initiatives across Canada.

In the synopsis of the literature below, we provide a overview of approaches to evaluation of complex health information systems, including: (1) the most common perspectives on evaluation; (2) a brief description of some of the models and frameworks which have either been used to guide previous evaluation efforts, are proposed for future evaluation projects, and/or have been developed by Canadian researchers; and (3) a summary of the key recommendations which emerged from the literature regarding future approaches to evaluation in this field.

Perspectives on Evaluation

Objectivist Versus Subjectivist

One of the best known classifications of perspectives on evaluation of health information systems (objectivist versus subjectivist) was proposed by Friedman and Wyatt (1997). The objectivist perspective is one in which: (a) agreement exists regarding the aspects of a system which are important to measure; (b) "gold standards" in terms of optimal systems performance exist and the outcomes of a given system can be compared against these standards; (c) system attributes can be described and measured using quantitative methods, which permit precision in analysis of findings, and replication of study findings in other similar settings.

The subjectivist perspective, in contrast, is one in which: (a) there are differing views on which aspects of a system are important to measure; (b) there is no "gold standard" against which to compare results and (c) qualitative methods are employed to understand the different opinions and conclusions legitimately reached by different observers in the same setting. The findings are not necessarily transferable to other settings, as the results are impacted by the context of the investigation.

Moehr (2002) reviewed the objectivist and subjectivist approaches proposed by Friedman and Wyatt in 1997, noting that these terms are preferable to the more common terms of quantitative and qualitative methods. Both quantitative and qualitative approaches are used in objectivist and subjectivist research, and the more important distinction is the focus on achieving maximum objectivity versus exploiting subjectivity in the investigation (p.114). Limitations of the objectivist approach to studying the complex world of health information systems include: (1) it is not possible to study the intervention in a vacuum, as health information systems are built to replace or complement existing systems, and the instead of evaluating the impact of one new product you are evaluating the dynamic process of adaptation of a new information system; (2) rigorous comparison studies, including RCTs, consume tremendous resources such as time, money and personnel, and the results are often not available in a timeline where input to system redesign is feasible; (3) it is often not possible to adhere to the constraints of RCT design, i.e. it is not possible to randomly select hospitals and fit them with complex information systems in order to study their effects. Moehr proposes that the subjectivist approach holds more promise, in that it addresses what people really want or need to know, attempts to describe the health information system, environment and effects as perceived by people, using detailed observation and inductive reasoning. He suggests that methodological extensions such as: (a) the inclusion of systems engineering approaches in the early phases of system development, and (b) an assessment of cognitive and social effects in the operational phases is desirable.

Healthfield, Peel, Hudson, Key, Mackay, Marley, Nicholson, Roberts and Williams (1997) note that today we are faced with the evaluation of large scale health information system projects which are incrementally developed from legacy systems. Many methodological and practical problems arise which are different from the issues faced in the past, when evaluations of health information systems were concerned with relatively small scale initiatives which replaced or enhanced paper-based records. Hence, subjectivist approaches may be more appropriate for some of these new evaluation challenges.

Formative Versus Summative

Formative evaluation occurs while a program is still developing and can be modified on the basis of the findings. In formative studies, the role of the researcher is to feed back results to those involved in the evaluation in order to inform ongoing program planning, development and refinement (King, Lyons Morris and Fitzgibbon, 1987; Fulop, Allen, Clarke & Black, 2001). Formative evaluations may be quite simple or very complex, depending on the focus of the inquiry (Rossi and Freeman, 1993). Activities undertaken during the design and pre-testing of programs to guide the design process, as well as activities related to monitoring program implementation and progress reporting, are all examples of formative evaluation (King et al, 1997).

Summative evaluations occur after a program has been established and are used to determine what has been achieved as a result of the program, i.e. outcomes/impacts, attainment of goals, unanticipated consequences, and possibly comparisons with alternative programs (including the pre-existing program) in terms of efficiency and effectiveness.

Scientific Versus Pragmatic Evaluation Perspectives

Donald Campbell is perhaps the best known proponent of the "scientific" social science research paradigm, which supports the use of experimental methods in social science evaluation research. Scientific studies attempt to meet a set of design and conduct standards set by peers in their field and the value of their work is judged against these standards (Rossi and Freeman, 1993). Evaluation methods are ranked according to their capacity to link cause and effect and mediate threats to internal and external validity. The randomized clinical trial is considered to be the "gold standard" method for scientific evaluation research (Cook and Campbell, 1979).

The "pragmatic' perspective acknowledges that while scientific investigations and evaluation efforts may use the same logic of inquiry and the same research procedures, the intent of evaluation studies differentiates them from purely scientific investigations (Rossi and Freeman, 1993). The purpose of evaluation is to (a) produce maximally useful evidence within a specified budget and time constraints (Cronbach, 1982); and (b) address the policy and program interests of the sponsors and stakeholders (Rossi and Freeman, 1993).

Accountability, Developmental and Knowledge Perspectives

Heathfield and Pitty (1998 ) identify 3 general categories of perspectives on evaluation: The accountability perspective (wherein the task is to answer the question about whether a particular intervention caused a particular outcome, i.e. . a cause and effect type question) usually involves the use of summative and quantitative methods, such as the use of randomized clinical trials. The developmental perspective (wherein the task is to strengthen institutions, improve agency performance or help managers with their planning, evaluating and reporting of tasks) usually involves formative evaluation methods (often qualitative but can be quantitative ) . The knowledge perspective (acquisition of a more profound understanding of some specific field) employs both qualitative and quantitative methods, dependent on the academic discipline of the researcher involved.

Heathfield and Pitty (1998 ) note that current health information system evaluations have tended to focus on the accountability perspective, with a subsequent pre-occupation with RCTs and quantitative approaches. They emphasize that new multi-method approaches are required. While sensitivity to accountability is heightened in resource-constrained times, they argue that evaluation focused on accountability in order to regain public trust is shortsighted and limits the gains that can be achieved from the developmental and knowledge perspectives on evaluation in the health information system field.

Models and Frameworks Commonly Used To Guide Evaluation Projects

The Delone and McLean Information Systems (IS) Success Model

In a landmark article, focusing primarily on Management Information System (MIS) applications, Delone and McLean (1992) provided a framework for characterizing and measuring the success of information systems. The framework includes 6 major dimensions or categories: system quality, information quality, use, user satisfaction, individual impact, and organizational impact. System quality measures (measures of the information processing system itself) tend to be engineering-oriented characteristics of the systems under study, such as response time, ease of use, system reliability, system accessibility, system flexibility and system integration. Information quality measures (measures of information system output) are addressed mostly from the perspective of the user and are therefore subjective in nature, such as information accuracy, timeliness, completeness, reliability, conciseness, and relevance. Frequently these measures are included as measures of user satisfaction as well. Measures of information use (recipient consumption of the output of an information system), including self-reported versus documented use, use by whom, frequency of use and extent of use, are valid only if system use is voluntary or discretionary. Measures of user satisfaction (recipient response to the use of the output of an information system) are the most widely utilized indicators of system success, primarily because of their inherent face validity, and the availability of reliable measurement instruments, such as satisfaction questionnaires. Individual impact measures (measures of the effect of information on the behavior of the recipient) are strongly tied to measures of performance, such as quality of decision making, change in decision behavior, time efficiency of task accomplishment, time to decision making, and confidence in decision making. Studies of this success indicator, while numerous are most often undertaken in laboratory settings, using students and computer simulations. Measures of organizational impact (the effect of information on organizational performance) have been derived primarily from the business sector and include cost reduction, cost effectiveness, contribution to profitability and return on investment (ROI).

The I/S success model is predicated on process and ecology concepts from the organizational effectiveness field, and proposes that success is a process construct which must include both temporal and causal influences on IS success. The authors suggest that there are many success measures which fall into the 6 dimensions described above. They emphasize that it is important to study the interrelationships among these dimensions, and to avoid arbitrarily selecting items from among the 6 dimensions to measure overall success if a clearer understanding of what constitutes information system success is to be achieved. They propose combining measures from the 6 categories to create a comprehensive measurement instrument. Furthermore, they suggest that selection of success measures should consider contingency variables, such as: the independent variables being researched, the size, structure, strategy and environment of the organization being studied, and the characteristics of the system itself.

In a ten-year follow-up article (DeLone and McLean, 2003), the authors provided a review of the I/S Success Model and an overview of how the model has been validated by research in the field. Suggestions for updating the model include; (1) adding a third dimension, "service quality" to the 2 original system characteristics, "system quality" and "information quality"; (2) substituting "intention to use" for "use" as a measure of system usage some contexts; and (3) combining the" individual impact" and "system impact" variables into a "net benefits" variable. They further suggest that "net benefits' variable must be defined within the context of the system under study and within the frame of reference of those assessing the system impact, as these variables substantially influence what constitutes net benefits and hence IS success

Social Interactionist Models

Social Interactionalist Models (Kaplan 1997, 1998) consider relationships between system characteristics, individual characteristics and organizational characteristics and the effects among them. Consequently, evaluations based on these models consider not only the impact of an information system on a organization, but also the impact of the organization on the information system, and tend to be process-focused. The framework is informed by theoretical models of organizational change, user reactions to health information systems and Roger's work on innovation diffusion (Rogers, 1993).

Evaluation questions within an interactionist framework address issues of Communication, Care, Control and Context (the 4 Cs). The evaluation questions are: (1) what are the anticipated long term impacts on the ways that departments linked by computers interact with each other; (2) what are the anticipated long term effects on the delivery of medical care; (3) will system implementation have an impact on control in the organization; and (4) to what extent to medical information systems have impacts that depend on the practice setting in which they are implemented? Kaplan suggests that it is difficult to study processes over time and proposes 5 methodological guidelines that can be useful when developing a comprehensive evaluation framework. The evaluation framework should: (1) focus on a variety of technical, economic and organizational concerns; (2) use multiple methods; (3) be modifiable; (4) be longitudinal; and (5) be formative and summative

Cognitive Evaluation Approaches

Kushniruk, Patel and Cimino (1997) identify the need for improved methodologies for the assessment of medical systems and their user interfaces. Conventional methods of evaluation, such as questionnaires and interviews with users, rely on the user's memory of their experience with using a computer system (what they think they did when using the system) which may be quite different from their actual behavior. Therefore, there is a need to incorporate into system design and evaluation processes sound methodologies for the assessment of medical systems and their user interfaces.

Cognitive evaluation approaches encompass a continuum of methods ranging from experiments (laboratory based usability testing where test conditions are tightly controlled), to simulations (laboratory based low and high fidelity simulators) to naturalistic approaches (field based observations using ethnographic methods and unobtrusive recording). Methods which can be applied in the study of health information systems in both the laboratory and real life settings include (1) usability testing - evaluation of information systems that involves subjects who are representative of the target user population; (2) cognitive task analysis - characterization of the decision-making and reasoning skills of subjects as they perform activities involving the processing of complex information; and (3) computer supported video analysis - video recording of subjects as they interact with user interfaces in carrying our specific tasks. The 8 steps employed in carrying out cognitive evaluations of health care systems and user interfaces include: (1) development of the test plan; (2) study design, including selection of representative users; (3) selection of representative task /contexts; (4) set up of the test environment; (5) conducting the usability test; (6) data analysis; (7) recommendations to designers; (8) iterative input to design.

Kushniruk et al (1997) note that while cognitively-based usability testing can be applied throughout the lifecycle of information systems (from early formative evaluation during design work to summative evaluation to determine if a computer system has met usability criteria), their experience to date has found that the greatest benefits come from the formative analysis work (p. 221). Kushniruk (2002) suggests that future evaluation efforts with health information systems should integrate evaluation approaches which examine process variables (such as usability engineering) with approaches which address measurement of outcome variables

PROBE

The purpose of this document is to provide practical guidance for those involved in the evaluation of Electronic Patient and Health Records in the NHS in Britain and Wales. The PROBE (Project review and objective evaluation for electronic patient and health records projects) guidance was prepared by the UK Institute of Health Informatics for the NHS Information Authority (NHS Information Authority, March 2001), as an extension and update of the earlier PROBE guidance issued in 1996 by the NHS and as a companion document to the Evaluation of Electronic Patient and Health Records Projects document released in January 2001. It extends the original PROBE document in 2 ways: first by focusing on evaluation questions which are important to EPR/EHR projects and secondly by providing more detailed information about how to evaluate, including a review of the various tools and techniques available.

PROBE suggests that there are 4 essential standards for an evaluation study which need to be tested throughout the evaluation planning stage: utility; feasibility; propriety and accuracy. It also stresses the importance of an evaluation framework, which focuses stakeholders on the expected benefits and barriers of an EPR/HER and methods of measuring these. The key principles of evaluation emphasized are the need for both formative and summative elements, advance planning, close integration to the project lifecycle; clearly defined aims and objectives, the inclusion of a before and after (comparative) element, and the collection of quantitative and qualitative data.

Six steps are proposed to plan an evaluation of an electronic patient record or electronic health record initiative: (1) agree why an evaluation is needed; (2) agree when to evaluate; (3) agree what to evaluate; (4) agree how to evaluate; (5) analyze and report; and (6) assess recommendations and decide on actions.

A suggested format for such an evaluation framework is a tabular summation of the following: (1) timing of the review, which includes recommendations for pre-implementation assessment of readiness to implement; implementation reviews carried out at each stage of the process, and operational evaluations which are carried out on the system as it is used in practice post implementation; (2) the research objectives/questions the system is designed to test (in the case of the NHS these questions were organized around 5 themes, which were strategic, technical, operational, human and financial considerations); (3) one or more specific measurement criteria for each research question; (4) the study design to be used; (5) sources of data to be collected for each measurement criterion.

Total Quality Management (TQM)

Drazen and Little (1992) suggest that new approaches are needed to evaluate clinical and management applications of health information systems in order to measures benefits that are important to the institutional sponsors of health information system projects. Proposed enhancements to the traditional cost-benefit approach to evaluation include: (1) driving to achieve benefits as the primary evaluation goal, including more than direct cost savings, i.e. improvement in level of service and improvement in the outcomes of care; (2) focusing on critical issues and using standard tools to achieve efficiencies, i.e. measure what is important, not what is easy to measure; (3) maintaining independence, given the involvement of the private sector in many of the evaluation initiatives; (4) fitting with the institutional philosophy.

Drazen and Little (1992) propose a TQM framework for evaluation which incorporates the concept of continuous quality improvement. An example of a TQM approach to benefits assessment is then outlined: (1) identify improvement opportunities - identify the information processes that need improvement. If a large number of processes are identified, the priorities can be established by considering their importance to a multiple stakeholders; the difficulty in achieving improvement and the strategic importance of improvement. (2) understand priority processes, from the perspectives of relevant stakeholders; (3) find the root cause of the problem; (4) confirm the root cause; (5) identify improvement options; (6) track progress; (7) monitor to insure continuous improvement.

The Team Methodology

A systems perspective informs the model developed Grant, Plante and LeBlanc (2002) to evaluate the overall function and impact of an information system. Key tenets include: (1) the processing of information by a system can be distinguished at 3 different interacting levels: strategic, organizational, and operational, and these levels are a useful way of situating an evaluation; (2) the evaluation should be dynamic and include both formative and summative analyses; (3) the evaluation approach must be acceptable in terms of the resources and time it requires to complete; and (4) the evaluation should be longitudinal. The authors propose that an evaluation exercise should address the (a) who - role categories of persons who should participate in the evaluation; (b) when- time requirements and the timing of stages of evaluation; and (c) what- state the main and sub objectives of the evaluation exercise; the key perspectives which will be addressed, identify measures to be used and to specify the documentation required for the evaluation exercise.

Health Technology Assessment

Kazanjian and Green (2002) propose a Health Technology Assessment Framework as a conceptual tool for decision-making about health technologies, including information technologies. Although the Comprehensive Health Technology Assessment Framework discussed in this paper is primarily aimed at stakeholders involved in the adoption of new health technologies, the authors propose that it has relevance for decision makers who need to compare the impact of information system technologies within a framework that is inclusive of all competing health technologies. Impacts are considered at the societal level, not just the organizational setting in which the health information system is implemented, and from the perspective of patients and society as primary stakeholders. The major framework dimensions are (1) population at risk, (2) population impact, (3) economic concerns, (4) social context (including ethical, legal, and political concerns), and (5) technology assessment information.

Framework For Action Research

Action research is an approach to conducting research which emphasizes the importance of doing research with and for people as opposed to on people; it focuses on generating knowledge about a social system and using that knowledge to change the system as part of the research process itself (Meyer, 2001, p 172-173). Lau (1999) notes that action research has been used in social sciences since the 1940s to integrate theory with practice through an iterative process of problem diagnosis, action intervention and reflective learning, but is still not well recognized as a method of inquiry among mainstream IS researchers and journals.

The four dimensions of the Framework for Action Research proposed by Lau are : (1) conceptual foundation; (2) study design to describe the methodological details; (3) the research process of diagnosis, actions, reflections and general lessons ; and (4) the respective roles of the researcher and participants.

Four main role categories are identified: (1) those involved in the conception and design of the information system; (2) those who are responsible for the implementation and functioning of the system (specialist user); (3) those who use the system (end user) and (4) those who have a stakeholder interest that the information system is a success. There is a requirement for a definition of evaluation priorities from each role category's point of view and a recognition by all of the constraints attached to the evaluation process so that the evaluation program is valid and achievable.

Balanced Score Card

The balanced scorecard (BSC) is a means to evaluate corporate performance from four different perspectives: the financial perspective, the internal business process perspective, the customer perspective, and the learning and growth perspective (Kaplan and Norton, 1992). When Denis Protti (founding Director of the School of Health Informatics, University of Victoria) was invited to assist in the development of the evaluation methodology for the NHS Information Strategy, he proposed the use of the BSC (Protti, 2002). Protti noted that investments in health information systems are costly ventures and frequently asked questions include concerns about the success of such systems and the degree to which substantial investment has proved worthwhile. Challenges to addressing these concerns include: (1) efficiency (doing things right) is easier to measure than effectiveness (doing the right thing); (2) new systems are intended to change difficult to measure actions; (3) strategic systems elude measurement; and (4) infrastructure investments can not be justified on a ROI basis. IT infrastructures, like many other public infrastructures such as roads and hospitals require large long term investments, and are difficult to cost-justify in advance. It is often difficult to show benefits in hindsight as well.(p. 229).

Although the process of building a BSC for a health information systems initiative would not be simple, the author posits that the benefits would be worthwhile, as a BSC would allow managers to see the positive and negative impacts of information management and technology activities on the factors that are important to the National Health Service as a whole.

Key Messages Regarding Future Efforts To Evaluate Complex Health Information Systems

Green and Moehr (2002) have observed that the common core components of Canadian performance evaluation frameworks in health care include: clinical outcomes/effectiveness, accessibility, customer/stakeholder satisfaction; coordination; financial/efficiency, quality; innovation and internal business production. Less frequently included components are appropriateness, safety, health status and integration. No framework, except the Canadian Institute for Health Information (CIHI) framework considers the optimal use of health information system capacity. This omission suggests that the integration of the EHR initiatives into the overall strategic planning efforts of the Canadian health care system still has a long way to go.

Until electronic health records are considered a key strategic initiative in the management and delivery of health services in Canada, difficulties in evaluating the impact of such initiatives will be compounded by lack of progress in implementation. Healthfield and Buchan (1996) described this quandry as a "catch 22" situation. Information technology initiatives are viewed with suspicion by many. Less than positive results from early evaluations (which focused soley on economic benefits) have mounted additional barriers to future system development. In most jurisdictions, decision makers, including the central funding agencies of government, require evidence to support the investment of millions of dollars in health information system infrastructure. However, until we first build the systems and simultaneously introduce broader evaluation, we will not have the evidence required to support nation-wide implementation of EHRs (Healthfied 1999).

Two very difficult general problems regarding evaluation of complex health information systems (such as Electronic Health Records or Electronic Patient Records) remain: (1) there are many versions of an EHR or EPR and no 2 implementation processes are alike, making comparisons difficult; and (2) there are a plethora of possible evaluation questions and it is difficult to decide which one to address (Healthfield, 1999). Some of the key messages extracted from the review of the literature concerning the need for broader, more inclusive, and yet flexible approaches to evaluation of complex health information systems include:

• A planned evaluation, introduced at the initial project stages, can help overcome many obstacles. (Healthfield, 1999).
• It is important to develop a process for engaging stakeholders, particularly physicians, in establishing principles and premises for large IS projects (Protti, 2002).
• Evaluation frameworks should: (1) focus on a variety of technical, economic and organizational concerns; (2) use multiple methods; (3) be modifiable; (4) be longitudinal; and (5) be formative and summative (Kaplan, 1997)
• Many formal evaluations of major information technology investments in the public sector have focused on critiques of implementation rather than assessment of health care benefits. The time has come to attempt to quantify benefits not just in organizational, business or financial terms, but also with respect to health outcomes and the intermediary variables which lead to improved health outcomes in the health care delivery system, including improved diagnosis, more effective treatment, more focus on prevention, less errors and more evidence-based decision making (Donaldson, 1996).
• Evaluation is not just for accountability, but also for development and knowledge building. Future evaluations should be multi-perspective, multi-method, include qualitative methods and involve diversely constituted research teams (Healthfield et al, 1998).
• Limitations of RCTs identified include: (1) low power - not enough observations (Burkle et al, 2001); (2) inability to blind subjects to their assigned group ( Burkle et al, 2001); (3) costs (Healthfield et al, 1998); (4) limited external validity (Healthfield et al, 1998)
• When faced with the challenge of evaluating complex systems which have been implemented in less than standardized fashion, it is reasonable to focus on the form and function of the systems implemented (ie the concept of a total health record) instead of trying to distinguish, for evaluation purposes, the difference between different systems (Healthfied, 1999).
• Lessons learned from the evaluation of district health information systems in South Africa include: (1) avoid the use of overly complex handbooks, guides to evaluation or instruments; (2) identify core evaluation criteria which can be used for either self assessment by the participating sites or as baseline assessments for the project as a whole; and (3) develop evaluation protocols in consultation with the sites (Hammer,1999).

Proposed Approach to Planning an Evaluation of an EHR Initiative

The proposed approach to planning an evaluation of an EHR initiative presented below was informed by: (1) a review of the current EHR related initiatives across Canada (see Health Information Systems Inventory); (2) the team's personal involvement with EHR initiatives in Newfoundland and Labrador through the Newfoundland and Labrador Centre for Health Information (NLCHI); (3) a systematic review of the literature (see synthesis above and the Annotated Bibliography) ; and (4) feedback from key informants on earlier drafts of the framework.

This framework is designed to be a guide to designing an evaluation initiative which is useful to a wide range of stakeholders involved in the EHR initiatives across Canada, including those who fund the system development and implementation, policy makers, decision makers at all levels of the system, users of the system, and researchers. It is not an academic document, and it does not propose a conceptual model for understanding the design, implementation or impact of complex health information systems. Rather, it seeks to provide a practical guide to the types of questions which can be asked of the EHR initiatives, the options available to address these questions, and some of the tradeoffs that will occur if one or another approach to evaluation is selected. As a guide, it is illustrative, not exhaustive.

Our review of the EHR initiatives in Canada indicates that there is little uniformity in the design and planned implementation of the identified core components of an EHR (Unique Personal Identifier/Client Registry; Pharmacy Network, Laboratory Network and Diagnostic Services Network), and each jurisdiction has a different configuration of legacy system upon which it is building its EHR. Faced with a similar scenario in the National Health Service in the United Kingdom, evaluators such as Heathfield and colleagues (1999) chose to study the system in terms of form and functionality (i.e. the Electronic Health Record) , as opposed to distinguishing between different systems for evaluation purposes. In the evaluation framework presented below, the system can refer to the full EHR in each jurisdiction, or one or more subcomponents of the EHR, irrespective of the particular technology which was implemented to achieve the desired functionality.

We hope that this framework will serve as a springboard and guide for discussions among key stakeholders regarding what is important to measure about the EHR initiatives in Canada, and how to feasibly address it in a rigorous manner. If the framework is used in several jurisdictions, then it will be possible to begin identifying common evaluation priorities, track and compare evaluation questions and methods, begin to compile a national inventory of EHR evaluation projects, and identify opportunities for collaborative projects across jurisdictions and stakeholder groups.

Steps for Framework Development

The framework is organized around several steps, as presented below, and is complemented with appendices which provide source data for some of the questions and indicators.

Step 1: Identification of Key Stakeholders in Each Jurisdiction

We have identified several categories of stakeholders who would be considered core to an evaluation of the full EHR initiative in each jurisdiction. Readers will note that representatives of a variety both national and provincial/territorial sectors are included in this list. It is t important that a wide range of stakeholders be involved in and apprised of the evaluation efforts within their own jurisdictions. It is also crucial that a number of individuals and organizations are aware of the initiatives across the country , because it will improve the likelihood that: (1) evaluation of EHR initiatives will get on and remain on the radar of these organizations as a strategic initiative and one which requires dedicated resources for input; (2) greater strategic alignment between the goals of the broader health system and the goals of the EHR initiatives will occur; (3) information exchange across jurisdictions will occur; (4) comparable evaluation approaches will be introduced across the country where feasible; (5) long term, stable champions for evaluation of EHR initiatives will be engaged at both the national and provincial/territorial levels.

Key Stakeholders to Initially Engage in Planning for Evaluation

Funders	Health System Administration	Other Health Sytem - Related Agencies	User Groups	Researchers /Academics
Provincial Government Treasury Board Chief Information Officers Health Department Personnel Provincial Organizations with the mandate to build EHRs (such as NLCHI, WellNet) INFOWAY Private Sector/IT companies if public /private partnerships are in place	CEOs VPs Finance Information Technology Managers Medical Administrators· Nursing Administrators Managers of Other Clinical Departments impacted by the EHR initiative	Canadian Institute of Health Information Canadian Office of Health Technology Assessment F/P/T Council of Deputy Ministers Privacy Advocates Patient/Consumer Groups	Physicians Nurses Pharmacists laboratory technologists IT support personnel Patients	Canadian Institute of Health Information Canadian Health Services Research Foundation Provincial Health Services Research Foundations University Faculties (Business, Economics, Nursing, Medicine, Pharmacy, Computer Science, Psychology, Sociology etc.)

Step 2: Orient Key Stakeholders to the EHR Initiative and Reach Agreement on WHY an Evaluation is Needed

It is important to orient key stakeholders to the EHR initiative and the evaluation process as early as possible, to determine their: (a) expectations of the EHR initiatives in their jurisdiction and (b) views on what an evaluation plan should address. A workshop format has proved useful for this type of stakeholder engagement, wherein an overview of the EHR initiative is presented; expectations documented and views on evaluation elicited.

Healthfield (1998) suggests that there are 3 general types of rationale for why evaluation is conducted in the field of health information systems: (1) to insure accountability for expenditure of resources; (2) to develop and strengthen performance of agencies, individuals and/or systems; and (3) to develop new knowledge. Given the diversity of key stakeholders involved with EHR initiatives, it is highly likely that they will identify different rationales for conducting evaluation. For example, one could expect that individuals/agencies responsible for the administration of public funds would highlight accountability as a major reason for evaluation; clinicians and administrators would be most interested in performance enhancements, and academics would likely value most highly the opportunity to gain new knowledge in their respective fields.

While each of these rationales for evaluation may consider evidence collected by a variety of approaches, both qualitative and quantitative, they carry with them: (1) assumptions about what evaluation can contribute; (2) orientation towards particular evaluation methods; and (3) requirements in terms of the timelines and resources necessary to address them.

Rationale For Evaluation	Assumptions	Evaluation Methods	Requirements
Accountability: (measurement of results, such as efficiency, effectiveness, impacts on costs, health outcomes etc)	Interventions result in outcomes which can be accurately measured. There is a "gold standard" against which results can be compared to determine if they are positive The measuring process does not affect the intervention under study Numerical measurement is superior because it allows statistical analysis	Summative Economic evaluations, such as cost effectiveness analysis, cost benefit analysis, cost minimization analysis Randomized Clinical Trial, quasi-experimental designs, before and after studies	Pre and post implementationdata collection Long time period Resource intensive Provides data on impact and outcomes Results not necessarily generalizeable but important to show trends in the findings across jurisdictions
Performance Enhancement: evaluation to improve the performance of individuals /organizations	Providing people /organizations with feedback about their performance empowers them to improve their performance. There is no gold standard; results obtained from observation are dependent on the context and the observer. Understanding and documenting differences of opinion is an important part of evaluation; subjectivity is ok. Qualitative data is valued for its richness and detail	Primarily qualitative data, observations, interviews Quantitative methods also employed, primarily surveys	Data collection during implementation and post implementation Measurements pre -implementation desirable for many questions Less costly than impact studies Provides information about process which is important on its own but also helps interpret information gathered about impacts /outcomes.
Knowledge development (to gain more in-depth understanding in some specific discipline or field	Dependent on the academic discipline, the research orientation of the investigators and the research question being addressed	Academic discipline May include qualitative methods, quantitativemethods or both	Dependent on study design Essential to moving the field of health informatics forward Direct relevance of results not always obvious to practitioners

Step 3: Agree on When To Evaluate

Ideally, evaluation of complex information systems should involve longitudinal evaluation, that is, evaluation that occurs over time, and/or involves multiple data collection points (reference and rationale). We recommend that whenever possible, the evaluation of EHR projects in Canada involve data collection at 3 or more points: (1) baseline (pre-system implementation); (2) during implementation and (3) post implementation (preferably multiple measures at 6 and 12 months post implementation). We recognize that many jurisdictions have introduced (or are about to introduce) one of more components of a province wide EHR, and hence new baseline collection of data is not possible. However, pre-implementation data may be available from Scoping Exercises conducted prior to system implementation, or from separately conceived and completed evaluations of work flow, audits of patient charts, or research projects. Whenever pre-existing measures are available they should be noted so as to inform the design of any evaluation projects which are conducted during the implementation and post implementation phases.

Step 4: Agree on What to Evaluate

It is well recognized that there are virtually an endless number of research and evaluation questions which could be posed about complex health information systems such as the Canadian EHR initiatives (see Appendix A). However, resources to pursue these issues are limited, in terms of funding and availability of personnel with expertise to conduct the evaluation. Therefore it is very important that each jurisdiction feels that it is gaining the maximum benefit it can from the investment of scarce resources in evaluation. A priority setting exercise with key stakeholders is one way to (a) identify the questions that it is important to answer (versus the questions that it is easy to answer) and (b) insure that all key stakeholders have an investment in the evaluation projects which are undertaken. If the evaluation framework proposed in this document experiences wide uptake across Canada, there will also be an opportunity to avoid duplication of effort where possible, or to strengthen the design of a project by conducting it simultaneously in more than one jurisdiction. One approach to priority setting would be build on the stakeholder identification of why an evaluation is important (accountability, performance enhancement and/or knowledge development) and then identify core and optional questions within each category.

Step 5: Agree on How to Evaluate

As noted above, both the rationale for undertaking evaluation, and the particular questions which are important to each stakeholder, have implications for the methods which can be used to conduct the evaluation. The Tables below provide an illustration of Steps 4 and 5, with a sample of potential core questions for each evaluation period and category highlighted in bold type.

A discussion of the most feasible methods for approaching the selected evaluation questions will involve consideration of the tradeoffs involved with the methods chosen. Each jurisdiction will need to consider the resources they have available to devote to the evaluation and determine the best use of those resources in term of evaluation questions addressed and methods used. In addition, we support the recommendations of Kaplan (1997) that all jurisdictions undertake an evaluation which: (a) focuses on a variety of concerns; (b) uses multiple methods: (c) is modifiable; (d) is longitudinal; and (e) includes both formative and summative approaches (formative evaluation involves mostly process evaluation of a system during implementation; summative evaluation assesses a system once it has been implemented and operational for a period of time). Grant et al (2002) further suggest that the evaluation be timely, realistic, practical and endorsed by key stakeholders. The current thinking around evaluation of complex health information systems leans towards evaluation geared to performance enhancement and knowledge development, and away from accountability, particularly costing approaches to net benefits assessments. However, accountability remains a strong value in Canadian society in general and increasingly in the health and technology sector, and therefore we recommend that some type of accountability question be included in the evaluation approaches in each jurisdiction.

Step 6: Analyze and Report

Many researchers have noted that the task of consolidating the findings of a multi- method evaluation is perhaps the most difficult component of the study of complex health information systems (Healthfield et al, 1999; Herbst et al, 1999; Moehr, 2002; Lau, 1999). It is likely that most jurisdictions will select one or more evaluation questions to address, and the evaluation effort will consist of several sub components which are in fact separate evaluation projects, involving different methods and disciplines. We recommend that the findings from each evaluation project within the evaluation initiative be shared with those key stakeholders identified in Step 1, preferably in a workshop setting. This approach will permit fuller discussion of the interpretation and implications of the results obtained through different projects, or through the use of multiple methods within each project.

Step 7: Agree on Recommendations and Forward Them to Key Stakeholders

The network of key stakeholders attending the Workshop (Step 6) are also those who should be involved in generating the recommendations which arise from the findings of the evaluation. Those responsible for knowledge-generation oriented studies will have responsibilities to generate recommendations specific to their discipline/field of inquiry. These recommendations may prove to be relatively straightforward and not subject to much broad debate within the evaluation team ; the debates which occur in academic circles may be more contentious but of little direct impact on the evaluation team as a whole. Development oriented studies will face more discussion from the evaluation team and hence disagreements regarding recommendations may arise. Accountability-oriented studies, which impact on all evaluation team members (and on the users and funders of the information system initiatives), can anticipate more lively debate regarding interpretation of findings. Subsequent development of recommendations, particularly if the recommendations arising are negative in terms of continuation of the initiative may be challenging.

There is no guarantee that the process of engagement used to generate the evaluation questions and approaches will ensure a consistent interpretation of what recommendations can be supported by the results. There is however a greater likelihood that common stances on at least some of the key issues will be found if those involved are: (a) familiar with the main issues from the start; (b) aware of the different perspectives each team member brings to the discussion; and (c) comfortable that the variety of methods used in the evaluation produced the most unbiased results possible.

Proposed Framework: Time Frames, Core Questions, Indicators, Data Sources and Study Design

Evaluation for Accountability: Measurement of Results

Time Frame	Sample Questions	Indicators	Data SOURCES	Study Design
Pre-Implementation	1. What are the predicted benefits and costs of this system?	Projections of system : costs benefits return on investment (ROI)	Project scoping documentation Business case	Descriptive
	2. Does this investment fit strategically with the direction and priorities in the jurisdiction?	Support given to EHR type systems development in the past, financial/political Recognition of the role of IS in other policy initiatives, i.e. primary care, regionalization, wellness, move to evidenced -based decision-making	Government and organizational strategic plans, annual reports, mission statements Interviews with key stakeholders
	3. Are the necessary management structures in place?	project management documents standards privacy protocols	Project scoping documentation Internal policy documents

Evaluation for Accountability: Measurement of Results

Time Frame	Sample Questions	Indicators	Data Sources	Study Design*
Implementation and Post Implementation (Most studies with an accountability focus will build on pre-implementation documentation and then require data collection towards the end of the implementation period and at least one point, preferably 2 or more post implementation (i.e. 6 months and 12 months post implementation)	1. What were the costs of implementing this system and how do they compare with projected costs?	Cost of the technology Personnel costs Cost of training/user support	Project budget documents Host budget documents	Economic evaluations such as cost effectiveness analysis and cost benefit analysis Before and after studies
	2. What benefits were achieved and how do they compare with projected benefits? Clinical benefits	Avoidance of errors Avoidance of adverse events Improved patient outcomes Improved information quality	Quality and performance indicators Clinical indicators System logs and audit trails	Before and after studies Randomized clinical trials
Implementation and Post Implementation (Most studies with an accountability focus will build on pre-implementation documentation and then require data collection towards the end of the implementation period and at least one point, preferably 2 or more post implementation (i.e. 6 months and 12 months post implementation)	2. What benefits were achieved and how do they compare with projected benefits? Administrative benefits Economic/ resource benefits	Improved communications Enhanced capacity to achieve strategic goals	Interviews with key personnel System logs and audit trails	Before and after studies Repeated measures studies Randomized clinical trials
		Operating costs Length of stay Use of unnecessary tests Visits per clinician Waiting times	Operational budgets Chart audits Interviews with clinicians and patients Scheduling records	Before and After Studies Repeated measures studies Randomized Clinical Trials Useability engineering studies

Evaluation for Performance Enhancement

Time Frame	Sample Questions	Indicators	Data Sources	Study Design
Pre-Implementation	1. What is the state of readiness within the sites for implementation of the system?	Training and support programs in place Project management structures in place Security /privacy structures in place	System implementation plan Privacy policy statements Privacy impact statements	Descriptive
	2. What are the expectations and concerns of key stakeholders?	Stated expectations for the system's impact on: patient safety clinical productivity relationship with patients costs privacy communication	Surveys Questionnaires Interviews Focus groups	Descriptive; cross-sectional data collection; May be used as baseline data for comparative study designs in the implementation and post implementation phases
	3. What are the current levels of data quality?	Data availability Data completeness Data accurateness	Project scoping documents Current indicators and benchmarks Quality indicators Current audits	Descriptive; can be used as baseline for before and after studies
	4. Is the new system technically able to perform the functions it is expected to?	Data availability Data completeness Data accurateness	Prototype testing On-site pilot testing	Usability engineering approaches, usability testing Usability walkthrough Design walkthrough usually laboratory based with a high degree of experimental control
	5. What are the current work processes in the areas which will be impacted by the new system	Patient scheduling Discharge planning Medication prescribing Turn around time for lab and diagnostic tests Access to clinical information when needed Workflow analysis analysis of decision-making	Project scoping documents Current indicators and benchmarks Quality indicators Current audits Interviews observations	descriptive; may be used for baseline in before and after studies

Evaluation For Performance Enhancement

Time Frame	Sample Questions	Indicators	Data Sources	Study Design
Implementation and Post Implementation	1. Is this system useable?	system response times user satisfaction with user interface and system functionality time for task completion ease of access	Observations Video analysis Interviews with users	Usability testing approaches; descriptive methods in the setting; may involve some laboratory simulations
	2. Does the system deliver the information clinicians and managers need to make decisions?	time to complete tasks use of the system to make decisions Routine use of the system	Observations Video analysis Interviews with users System audits System logs	Descriptive; may be used as part of a before and after study or a repeated measures study Useability engineering approaches
Implementation and Post Implementation	3. Is the necessary level of support available to individuals to allow them to use the system efficiently and effectively?	Routine use of the system Use of on-line help functions Use of technical support personnel Time to complete tasks	Questionnaires Surveys Interviews Focus groups Observation	Repeat measures, during implementation and post implementation. Usability studies in the setting to evaluate how well the system supports clinical and management decision- making.
	4. Is the implementation proceeding as anticipated?	Implementation timelines Change requests Costs	Project management records Observations Interviews Focus groups	Repeat measures, during implementation and immediately post implementation

Evaluation For Knowledge Development*

Time Frame	Sample Question	Indicators	Study Designs /Discipline Approaches
Pre- Implementation	1. Can the costs and benefits of these EHR systems be quantified?	Validity and reliability estimates of cost and benefit indicators	Econometric measurement approaches such as cost effectiveness analysis
	2. How may information technologies be tailored for use by a wide variety of individuals in a wide variety of places?	User performance in simulations User feedback Task analysis	Cognitive psychology approaches Useability engineering

The majority of these sample questions were extracted from the article by Kaplan and colleagues, 2001, titled 'Towards an Informatics Research Agenda: Key People and Organizational Issues". There are a tremendous variety of evaluation research questions, disciplines with expertise to address them, and potential study designs and information sources, as summarized in the description of this article in Appendix B.

Evaluation For Knwoledge Development*

Time Frame	Sample Question Score = Bold	Indicators	Study Designs /Discipline Approaches
Implementation and Post Implementation	1. What is a successful implementation of an information system and what is the best way to measure it?	Delone and Maclean (1992 and 2003) suggest information quality system quality service quality use/intention to use user satisfaction net benefits	Variety of study designs, usually comparative, some before and after studies and some RCTs
	2 How do linkages through IT affect organizational identity and integrity?	Workflow processes Communication patterns Workforce satisfaction Workforce loyalty Strategic use of IT in decision making	Social/interactionalist Sociology/organizational behavior Mainly qualitative methods

Concluding Remarks

As noted above, this framework is meant to serve as template for the design and conduct of evaluation studies to assess the Electronic Health Records initiatives in Canada.. We hope that the seven step approach outlined in this framework documents resonates with key stakeholders as a practical and useful guide.

As a guide, this framework is illustrative, not exhaustive. The appendices do however provide a large inventory of additional potential evaluation questions and indicators to choose from, and this information is supplemented by the companion document to this report: Towards An Evaluation Framework for Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Program Reports ( Neville et al, February 2004).

There are several key points that we would like to emphasize in our closing comments:

• In Canada today, we have a tremendous opportunity to collaborate across jurisdictions and stakeholder groups to: develop: (1) a standardized approach to assessment of EHR initiatives and (2) a national inventory of evaluation protocols, instruments and evidence. If this framework serves as a springboard for discussion among key stakeholders regarding what is important to measure about EHR initiatives in Canada and how to measure it, then we will have laid a foundation upon which common evaluation priorities across jurisdictions can be identified and pursued.
• It is crucial that collaborative evaluation efforts around EHR initiatives focus on the functionality of the systems being introduced, as opposed to the specific form of the technology being employed. This is not to suggest that there is no need for evaluation of proprietary technology, but rather to urge the key stakeholders identified in this document to find ways to work together to answer the big picture questions around the implementation of EHRs.
• It is important to be clear about what you hope to learn from an evaluation, and your underlying assumptions about what evaluation can and cannot achieve. We recommend use of the accountability/performance enhancement/knowledge development classification of perspectives presented in this framework to: (1) stimulate discussion among stakeholders about their primary rationale for expending scarce resources on evaluation activities and (b) aid the group to identify the types of tradeoffs which will be required as a consequence of the evaluation questions they pursue and the resources they can access to complete the study.
• This framework document and its companion documents are designed as a springboard for future discussion and dialogue, and as such will be widely circulated. Comments and suggestions should be directed to:
  
  Doreen Neville, ScD
  Associate Professor,
  Health Policy and Health Care Delivery
  Faculty of Medicine
  Memorial University of Newfoundland
  
  Email: dneville@mun.ca
  Phone: 709-777-6215
  Fax: 709-777-7382

The research team would like to express its sincere appreciation to all the key informants who assisted with the development of this framework. Thank you very much!

Bibliography

Adams WG, Mann AM, Bauchner HB. Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics 2003; 111:626-632.

Adragna L. Implementing the enterprise master patient index. Journal of AHIMA 1998; 69(9):46-52.

Aller RD. Creating integrated regional laboratory networks. Clinics in Laboratory Medicine 1999; 19(2): 299-316.

Alvarez R. The promise of e-health - a Canadian perspective. EHealth International 2002; 1(1):4.

Alvarez RC, Zelmer J. Standardization in health informatics in Canada. International Journal of Medical Informatics 1998; 48(1-3):13-18.

Amatayakul M. Critical success factors - steps to take to achieve a truly integrated information system. Health Management Technology 2000; 21(5):14-18.

Anderson J, Aydin C, Jay S. (Eds). Evaluating Health Care Information Systems: Methods and Applications. Thousand Oaks, California: Sage Publications, 1994.

Anderson J, Jay S, Anderson M, Hunt T. Evaluating the potential effectiveness of using computerized information systems to prevent adverse drug events. Proc AMIA Symp 1997; 228-232.

Anderson JG. Evaluation in health informatics: computer simulation. Computers in Biology and Medicine 2002; 32(3):151-164.

Anonymous. Computerized provider order entry systems. Health Devices 2001; 30(9-10):323-359.

Aspinall MB, Whittle J, Aspinall SL, Maher RLJ, Good CB. Improving adverse-drug-reaction reporting in ambulatory care clinics at a Veterans Affairs hospital. American Journal of Health System Pharmacy 2002; 59(9):841-845.

Bakker AR, Leguit FA. Evolution of an integrated HIS in the Netherlands. International Journal of Medical Informatics 1999; 54(3):209-224.

Balas EA. Information systems can prevent errors and improve quality. Journal of the American Medical Informatics Association 2001; 8(4):398-399.

Baldwin FD. Once is enough. Healthcare Informatics 2001(July); 30-33.

Bamford W, Rogers N, Kassam M, Rashbass J, Furness P. The development and evaluation of the UK national telepathology network. Histopathology 2003; 42:110-119.

Baorto DM, Cimino JJ, Parvin CA, Kahn MG. Using Logical Observation Identifier Names and Codes (LOINC) to exchange laboratory data among three academic hospitals. Proc AMIA Symp 1997; 96-100.

Bates D, Leape L, Cullen D, Laird N, Petersen L, Teich J et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Journal of the American Medical Association 1998; 280(15):1311-1316.

Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N et al. The impact of computerized physician order entry on medication error prevention. Journal of the American Medical Informatics Association 1999; 6(4):313-321.

Bates DW, Gawande AA. Improving safety with information technology. New England Journal of Medicine 2003; 348(25):2526-2534.

Bates DW, Pappius E, Kuperman GJ, Sittig D, Burstin H, Fairchild D et al. Using information systems to measure and improve quality. International Journal of Medical Informatics 1999; 53(2-3):115-124.

Bayegan E, Nytro O, Grimsmo A. Ranking of information in the computerized problem-oriented patient record. Medinfo 2001; 10(Pt 1):594-598.

Bayegan E, Nytro O. A problem-oriented, knowledge-based patient record system. Studies in Health Technology and Informatics, Health Data in Information Society (Volume 90), Proceedings of MIE2002 2002.

Bayegan E, Tu S. The helpful patient record system: problem-oriented and knowledged-based. Proc AMIA Symp 2002; 36-40.

Benjamin SD. The electronic clinical patient record. Practical Procedures and Aesthetic Dentistry 2001; 13(9):744-748.

Bingham A. Computerized patient records benefit physician offices. Healthcare Financial Management 1997; 51(9):68-70.

Birkmeyer CM, Bates DW, Birkmeyer JD. Will electronic order entry reduce health care costs? Effective Clinical Practice 2002; 5(2):67-74.

Blaine GJ, Cox JR, Jost RG. Networks for electronic radiology. Radiologic Clinics of North America 1996; 34(3):505-524.

Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? JAMA 2003; 290(2):259-264.

Bomba D, de Silva A. An Australian case study of patient attitudes towards the use of computerised medical records and unique identifiers. Medinfo 2001; 10(Pt 2):1430-1434.

Branger P, Duisterhout J. Electronic data interchange in medical care: an evaluation study. Proc Annu Symp Comput Appl Med Care 1991; 58-62.

Branger P, van der Wouden, Schudel B, Verboog E, Duisterhout J, van der Lei J et al. Electronic communication between providers of primary and secondary care. BMJ 1992; 305(6861):1068-1070.

Brennan S, Dodds B. The electronic patient record programme: a voyage of discovery. The British Journal of Healthcare Computing and Information Management 1997; 14:16-18.

Brown SH, Hardenbrook S, Herrick L, St.Onge J, Bailey K, Elkin PL. Usability evaluation of the progress note construction set. Proc AMIA Symp 2001; 76-80.

Buffone GJ, Petermann CA, Bobroff RB, Moore DM, Dargahi R, Moreau DR et al. A proposed architecture for ambulatory systems development. Medinfo 1995; 8(Pt 1):363-366.

Burgess B, Wager KA, Lee FW, Glorioso R, Bergstrom L. Clinics go electronic: two stories from the field. Journal of the American Health Informatics Management Association 1999; 70(6):42-46.

Burkle T, Ammenwerth E, Prokosch H, Dudeck J. Evaluation of clinical information systems. What can be evaluated and what cannot? Journal of Evaluation in Clinical Practice 2001; 7(4):373-385.

Bush J. Computers: looking for a good electronic medical record system? Family Practice Management 2002; 9(1):50-51.

Canfield K. Clinical resource auditing and decision support for computerized patient record systems: a mediated architecture approach. Journal of Medical Systems 1994; 18(3):139-150.

Carine F, Parrent N. Improving patient identification data on the patient master index. Health Information Management 1999; 29(1):14-17.

Chin HL, McClure P. Evaluating a comprehensive outpatient clinical information system: A case study and model for system evaluation. Proc Annu Symp Comput Appl Med Care 1995; 717-721.

Chin HL, Krall MA. Successful implementation of a comprehensive computer-based patient record system in Kaiser Permanente Northwest: strategy and experience. Effective Clinical Practice 1998; 1(2):51-60.

Chin HL. Embedding guidelines into direct physician order entry: simple methods, powerful results. Proc AMIA Symp 1999; 221-225.

Chronbach LJ. Designing Evaluations of Educational and Social Programs. San Francisco: Jessey-Bass, 1982.

Cimino JJ, Li J, Mendonca EA, Sungupta S, Patel VL, Kushniruk AW. An evaluation of patient access to their electronic medical records via the world wide web. Proc AMIA Symp 2000; 151-155.

Claflin N. Computerized interdisciplinary assessment. Journal for Healthcare Quality 2000; 22(2):25-33.

Connelly DP. Integrating integrated laboratory information into health care delivery systems. Clinics in Laboratory Medicine 1999; 19(2):277-297.

Cook TD, Campbell DT. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Company, 1979.

Covvey HD. IT capabilities for the realization of the laboratory without walls. Proc AMIA Symp 1996; 613-617.

Creighton C. A literature review on communication between picture archiving and communication systems and radiology information systems and/or hospital information systems. Journal of Digital Imaging 1999; 12(3):138-143.

Cupito M. How to find who. Health Management Technology 1998; 19(6):32-37.

Darbyshire P. User-friendliness of computerized information systems. Computers in Nursing 2000; 18(2):93-99.

Dayhoff RE, Kuzmak PM, Frank SA, Kirin G. Extending the multimedia patient record across the wide area network. Proc AMIA Symp 1996; 653-657.

DeLone W, McLean E. Information systems success: the quest for the dependent variable. Information Systems Research 1992; 3(1):60-95.

DeLone W, McLean E. The DeLone and McLean Model of Information Systems Success: A ten-year update. Journal of Management Information Systems 2003; 19(4):9-30.

Donaldson LJ. From black bag to black box: will computers improve the NHS? BMJ 1996; 312(7043):1371-1372.

Doran B, DePalma JA. Plan to assess the value of computerized documentation system: adaptation for an emergency department. Topics in Emergency Medicine 1996; 18(1):63-73.

Doupnik AM. An overview of electronic document management system product offerings. Topics in Health Information Management 2002; 23(1):62-73.

Drazen E. Why don't we have computer-based patient records? Journal of the American Health Informatics Management Association 1996; 67(6):56-60.

Drazen EL, Little AD. Beyond Cost Benefit: An assessment approach for the 90's. AMIA 1992: 113-17

Drazen E, Waegemann CP. Point counterpoint - computer-based patient record. Healthcare Informatics 1998; 15(5):84-96.

Drazen E. Is this the year of the computer-based patient record? Healthcare Informatics 2001; 18(2):94-98.

Effler P, Ching-Lee M, Bogard A, Ieong MC, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from clinical laboratories. Journal of the American Medical Association 1999; 282(19):1845-1850.

Elson RB, & Connelly DP. Computerized patient records in primary care: Their role in mediating guideline-driven physician behaviour change. Archives of Family Medicine 1995; 4(8):698-705.

Endoh A, Minato K, Komori M, Inoue Y, Nagata S, Takahashi T. Quantitative comparison of human computer interaction for direct prescription entry systems. Medinfo 1995; 8(Pt 2):1101-1105.

Fletcher RD, Dayhoff, R.E., Frank S, Jones R, Wu C et al. The integrated multimedia electronic patient record. 1999.

Fletcher RD, Dayhoff RE, Wu CM, Graves A, Jones RE. Computerized medical records in the Department of Veterans Affairs. Cancer 2001; 91(8 suppl):1603-1606.

Forsythe DE, Buchanan B. Broadening our approach to evaluating medical information systems. Proc Annu Symp Comput Appl Med Care 1991; 8-12.

Francis L, Hebert M. Experiences from Health Information System Implementation Projects Reported in Canada Between 1991 and 1997. Journal of End User Computing 2001; 13(4):17-25.

Freriks G. Identification in healthcare: Is there a place for unique patient identifiers? Is there a place for the master patient index? 2000.

Gadd CS, Penrod LE. Assessing physician attitudes regarding use of an outpatient EMR: A longitudinal, multi-practice study. Proc AMIA Symp 2001; 194-198.

Gadd CS, Friedman CP, Douglas G, Miller DJ. Information resources assessment of a healthcare integrated delivery system. Proc AMIA Symp 1999; 525-529.

Gamm L, Barsukiewiez C, Dansky K, Vasey J. Pre- and post- control model research on end-users' satisfaction with an electronic medical record: preliminary results. Proc AMIA Symp 1998; 225-229.

Gardner J. VA leads the way. Modern Healthcare 1997; 27(48):24.

Gates K. Evaluation of a system for electronic exchange of laboratory information: A pre-implementation study. Master's Thesis, Memorial University Faculty of Medicine, March 2004.

Goddard BL. Termination of a contract to implement an enterprise electronic medical record system. Journal of the American Medical Informatics Association 2000; 7(6):564-568.

Golob R, Quinn J. Goals & roles: integrated delivery systems & the master patient index. Healthcare Informatics 1994; 11(11):68-72.

Grant A, Plante I, Leblanc F. The TEAM methodology for the evaluation of information systems in biomedicine. Computers in Biology and Medicine 2002; 32(3):195-207.

Green CJ, Moehr JR. Performance evaluation frameworks for vertically integrated health care systems: shifting paradigms. Proc AMIA Symp 2000; 315-319.

Greenes RA, Peleg M, Boxwala A, Tu S, Patel V, Shortliffe EH. Sharable computer-based clinical practice guidelines: rationale, obstacles, approaches, and prospects. Medinfo 2001; 10(Pt 1):201-205.

Gritzalis DA. Enhancing security and improving interoperability in healthcare information systems. Medical Informatics (Lond) 1998; 23(4):309-323.

Gustafson DH, Hawkins RP, Boberg EW, Bricker E, Pingree S, & Chan CL. The use and impact of a computer-based support system for people living with AIDS and HIV infection. Proc Annu Symp on Comput Appl Med Care 1994; 604-608.

Gustafson DH, Hawkins R, Boberg E, Pingree S, Serlin RE, Graziano F et al. Impact of a patient-centered, computer-based health information/support system. American Journal of Preventive Medicine 1999; 16(1):1-9.

Hammond WE, Hales JW, Lobach DF, Straube MJ. Integration of a computer-based patient record system into the primary care setting. Computers in Nursing 1997; 152(2 Suppl):S61-S68.

Hanmer L. Criteria for the evaluation of district health information systems. International Journal of Medical Informatics 1999; 56(1-3):161-168.

Hassey A, Gerret D, Wilson A. A survey of validity and utility of electronic patient records in a general practice. BMJ 2001; 322(7299):1401-1405.

Hawkins F. Evaluation of clinical documentation before and after EMR implementation. IT Health Care Strategist 2000; 2(12):8-11.

Hawkins HH, Hawkins RW, Johnson E. A computerized physician order entry system for the promotion of ordering compliance and appropriate test utilization. Journal of Healthcare Information Management 1999; 13(3):63-72.

Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316(7149):1959-1961.

Heathfield H, Hudson P, Kay S, Mackay L, Marley T, Nicholson L et al. Issues in the multi-disciplinary assessment of healthcare information systems. Journal of Information Technology and People 1999; 12(3):253-275.

Heathfield HA, Buchan IE. Current evaluations of information technology in health care are often Inadequate. BMJ 1996; 313(7063):1008-1009.

Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T et al. Evaluating large scale health information systems: from practice towards theory. Proc AMIA Ann Symp 1997; 116-120.

Heathfield HA, Pitty D. Evaluation as a tool to increase knowledge in healthcare informatics. Medinfo 1998; 9(Pt 2):879-883.

Herbst K, Littlejohns P, Rawlinson J, Collinson M, Wyatt JC. Evaluating computerized health information systems: hardware, software, and human-ware: experiences from the Northern Province, South Africa. Journal of Public Health Medicine 1999; 21(3):305-310.

Hersh WR, Patterson PK, Kraemer DF. Telehealth: the need for evaluation redux. Journal AMIA 2002; 9(1):89-91.

Hippisley-Cox J, Pringle M, Cater R, Wynn A, Hammersley V, Coupland C et al. The electronic patient record in primary care - regression or progression? A cross sectional study. British Medical Journal 2003; 326(7404):1439-1443.

Hocking J, Brown G, Malyuk D, Ensom R. Computerized integration of pharmacy and laboratory data: a prototype model. The Canadian Journal of Hospital Pharmacy 1993; 46(5):212-214.

Hripcsak G, Wilcox A. Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. Journal of the American Medical Informatics Association 2002; 9(1):1-15.

Jerant AF, Hill DB. Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? Journal of Family Practice 2000; 49(4):349-357.

Johnson SB, Haug P, Curtis C, Defa T, Davoren B, Kolodner R et al. Where are they now? CPR leaders assess their progress. Journal of AHIMA 2000; 71(8):35-39.

Kaplan B. Initial Impact of a clinical laboratory computer system: themes common to expectations and actualities. Journal of Medical Systems 1987; 11(2/3):137-147.

Kaplan B. An evaluation model for clinical information systems: clinical imaging systems. Medinfo 1995; 8(Pt 2):1087.

Kaplan B. Addressing organizational issues into the evaluation of medical systems. Journal of the American Medical Informatics Association 1997; 4(2):94-101.

Kaplan B. Social Interactionist framework for information systems studies: the 4C's. Proc of the IFIP WG 8 2 and 8 6 Joint Working Conference on Information Systems: Current Issues and Future Changes 1998; 327-339.

Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Towards an informatics research agenda. Journal of the American Medical Informatics Association 2001; 8(3):235-241.

Kaplan B, Lundsgaarde HP. Toward an evaluation of an integrated clinical imaging system: identifying clinical benefits. Methods of Information in Medicine 1996; 35(3):221-229.

Kaplan R, Norton D. The balanced scorecard - measures that drive performance. Harvard Business Review. 1992; 70(1):71-79.

Karmel M. The electronic medical record: good-bye paper charts, hello better patient care. Minnesota Medicine 2002; 85(3): 57-59.

Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework. Computers in Biology and Medicine 2002; 32(3):165-177.

Keshavjee K, Troyan S, VanderMolen D. Measuring the success of electronic medical record implementation using electronic and survey data. Proc AMIA Symp 2001; 309-312.

King JA, Morris LL, Fitz-Gibbon CT. How to Asses Program Implementation. Thousand Oaks, California: Sage Publications, 1987.

Kozyrskyj A, Brown T, Mustard C. Community pharmacist perceptions of a provincial drug utilization database. Canadian Pharmaceutical Journal 1998; 131:24-29.

Kozyrskyj AL, Mustard CA. Validation of an electronic, population-based prescription database. Annals of Pharmacotherapy 1998; 32(11):1152-1157.

Krall MA. Acceptance and performance by clinicians using an ambulatory electronic medical record in an HMO. Proc Annu Symp Comput Appl Med Care 1995; 708-711.

Kuhn KA, Giuse DA. From hospital information systems to health information systems: problems, challenges, perspectives. Methods of Information in Medicine 2001; 40(4):275-287.

Kukafka R, O'Carroll PW, Gerberding JL, Shortliffe EH, Aliferis C, Lumpkin JR et al. Issues and opportunities in public health informatics: a panel discussion. Public Health Management Practice 2001; 7(6):31-42.

Kushniruk A, Patel V, Cimino JJ, Barrows RA. Cognitive evaluation of the user interface and vocabulary of an outpatient information system. Proc AMIA Symp 1996; 22-26.

Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Computers in Biology and Medicine 2002; 32(3):141-149.

Kushniruk AW, Kaufman DR, Patel VL, Levesque Y, Lottin P. Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology. M D Computing 1996; 13(5):406-415.

Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems. Proc AMIA Annu Symp 1997; 218-222.

Kushniruk AW, Patel VL. Cognitive evaluation of decision making process and assessment of information technology in medicine. International Journal of Medical Informatics 1998; 51(2-3):83-90.

Kushniruk AW, Patel VL, Cimino JJ. Evaluation of web-based patient information resources: application in the assessment of a patient clinical information system. Proc AMIA Symp 2000; 443-447.

Kushniruk AW, Patel C, Patel VL, Cimino JJ. 'Televaluation' of clinical information systems: an integrative approach to assessing web-based systems. International Journal of Medical Informatics 2001; 61(1):45-70.

Larrabee JH, Boldreghini S, Elder-Sorrells K, Turner Z, Wender RG, Hart JM et al. Evaluation of documentation before and after implementation of a nursing information system in an acute care hospital. Computers in Nursing 2001; 19(2):56-65.

Lau F. Towards a framework for action research in information system studies. Information Technology and People 1999; 12 (2): 148-175.

Lau F, Hebert M. Experiences from health information system implementation projects reported in Canada between 1991 and 1997. Journal of End User Computing 2001: 13 (4): 17-25.

Lenson CM. Building a successful enterprise master patient index: a case study. Topics in Health Information Management 1998; 19(1):66-71.

Lim P. MediNet: Singapore's nationwide medical network. Annals of the Academy of Medicine, Singapore 1990; 19(5):656-661.

Lincoln MJ, Weir C, Moreshead G, Kolodner R, Williamson J. Creating and evaluating the Department of Veteran Affairs electronic medical record and national clinical lexicon. Proc Annu Symp Comput Appl Med Care 1994; 1047.

Littlejohns P, Cluzeau F. Guidelines for evaluation. Family Practice 2000; 17(Suppl 1):S3-S6.

Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lesson still to be learnt. BMJ 2003; 326:860-863.

Litzelman D, Dittus R, Miller M, Tierney, W. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine 1993; 8(6):311-317.

Liu Z, Sakurai T, Orii T, Iga T, Kaihara S. Evaluations of the prescription order entry system for outpatient clinics by physicians in the 80 university hospitals in Japan. Medical Informatics and the Internet in Medicine 2000; 25(2):123-132.

Lock C. What value do computers provide to NHS hospitals? BMJ 1996; 312(7043):1407-1410.

Lovis C, Payne TH. Extending the VA CPRS electronic patient record order entry system using natural language processing techniques. Proc AMIA Symp 2000; 517-521.

MacDonald D. Pharmacist's Expectations of a Pharmacy Network: A Baseline Evaluation. Master's Thesis. Faculty of Medicine, Memorial University of Newfoundland, March 2004.

Malone PM, Young WW, Malesker MA. Wide-area network connecting a hospital drug informatics center with a university. American Journal of Health System Pharmacy 1998; 55(11):1146-1150.

Marshall PD, Chin HL. The effects of an electronic medical record on patient care: clinician attitudes in a large HMO. Proc AMIA Symp 1998;150-154.

Martin-Baranera M, Planas I, Palau J, Miralles M, Sancho J, Sanz F. Assessing physician's expectations and attitudes toward hospital information systems: The IMASIS experience. M D Computing 1999; 16(1):73-76.

Mast CG, Caruso MA, Gadd CS, Lowe HJ. Evaluation of a filmless radiology pilot - a preliminary report. Proc AMIA Symp 2001; 443-447.

Mathews KA. Evaluation of clinical information systems. Nursing Management 1993; 24(7):104-105.

Mattern WD, Scott S. A fully integrated clinical information system to support management of end-stage renal disease. Dis Manage Health Outcomes 2001; 9(11):619-629.

Meyer J. Action Research. In N Fulop, P Allen, A Clarke, N Black (eds) Studying the Organization and Delivery of Health Services. New York, NY: Routledge, 2001.

Mbananga N, Madale R, Becker P. Evaluation of hospital information system in the Northern Province in South Africa. Report prepared for the Health Systems Trust, 2002, Medical Research Council of South Africa.

McDaniel JG. Simulation studies of a wide area health care network. Proc Annu Symp Comput Appl Med Care 1994; 438-444.

Medical Records Institute, 2003. Overview of the MRI fifth annual survey of EHR trends and usage.

Meyers JS. Electronic medical records: 10 questions I didn't know to ask. American Academy of Family Physicians 2001; 8(3):29-32.

Miller R. Reference standards in evaluating system performance. Journal of the American Medical Informatics Association 2002; 9(1):87-88.

Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980 - 1997. BMJ 2001; 322(7281):279-282.

Moczygemba J, Biedermann S. MPIs (master patient index) in healthcare: current trends and practices. Journal of AHIMA 2000; 71(4):55-60.

Modai I, Sigler M, Kurs R. The computerized lab alert system for patient management in clinical care. Psychiatric Services 1999; 50(7):869-885.

Moehr JR. Evaluation: salvation or nemesis of medical informatics? Computers in Biology and Medicine 2002; 32(3):113-125.

Monane M, Matthias D, Nagle B, Kelly M. Improving prescribing patterns for the elderly through online drug utilization review intervention. JAMA 1998; 280(14):1249-1252.

Murff HJ, Kannry J. Physician satisfaction with two order-entry systems. Journal of the American Medical Informatics Association 2001; 8(5):499-509.

Nazi KM. The journey to e-health: VA healthcare network upstate New York (VISN 2). Journal of Medical Systems 2003; 27(1):35-45.

Neame RL, Olson M. Measures implemented to project personal privacy for an on-line national patient index: a case study. Topics in Health Information Management 1996; 17(2):18-25.

Neville D, Keough M, Barron M, MacDonald D, Gates K, Tucker S, Cotton S, Farrell G, Hoekman T, Bornstein S, O'Reilly S. Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Across Canada. March 2004.

Neville D, Gates G, Tucker S, Keough M, MacDonald D, Barron M, Cotton S, Farrell G, Hoekman T, Bornstein S, O'Reilly S. Towards an Evaluation Framework For Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Program Reports. February 2004.

NHS Information Authority, 2001. PROBE: Project review and objective evaluation for electronic patient and health record projects. Prepared by the UK Institute of Health Informatics for ERDIP.

NHS Information Authority, 2001. Evaluation of electronic patient and health record projects. Prepared by the UK Institute of Health Informatics for the NHS Information Authority.

NHS Information Authority, March 2003. Electronic Record Development and Implementation Programme update. www.nhsia.nhs.uk/erdip/pages/publications/ERDIPUpdateJan03_5.pdf, Accessed September 2003.

Nielsen PE, Thomson BA, Jackson RB, Kosman K, Kiley KC. Standard obstetric record charting system: evaluation of a new electronic medical record. Obstetrics and Gynecology 2000; 96(6):1003-1008.

Ohmann C, Boy O, Yang Q. A systematic approach to the assessment of user satisfaction with health care systems: constructs, models and instruments. Studies in Health technology and Informatics 1997; 43(Pt B):781-785.

Ornstein SM, Jenkins RG, Edsall RL. Computerized patient systems: a survey of 28 vendors. Family Practice Management 1997; 4(10):45-59.

Ornstein SM. Electronic medical records in family practice: the time is now. The Journal of Family Practice 1997; 44(1):45-48.

Ornstein SM, Jenkins RG, MacFarlane LL, Glaser A, Snyder K, Gundrum T. Electronic medical records as tools for quality improvement in ambulatory practice: theory and a case study. Topics in Health Information Management 1998; 19(2):35-43.

Ornstein SM, MacFarlane LL, Jenkins RG, Pan Q, Wager KA. Medication cost information in a computer-based patient record system. Archives of Family Medicine 1999; 8(2):118-121.

Ornstein SM, Garr DR, Jenkins RG, Musham C, Hamadeh G, Lancaster C. Implementation and evaluation of a computer-based preventive services system. Family Medicine 1995; 27(4):260-266.

Osada M, Nishihara E. Implementation and evaluation of workflow based on hospital information system/radiology information system/picture archiving and communication systems. Journal of Digital Imaging 1999; 12(2 (Suppl 1)):103-105.

Ostbye T, Moen A, Erikssen G, Hurlen P. Introducing a module for laboratory test order entry and reporting of results at a hospital ward: an evaluation study using a multi-method approach. Journal of Medical Systems 1997; 21(2):107-117.

Patel VL, Kaufman DR, Allen VG, Shortliffe EH, Cimino JJ, Greenes RA. Toward a framework for computer-mediated collaborative design in medical informatics. Methods of Information in Medicine 1999; 38(3):158-176.

Patel VL, Kushniruk AW, Yang S, Yale J. Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. Journal of the American Medical Informatics Association 2000; 7(6):569-585.

Patel VL, Arocha JF, Kushniruk AW. Patients' and physicians' understanding of health and biomedical concepts: relationship of the design of EMR systems. Journal of Biomedical Informatics 2002; 35(1):8-16.

Poon EG, Kuperman GJ, Fiskio J, Bates DW. Real-time notification of laboratory data requested by users through alphanumeric pagers. Journal of the American Medical Informatics Association 2002; 9(3):217-222.

Powsner SM, Wyatt JC, Wright P. Opportunities for and challenges of computerisation. Lancet 1998; 352(9140):1617-1622.

Protti D, Peel V. Critical success factors for evolving a hospital toward an electronic patient record system: a case study of two different sites. Journal of Healthcare Information Management 1998; 12(4):29-38.

Protti D. A proposal to use a balanced scorecard to evaluate information for health: an information strategy for the modern NHS (1998 - 2005). Computers in Biology and Medicine 2002; 32(3):221-236.

Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthcare Management Forum 2002; 15(2):33-35.

Protti D. The power of principles and premises: using them to help define the EHR. Healthcare Management Forum 2002; 15(3):46-48.

Raschke RA, Gollihare B, Wunderlich TA, Guidry JR, Leibowitz AI, Peirce JC et al. A computer alert system to prevent injury from adverse drug events. Journal of the American Medical Informatics Association 1998; 280(15):1317-1320.

Rehm S, Kraft S. Electronic medical records: the FPM vendor survey. Family Practice Management 2001; 8(1):45-54.

Rigby M, Robins S. Practical success of an electronic patient record system in community care - a manifestation of the vision and discussion of the issues. International Journal of Bio-Medical Computing 1996; 42:117-122.

Rivkin S. Opportunities and challenges of electronic physician prescribing technology. Medical Interface 1997; 10(8):77-83.

Robbins J. The Northern Territory Client Master Index. Health Information Management 1999; 29(1): 35-37.

Robert G, Gabby J, Stevens A. Which are the best information sources for identifying emerging health care technologies? An international Delphi survey. International Journal of Technology Assessment in Health Care 1998; 14(4): 636-643.

Rogers E. Diffusion of Innovation (3rd edition). New York: The Free Press, 1993

Rossi PH, Freeman HE. Evaluation: A Systematic Approach 5. Newbury Park, California: Sage Publications, 1993.

Rothschild J, Lee T, Bae T, Bates D. Clinician use of a palmtop drug reference guide. Journal American Medical Informatics Association 2002; 9(3):223-229.

Sabo D. Clinical information systems: a gateway to the 21st century. Nursing Administration Quarterly 1997; 21(3):68-75.

Sado AS. Electronic medical records in the intensive care unit. Critical Care Clinics 1999; 15(3):499-522.

Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet 1995; 346(8971):341-346.

Safran C, Jones PC, Rind D, Bush B, Cytryn KN, Patel VL. Electronic communication and collaboration in a health care practice. Artificial Intelligence in Medicine 1998; 12(2):137-151.

Sailors RM, East TD. Clinical informatics: 2000 and beyond. Proc-AMIA-Symp 1999; 609-613.

Sarr MG. The electronic environment: how has it, how will it, and how should it affect us? Journal of Gastrointestinal Surgery 2001; 5(6):572-582.

Schiff GD, Rucker TD. Beyond structure-process-outcome: Donabedian's seven pillars and eleven buttresses of quality. Journal on Quality Improvement 2003; 27(3):169-174.

Schiff GD, Klass D, Peterson J, Shah G, Bates DW. Linking laboratory and Pharmacy: opportunities for reducing errors and improving care. Archives of Internal Medicine 2003; 163(8):893-900.

Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA 1998; 279(13):1024-1029.

Schuerenberg K. Electronic records find long-term use: Remote access to patient records enables Denver physicians to provide better services to long-tem care patients. Health Data management February 2003.

Shortliffe EH. The evolution of health-care records in the era of the internet. Medinfo 1998; 9(Pt 1):8-14.

Shortliffe EH. Clinical information systems in the era of managed care. Transactions of the American Clinical and Climatological Association 1993; 105:203-215.

Shortliffe EH, Bleich HL, Caine CG, Masys DR, Simborg DW. The federal role in the health information infrastructure: a debate of the pros and cons of government intervention. Journal of the American Medical Informatics Association 1996; 3(4):249-257.

Shortliffe EH, Barnett GO, Cimino JJ, Greenes RA, Huff SM, Patel VL. Collaborative medical informatics research using the internet and the world wide web. Proc AMIA Annu Symp 1996; 125-129.

Shortliffe EH. The next generation internet and health care: a civics lesson for the informatics community. Proc AMIA Symp 1998; 8-14.

Shortliffe EH, Wiederhold G, Fagan LM. Medical Informatics: Computer Applications in Health Care. Reading, Massachusetts: Addison-Wesley Publishing Company, 1990.

Sittig DF, Kuperman GJ, Fiskio J. Evaluating physician satisfaction regarding user interactions with an electronic medical record system. Proc AMIA Symp 1999; 400-404.

Smith PD. Implementing an EMR system: one clinic's experience. Family Practice Management 2003; 10(5):37-42.

Snaedal J. The ethics of health sector databases. EHealth International 2002; 1(1):6-8.

Soliman F, Soar J. Physician clinical communication systems--an Australian perspective. Journal of Medical Systems 1997; 21(2):99-106.

Soper WD. Why I Love my EMR. Family Practice Management 2002; 9(9):35-38.

Stevens CA, Morris A, Sargent G. Internet health information sources. The Electronic Library 1996; 14(2):135-147.

Talmon J, Enning J, Castaneda G, Eurlings F, Hoyer D, Nykanen P et al. The VATAM guidelines. International Journal of Medical Informatics 1999; 56(1-3):107-115.

Tan LT. National patient master index in Singapore. International Journal of Bio-Medical Computing 1995; 40(2):89-93.

Tang P, LaRosa M, Gorden S. Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions. Journal of the American Medical Informatics Association 1999; 6(3):245-251.

Tang PC, Fafchamps D, Shortliffe EH. Traditional medical records as a source of clinical data in outpatient setting. Proc Annu Symp Comput Appl Med Care 1994; 575-579.

Teich JM. Clinical information systems for integrated healthcare networks. Proc AMIA Symp 1998; 19-28.

Thiru K, Hassey A, Sullivan F. Systematic review of scope and quality of electronic patient record data in primary care. BMJ 2003; 326(7398):1070-1075.

Treweek S, Flottorp S. Using electronic medical records to evaluate healthcare interventions. Health Informatics Journal 2001; 7(2):96-102.

Twair AA, Torreggiani WC, Mahmud SM, Ramesh N, Hogan B. Significant savings in radiologic report turnaround time after implementation of a complete picture archiving and communication system (PACS). Journal of Digital Imaging 2000; 13(4):175-177.

van der Loo RP, van Gennip EMSJ, Bakker AR, Hasman A, Rutten FFH. Evaluation of automated information systems in health care: an approach to classifying evaluative studies. Computer Methods and Programs in Biomedicine 1995; 48(1-2):45-52.

Van der Meijden MJ, Tange HJ, Hasman TA. Determinants of success of inpatient clinical information systems: a literature review. Journal of the American Medical Informatics Association 2003; 20(3):235-243.

Villella R. Lab connections: building a case for a web-based lab results reporting system. Healthcare Informatics 2000; 17(10):119-120.

Waegemann CP. The five levels of the ultimate electronic health record. Healthcare Informatics 1995; 12(11):26-35.

Wager KA, Heda S, Austin CJ. Developing a health information network within an integrated delivery system: a case study. Topics in Health Information Management 1997; 17(3):20-31.

Wager KA, Ornstein SM, Jenkins RG. Perceived value of computer-based patient records among clinical users. M D Computing 1997; 14(5):334-340.

Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. Impact of an Electronic Medical Record System on Community-based Primary Care Practices. The Journal of the American Board of Family Practice 2000; 13(5):338-348.

Walker A. South Australia: best practice guidelines for patient master index maintenance. Health Information Management 1999; 29(1):43-45.

Wang D, Peleg M, Tu S, Shortliffe EH, Greenes RA. Representation of clinical practice guidelines for computer-based implementations. Medinfo 2001; 10(Pt 1):285-289.

Weir C, Lincoln MJ, Roscoe D, Turner C, Moreshead G. Dimensions associated with successful implementation of a hospital based integrated order entry system. Proc Annu Symp on Comput Appl Med Care 1994; 653-657.

Weir CR. Linking information needs with evaluation: the role of task identification. Proc AMIA Symp 19998; 310-314.

Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods of Information in Medicine 2003; 42(1):61-67.

Wenzel GR. Creating an interactive interdisciplinary electronic assessment. Computers, Informatics, Nursing 2002; 20(6):251-260.

Wills S. The 21st century laboratory: information technology and health care. Clinical Leadership Management Review 2000; 14(6):289-291.

Wolfe H. Cost-benefit of laboratory computer systems. Journal of Medical Systems 1986; 10(1):1-9.

Wu SC, Smith JW, Swan JE. Pilot study on the effects of a computer-based medical image system. Proc AMIA Symp 1996; 674-678.

Appendix A: Evaluation of Complex Health Information Systems

Sample Evaluation Questions

Littlejohns et al (2003); Evaluation of an Integrated Hospital Information System in South Africa: 10 evaluation projects:

1. Are training, change management and support optimal?
2. Is the reliability of the system (including peripherals, network, hardware and software) optimal?
3. Assessing the project management
4. Does the system improve the communication of patient information between healthcare facilities?
5. Is data protection adequate?
6. Assessing the quality and actual use of decision-making information to support clinicians, hospital management, provincial health executives and the public.
7. Are patient administration processes more standardized and efficient?
8. Has revenue collection improved?
9. Is information being used for audit or research?
10. Are costs per unit service reduced?

Healthfield et al (1997) Evaluating Large Scale Health Information Systems: From Practice Toward Theory: Evaluation of 2 NHS projects, an Electronic Patient Record Project and an Integrated Clinical Workstation (ICWS); 6 major evaluation questions identified:

1. What is the impact of the technology on clinical management at 3 levels: individual patient care, management of services, and resource management?
2. What is the impact on the roles, the organization of work and work satisfaction of staff? What is the experience of working and living at the implementation sites?
3. Can the costs and benefits of such developments/technologies be valued?
4. Patient record systems and technologies: How useful and useable are they?
5. What is the relationship between electronic and paper records for the EPR/ICWS sites in respect of: availability of data, integrity, compliance with standards, volume of paper generated and reduction in clerical activity?
6. What is the relationship between the technology and the general management of the trust?

Anderson JG, Aydin CE, Jay SJ. Evaluating Health Care Information Systems: Methods and Applications. Thousand Oaks, California: Sage Publications, 1994.

Adapted from Table 1.1, page 13-14.

Evaluation Questions and Suggested Methods

Evaluation Question	Suggested Methods
1. Does the system work as designed?	Qualitative (interviews, observation, documents) Survey Laboratory/quasi-experiment Cost-benefit analysis Clinical information processing scenarios
2. Is the system used as anticipated?	Qualitative (interviews, observation, documents) Survey Quasi-experiment Cohort/time series study Post-intervention study
3. Does the system produce the desired results?	Qualitative (interviews, observation, documents) Survey Laboratory/quasi-experiment/simulation Cohort/time series study Post-intervention study Cost-benefit analysis
4. Does the system work better than the procedures it replaced?	Qualitative (interviews, observation, documents) Survey Quasi-experiment Simulation Cohort/time series study Cost-benefit analysis
5. Is the system cost-effective?	Cost-benefit analysis
6. How well have individuals been trained to use the system?	Qualitative (interviews, observation, documents) Survey Cohort/time series study
7. What are the anticipated long-term impacts on how departments interact? (Con't)	Cohort/time series study Network analysis
8. What are the long-term effects on delivery of medical care?	Qualitative (interviews, observation, documents) Survey Quasi-experiment Cost-benefit
9. Will the system have an impact on control in the organization?	Qualitative (interviews, observation, documents) Survey Network analysis Cost-benefit analysis
10. To what extent do impacts depend on practice setting?	Qualitative (interviews, observation, documents) Survey Quasi-experiment

Amatayakul M. Critical Success Factors. Focus on Evaluating CPR Systems. Health Care Management Technology, May 2000, 14-17

Adapted from section on Steps to Evaluating CPR Systems, p.15.

(1) Does the system support the mission of the organization and provide for continuity of care?

(2) Does the system enable the business goals of the organization?

(3) Does the CPR contribute to improved patient care and not just administrative efficiencies?

(4) Are end-users engaged in evaluating present systems and creating a vision for the CPR? Do the end-users have an easy way to communicate issues and ideas to management and information systems services?

(5) Is there a vision of an information infrastructure to support continuous clinical service?

(6) Have knowledge requirements been assessed and new approaches to data management taken?

(7) Do present systems support the data content requirements of the new transaction standards?

(8) What planning has been done to support additional uniform data standards?

(9) Do users see the value in using the system?

(10) Does management understand the nature and timeframe for return on investment?

(11) Have resources been assigned to continuously monitor benefits realization?

(12) Have the goals of service effectiveness, operational efficiency, and informational empowerment been achieved?

NHS PROBE 2001. Evaluation Framework for NHS Electronic Patient Record and Electronic Health Record. Evaluation Questions posed in 3 time frames and along 5 dimensions: strategy, operational, technical, financial and human.

Sample questions extracted from Appendix C: Table Showing Suggested Focus of Evaluations, p 21-27.

Timing Of Evaluation	Suggested Focus	Sample Questions
Pre-Implementation Review	Strategic	Are organizations ready for EPR/EHR implementation? Are stakeholders ready? Does the investment 'Fit' with other strategies: LIS HimP Clinical Governance?
	Operational	Are business processes being reviewed in preparation for EPR/EHR?
	Financial	Is the investment affordable? Have the risks been assessed and are they affordable?
	Human	Are individuals and teams ready for EPR? Are the training and support programmes in place?
	Technical	Are suitable project management structures in place? Is the IT infrastructure capable of support EPR? Are the necessary security policies in place? Are contract management teams in place?
Implementation Review This is most important early in the project lifecycle, as technical problems are likely to alienate users. Therefore, it should be a strong focus of post-implementation. Reviews rather than Post Operational Evaluations.
	Technical	Has the IM&T Strategy been effective? Has it delivered the information systems that meet the information requirements of users? Has the EPH/EHR project deployed technology in the most effective way?Is the system flexible enough to cope with changing requirements? Is the EPR/EHR secure? Was the EPR/EHR procurement defensible? Are the project management, implementation and contract management structures effective? Is there efficient use of IM&T resources? Could IM&T resources be deployed more efficiently? Has testing been effective? Is there evidence of faults that could have been eliminated by further testing? Are the IM&T support mechanisms effective?
Implementation Review Or Operational Evaluation	Financial	The business case. Was the business case realistic? Have the quantifiable costs and benefits been realized? Were the risks understood? Are they being managed? How was the EPR/EHR affected the financial position of the stakeholder organizations and the community as a whole? How has the financial context changed through the life of EPR/EHR? How has the EPR/EHR development affected benchmarked positions (reference costs, cost per bed day, cost per FCE etc)?
Implementation Review Or Operational Evaluation	Strategic (Likely to be more suited to Post Operational Evaluation because of the time required to assess the degree of strategic change)	Has EPR/EHR development delivered the strategic change predicted in LIS and the business case? Has EPR/EHR contributed positively to the implementation of the Himp and Clinical Governance? Has EPR/EHR enabled the stakeholders to address changing strategic priorities (such as the NHS Plan) since the EPR/EHRs' inception? What are the strategic dependencies between the EPR/EHR project and other LIS projects? How has the EPR/EHR development contributed to other dependent projects within the LIS?
	Operational	Is there evidence of clinical benefits in areas such as: The avoidance of errors and adverse effects Improved patient outcomes Benefits from faster interventions and improved communications. Electronic prescribing and formulary management. Enhanced risk management Discharge planning Clinic scheduling Has the introduction of care pathways and protocols delivered benefits for patients and clinicians? Has the clinical time been saved through the introduction of more efficient administration and faster access to records? Has there been a reduction in paper, and reduced use of the paper records?Do clinicians use the EPR/EHR? Does the EPR/EHR deliver the information required to support clinical governance and clinical audit? Has data quality improved?
Operational Evaluation	Human Human factors are important early in the implementation process, and to ensure that the system is used as intended and also that the user interface is acceptable Stakeholder groups: Clinical users Patients Careers Managers	What has been the impact of the project and the EPR/EHR on individuals and the way they provide or receive care? Has EPR/EHR delivered the benefits that they personally expected? Has the EPR/EHR improved the patient ad carer experience of the NHS? Is the system or information used as often as they expected? Have person-to-person communications improved? Specifically, have patient-clinician and clinician - clinician communications improved? Have business processes changed? Is there a learning and personal development culture? Are individuals supported to enable them to optimize their use of EPR/EHR? Are staff aware of their responsibilities within the benefits realization plan? Are staff able to use EPR/EHR to react to changing internal and external priorities and demands?

Kaplan B et al. Towards An Informatics Research Agenda: Key People and Organizational Issues. Journal of the American Medical Infomatics Association, 2001, 8(3) 235-241.

Excerpts From Table 1, Page 236
Research Agenda Model: Key People and Organizational Issues -- Sample Questions at Different Levels

Social Science Discipline Level (Low Aggregation To High Aggregation)

Individual /Cognitive Psychology	Workgroup /Social Psychology	Organization /Sociology	Culture /Cultural Anthropological
What affects information-seeking behaviour and how information is used? How do different information sources influence health care decisions and outcomes? What information and information designs are effective for different individuals? How can multiple sources and formats of individual data be integrated or aggregated?	How do rapid communication and changing roles of health care providers affect professional relationships? How does widespread availability of health information affect the patient role and patient decision-making? How do auditing and monitoring of care affect professional identity and cohesion? How does the potential for distributed information and workflow affect how processes are organized and carried out?	How does widespread availability of health information affect relationships and roles between providers and patients or consumers? What would constitute an acceptable lifetime health record for each individual? Acceptable to whom? How do linkages through IT affect organizational identity and integrity?	How should information be tailored to suit individuals from different cultural groups? How does one's culture affect one's use of IT? How can data e integrated and aggregated across organizations to obtain indicators and guidelines for improving care? How will clinicians and patients at an institution react to global indicators and guidelines?

Appendix B: Evaluation of Complex Health Information Systems

Sample Indicators

van der Loo et al (1995) Evaluation of automated information systems in health care: an approach to classifying evaluative studies.

Effect measures in reviewed studies included:

• (1) costs
• (2) time changes for personnel (for example faster diagnosing)
• (3) time changes for personnel (for example waiiting times)
• (4) time changes in logistical processes (for example, retrieval of images)
• (5) database use
• (6) use of medical tests
• (7) the performance of the user (for example compliance with on-line documentation requirements)
• (8) performance of the system (for example the number of correctly indicated patients with hypertension)
• (9) patient outcomes ( length of stay, quality of life)
• (10) job satisfaction
• (11) patient satisfaction

Bates et al (1999) The Impact of Computerized Physician Order Entry on Medication Error Prevention

Types of medication errors measured included:

• (1) dose errors
• (2) frequency errors
• (3) error routes
• (4) substitution errors
• (5) allergies
• (6) non-missed dose medication error (major variable of interest)

Darbyshire P. User-Friendliness of Computerized Information Systems. Computers in Nursing. March/Arpil 2000. 93-99.

Indicators of user-friendliness for nurse clinicians using a computerized information system included:

• (1) ease of access (uncomplicated password function)
• (2) availability of terminals
• (3) clarity and navigatability of computer screens
• (4) use of intuitive icons and graphics
• (5) availability of on-line help
• (6) availability of on-screen prompts and reminders
• (7) ease of printing clinical documentation when required
• (8) speed and responsiveness of the system

Keshavjee K, Troyen S, Holbrook AM, VanderMolen D. Measuring the success of electronic medical record implementation using electronic and survey data.

Meaures at pre-implementation, and 6 and 18 months post implementation

Staff related administrative measures:

• (1) time taken for chart pulls (for day visits, filing lab results and consult notes)
• (2) time spent in writing in the chart

Physician related clinical measures:

• (1) time spent writing in the chart
• (2) time spent writing prescriptions
• (3) time to review consult notes
• (4) perception of length of day worked (number of hours/day worked)
• (5) perception of the quality of the chart
• (6) number of patients seen per day
• (7) perception of volume of work

Krall MA. Acceptance and Performance by Clinicians Using an Ambulatory Electronic Medical Record in a HMO. AMIA, 1995, 708-711.

Project was evaluated using pre-implementation and 2 and 4-6 months post implementation user surveys, management engineering studies and monitoring of clinician productivity.

Pre-implementation Survey:

• (1) pre-existing computer experience and attitudes
• (2) preferred learning methods

Post Implementation Survey

• (1) perceived efficiences of new system compared to the previous one
• (2) suggestions re system improvements

Management Engjneering Studies pre and post implementation:

• (1) clinician time to complete tasks (chart review, exam and treat, orders and diagnosis, and charting)

Clinical productivity measures:

• (1) visits per hour per clinician, pre and post implementation

Protti D, Peel V. Critical Success Factors for Evolving a Hospital Toward an Electronic Patient Record System: A Case Study of Two Different Sites. Journal of Healthcare Information Management; 1998: 12(4): 29-37

Critical Success Factors include:

• (1) a clinical, not just medical focus
• (2) routine clinical use of the systems
• (3) executive leadership and sound management
• (4) nuturing of a new culture
• (5) stablility and a mature management-clinical partnership

Tangible and iIntangible bennefits include:

• (1) ease of location of clinical information needed to care for patients
• (2) reduction in unnecessary or duplicate testing
• (3) waiting times for treatment reduced
• (4) improved communication between disciplines

Delone WH, McLean ER. Informations Systems Success: The Quest for the Dependent Variable. Information Systems Research. 1992, 3(1), 60-95

Empirical Measures of Information Quality (summarized from the review of 9 studies presented in Table 2, page 67)

• (1) Accuracy
• (2) Timliness
• (3) Reliability
• (4) Completeness
• (5) Format
• (6) Relevance to decisions
• (7) Understandability

Delone WH, McLean ER. The Delone and McLean Model of information System Success: A Ten Year Update. Journal of Management Information Systems. 2003: 19(4) 9-30.

E-Commerce Success Metrics (Table 1, page 26)

Systems quality:

• (1) Adaptability
• (2) Availability
• (3) Reliability
• (4) Response time
• (5) Usability

User satistifaction:

• (1) Repeat purchases
• (2) Repeat visits
• (3) User surveys

Information quality:

• (1) Completeness
• (2) Ease of understanding
• (3) Personalization
• (4) Relevance
• (5) Security

Net benefits:

• (1) Cost savings
• (2) Expanded markets
• (3) Incremental additional sales
• (4) Reduced search costs
• (5) Time savings

Service quality:

• (1) Assurance
• (2) Empathy
• (3) Responsiveness

Use:

• (1) Nature of use
• (2) Navigation patterns
• (3) Number of site visits
• (4) Number of transactions executed

Kaplan B, Lundsgaarde HP. Toward an Evaluation of an Integrated Clinical Imaging System: Identifying Clinical Benefits. Methods of Information in Medicine. 1996, 35, 221-9.

Benefits identified by phyicians as a result of introducing a PACS system include:

Patient Care Benefits:

• (1) improved clinical communication and decision making
• (2) care becomes more patient-based
• (3) reduction in the number o precedures and patient risk
• (4) improvement in medical record keeping

Educational Benefits:

• (1) improved communication between teaching physicians and residents
• (2) provision of "real" patient learning experience (access to more complete patient information)
• (3) improved student supervision

Productivity and Cost Reduction Benefits

• () elimination of time gaps between the production of images and written reports
• (2) convenience in terms of writing, storing and reviewing notes with images during or immedicately after a procedure
• (3) continuous availability of images with patient records

Bates D, Pappius E et al. Using Information Systems to Measure and Improve Quality. International Journal of Medical Informatics. 1999, 53, 115-124.

Measures of quality that were extracted from a hospital information system include:

• (1) use of unnecessary laboratory testing
• (2) speed of report of abnormalities in results to providers
• (3) prevention and detection of adverse drug events
• (4) clinical department's selection of measures for efficiency, critical variances and sentinel events

Chin HL, McLure P. Evaluating a Comprehensive Outpatient Clinical Information System: A Case Study and Model for System Evaluation. AMIA, 1995, 717-721.

Indicators of a Successsful System Implementation, as excepted from Table 1, page 718:

• (1) high user acceptance
• (2) productivity of clinicians
• (3) high patient acceptance
• (4) high use of the system
• (5) technical adequacy (good perfornance, stability of the product, no loss of data)
• (6) flexible, modifiable and expandable system

Gadd CS, Penrod LE. Assessing Physician Attitudes Regarding Use of an Outpatient EMR: A Longitudinal, Multi-Practice Study. AMIA, 2001, 194-198.

Physician concerns, measured pre and post implementation of an EMR, as excepted from Table 2, page 197

• (1) time required to enter orders for tests or medications
• (2) time required for documentation, such as progress notes
• (3) rapport established between physican and patient during the visit
• (4) patient privacy
• (5) physician autonomy
• (6) patient's satisfaction with the quality of care they receive
• (7) the overall quality of health care that you give your patients

Abdelhak M. Health Information Management of a Strategic Resource. Philadelphia, Pa: W.B. Saunders Company, as summarized in F Hawkins. Evaluation of Clinical Documentation Before and After EMR Implementation. IT Health Care Strategist. 200, 2(12). 8-11.

General Medical Records Documentation requirements

• Name/medical record number on each page
• Summary sheet/problem list complete
• Current and past medications entered on the medication sheet
• All entries dated
• Presence or absence of allergies is documented in a prominent and uniform location
• Every patient visit includes documentation of:
  • chief complaint/purpose of visit
  • history and physical consistent with chief complaint
  • diagnosis or impression
  • treatment
  • patient disposition, referral, instructions; and
  • signature of practitioner
• Immunization record includes:
  • date
  • vaccine manufacturer name and lot number
  • test results filed in sequential order
  • all test results initiated and dated by practitioner and
  • informed consent present
• Pre-anesthesia evaluation:
  • is recorded prior to surgery
  • includes review of patient's medical history
  • includes previous anesthetic experiences
  • includes current medications and
  • includes date and signature of anesthesiologist
• Anesthesia record includes documentation of:
  • time-based monitoring of vital signs and level of consciousness
  • status of surgical dressing
  • status of tubes; and
  • date and signature of discharging practitioner
• The operative report includes documentation of:
  • preoperative diagnosis
  • postoperative diagnosis
  • findings
  • technique used
  • specimens removed
  • primary surgeon and assistants
  • date and signature of surgeon; and
  • documentation of postoperative instructions; and
• The pathology report is filed in the medical record

Birkmeyer CM, Bates DW, Birkmeyer JD. Will Electronic Order Entry Reduce Health Care Costs? Effective Clinical Practice. March/April 2002, 5( 2), 67-74.

Potential pre and post implementation measures of cost include:

• (1) patient bed-days per year
• (2) costs of adverse drug events
• (3) costs of serious medication errors
• (4) costs of unnecessary tests
• (5) use of lower cost medications as substitutes

Wager KA, Ornstein SM, Jenkins RG. Perceived Value of Computer-Based Patient Records Among Clinician Users. M.D. Computing. 344-340.

Perceived advantages of CPRs identified by 44 physician practices using the same CPR system:

• (1) improved documentation for patient care
• (2) quality of the patient record
• (3) access to the patient record
• (4) improved documentation for prevention services
• (5) improved documentation for quality improvement
• (6) ease of use
• (7) security of patient record
• (8) improved efficiency
• (9) adminstrative cost savings

Doran B, DePalma JA. Plan to Assess the Value of a Computerized Documentation System: Adaption for an Emergency Department. Top Emerg Med 1996, 18(1), 63-73.

Pre-Implementation assessemt of of ER documentation, as extracted from Table 1, page 65

Variable	Indicator	How Measured
Accuracy	Injection sites Intake and ouput Vital signs	Chart audit/checklist: Site Abbreviation Chart audit/checklist: Totals Correct math Percent missing data Check original order and chart for transcription errors
Quality	Standard by diagnosis Critical elements	Compare charting with checklist of critical charting elements for common diagnoses. Percent items charted Consistency among nurses Compare checklist with critical charting elements from standard of care: Percent items charted Consistency among nurses
Safety	Types of medication errors Medication transcription errors	Risk management data quarterly reports -- totals Check original order and medication record for transcription errors.
Physician satisfaction	Satisfaction with critical elements of documentation; Availability of data	Create survey with items for physicians' response: Laboratory data Intake and output Medication data Weights Nursing assessment Summary data
	Satisfaction with critical elements of documentation: Availability of chart in general Satisfaction with critical elements of documentation: Accuracy/completeness	Availability of chart Location of chart on unit Intake and output New information Nursing assessment information in general

Computerized Provider Order Entry Systems. Evaluation. Health Devices. 2001, 30 (9-10), 323-359

Evaluation Criteria for CPOE include:
User Interface:	order entry order processing order output
Patient Safeguards:	basic safety alerts and safeguards additional safety features
Order Monitoring
Knowledge Base
Data Management:	System access Integration with Ancillary Information Systems Data Access Network Degradation Management
(Note: this publication also contains very detailed indicators for the above listed criteria)

Sittig DF, Kuperman GJ, Fiskio J. Evaluating Physician Satisfaction Regarding User Interactions With an Electronic Medical Record System. AMIA, 1999, 400-404

The authors used the QUIS (Questionnaire For User Interaction Satisfaction) developed by Chin et al at the University of Maryland to measure user interaction satisfaction with an EMR in routine clincial use. The short form of the QUIS is divided into 5 sections of 4 questions each. Variables assessed, using a 9 point scale, include:

(1) overall user reactions

(2) screen design and layout

(3) teminology and system messages

(4) learning

(5) system capabilities

Mitchell E, Sullivan F. A Descriptive Feast But Evaluative Famine: Sytematic Review of Published Articles on Primary Care Computing during 1980-1997. BMJ 2001: 322:279-282.

61 studies exmained the effects of computers on practioners' performance, 17 evaluated their impact on patient outcome and 20 studied practioners' or patients' attitudes

Practioners' Performance:

(1) immunization rates

(2) performance of preventive tasks

(3) content of consultation

(4) disease management

(5) prescribing

Patient Outcomes:

(1) changes in diastolic blood pressure

(2) impact on anticoagulant management

(3) service utlilization

(4) location of service utilization (hospital versus community)

(5) rates of non attendance at scheduled appointments

(6) patient satisfaction

Practioners's and Patients' Attitudes:

(1) attitudes about use of computers in practice

(2) perceptions regarding record accuracy and accessibility

(3) privacy

(4) doctor-patient relationship

(5) cost

(6) time

(7) training

Glossary of Terms

Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects

See also:

• Toward an Evaluation Framework for Electronic Health Records Initiatives: A Proposal For an Evaluation Framework
• Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada

Prepared as a component of the study:

Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects

Doreen Neville, ScD
Stephen O'Reilly, MBA
Kayla Gates, BSc
Shelia Tucker, MLS
Michael Barron, MBA
Sandra Cotton. B.A.
Gerard Farrell, M.D.
Theodore Hoeknman, PhD
Stephen Bornstein, PhD
Montgomery Keough, BSc
Donald MacDonald, B. A.

Executive Summary

Background

An electronic health record (EHR) provides each individual in Canada with a secure and private lifetime record of their key health history and care within the health system. The record is available electronically to authorized health care providers and the individual anywhere, anytime, in support of high quality care. Recognizing the importance of the EHR in improving the quality and efficiency of health care in Canada, the Federal government established Canada Health Infoway (Infoway) to support and accelerate the development and adoption of interoperable electronic health records solutions across Canada. Infoway's strategic plan for achieving this goal involves each region taking a lead role in at least one major initiative with elements and results which can be transferred to other Canadian jurisdictions. The Newfoundland and Labrador Centre for Health Information (NLCHI) has taken a lead role in the Infoway priority area of client registries.

Infoway and NLCHI have identified four core components of an EHR: (1) a unique personal identifier/client registry; (2) pharmacy network; (3) laboratory network; and (4) diagnostic services network. This research project "Towards an Evaluation Framework for Electronic Health Records Initiatives" was designed to complement the Infoway/NLCHI work and to advance knowledge in the field of EHR through the development of an evaluation framework which can be used to assess projects/initiatives across the four components of an EHR.

Traditionally, evaluations of components of medical or health information systems have focused on (1) technical and systems features that affect systems use; (2) cost- benefit analysis; (3) user acceptance and (4) patient outcomes. More recently, evaluation approaches have addressed the context and processes that contribute to the outcomes of the project, and have incorporated aspects of change and innovation diffusion theories into the evaluation model. While randomized clinical trials (RCTs) have been the design of choice for evaluation of technical and economic outcomes, qualitative and formative designs have more frequently employed to address issues such as acceptance of the new technology and the influence of the host organization/system on the adoption process. Many evaluation models have been proposed and applied in different sectors and settings. However, personnel responsible for advancing the electronic health information systems initiatives in Canada have not yet identified and agreed upon a core evaluation framework which could assess their respective initiatives and contribute to a core Canadian database on the impact of electronic health information systems projects or programs

Project Goals and Objectives

The goals of this project were (1) to conduct a review of the current approaches to evaluating the impact of health information systems, in particular electronic health records (EHRs); and (2) to develop a framework for future evaluation efforts of these initiatives which builds on (a) the identified best practices, and (b) feedback received from key stakeholders during the process of the review.

The project objectives included:

1. To identify and contact key personnel involved with the electronic health record initiative federally and in each province and territory and invite them to participate in the development of a core evaluation framework for Canadian EHR initiatives. This group would include those involved in planning and implementing the initiatives, policy makers who make the decisions re funding and implementation, researchers/evaluation personnel associated with the projects and key user groups, such as health care providers;
2. To identify the priority information needs of the key stakeholders with respect to evaluating the impact and success of health information systems projects (EHRs);
3. To perform a systematic search of the published literature and relevant program documents to identify the evaluation models which have been applied to health information projects, specifically those components which relate to use of a unique identifier, implementation of a client registry, development of real-time medication databases accessible to both physicians and pharmacists, use of imaging technology, and implementation of systems to provide electronic access to other diagnostic test. Telehealth was excluded, as the research team was award of other research proposals in this area which were being submitted for consideration;
4. To develop a protocol for assessment of the published literature and program reports which (a) addresses the identified information needs of the key stakeholders and (b) critically assesses the evaluation approaches which can be used to obtain the necessary information;
5. To identify best practices with respect to evaluation of health information system projects and synthesize them into a proposal for a comprehensive core evaluation framework for Canadian EHR initiatives;
6. To submit the proposed evaluation framework involved with the development of the health information system in the provinces, territories and federally for review and input;
7. To revise the proposed framework based on feedback received and to disseminate it to study participants and other identified key stakeholders

Research Team

The research team was lead by Doreen Neville, an associate professor of health policy and health care delivery at Memorial University Faculty of Medicine, and Stephen O'Reilly, Chief Executive Officer of the Newfoundland and Labrador Centre for Health Information (NLCHI). NLCHI was established in 1997, with the mandate to develop a comprehensive and integrated, patient oriented health information system, and is leading the provincial effort towards a full electronic health record for each individual in the province. Several other key personnel involved in the work of NLCHI around the development of EHRs were also members of the research team, including Mike Barron (the Health Information Network Project Leader); Donald MacDonald (NLCHI's Director of Research and Development and a graduate student working with Dr. Neville on the evaluation of the pharmacy component of an EHR); Kayla Gates (an analyst with the Research and Development Division of NLCHI and a graduate student working with Dr. Neville on the evaluation of the diagnostic testing component of the EHR); Lucy MacDonald (NLCHI's Director of Communications and Privacy); and Sandra Cotton (NLCHI's Director of Standards). Other team members included Dr. Ted Hoekman ( Professor of Medical Informatics at Memorial University Faculty of Medicine); Dr. Gerard Farrell (Assistant Professor, Family Medicine, Memorial University Faculty of Medicine and the Technology Representative for the Newfoundland and Labrador Medical Association) and Dr. Stephen Bornstein (Professor, Political Science, and Director of the Newfoundland and Labrador Centre for Applied Health Services Research) and Montgomery Keough (research assistant on the project until August 31st, 2003).

Scope of the Project

The main purpose of this project was to develop an evaluation framework for EHR initiatives in Canada which would meet the information needs of a variety of key stakeholders in policy and decision making roles, as well as users of the system, such as managers and clinical care providers. This framework was informed by a review of the current initiatives underway across Canada, a systematic review of the published literature and project reports, and consultation with key informants. It was not intended to be an academic document which proposes a conceptual model for understanding the design, implementation or impact of complex health information systems such as Electronic Health Records. Rather, the framework was developed to provide a practical guide to: (1) the types of evaluation questions which can be asked of the EHR initiatives; (2) the methodological options available to address these questions; and (3) the tradeoffs that occur when one or another approaches to evaluation is selected.

We hope that this framework will serve as a springboard and guide for discussions among key stakeholders regarding what is important to measure about the EHR initiatives in Canada, and how to feasibly address these questions in a rigorous manner. If the framework is used in several jurisdictions, then it will be possible to begin identifying common evaluation priorities, track and compare evaluation questions and methods, compile a national inventory of EHR evaluation projects, and identify opportunities for collaborative projects across jurisdictions and stakeholder groups

Target Audience

The target audience for this project includes key decision makers and researchers who are involved with the Electronic Health Record Initiatives in Canada at the federal and provincial/territorial levels. Examples include individuals who are involved in planning and implementing the initiatives, policy makers who make the decisions re funding and implementation, researchers/evaluation personnel associated with the projects and key user groups, such as health care providers.

Participants

One hundred and twenty five (125) people from across Canada were contacted regarding participating in this project, and 50 of those approached agreed to participate. Participants were secured from all provinces and included 9 Assistant Deputy Ministers of Health, 2 Secretaries to Treasury Board, 2 Assistant Secretaries to Treasury Board, and 37 individuals who held a variety of positions in government, health care institutions, research organizations, universities and information technology organizations.

Deliverables

Three deliverables were produced from the project and released as separate (but complementary) documents:

1. Towards an Evaluation Framework for Electronic Health Records: An Inventory Of Electronic Health Record Initiatives Across Canada
2. Towards an Evaluation Framework for Electronic Health Records:
   An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports;
3. Towards an Evaluation Framework for Electronic Health Records: A Proposed Evaluation Framework for Assessing Electronic Health Records Initiatives Across Canada.

Key Findings

Inventory of Electronic Health Records Initiatives Across Canada

Our review of the EHR initiatives in Canada indicates that there is little uniformity in the design and planned implementation of the identified core components of an EHR (Unique Personal Identifier/Client Registry; Pharmacy Network, Laboratory Network and Diagnostic Services Network), and each jurisdiction has a different configuration of legacy system upon which it is building its EHR. Faced with a similar scenario in the National Health Service in the United Kingdom, evaluators such as Heathfield and colleagues (1999) chose to study their Electronic Patient Record/Electronic Health Record systems in terms of form and functionality (i.e. the Electronic Health Record).

Published Literature and Project Reports Relevant to The Electronic Health Record

Assessment of the published literature and project reports revealed that there is a dearth of information regarding evaluation of geographically dispersed health information systems. Most evaluations of information systems in health care have dealt with relatively small scale initiatives, wherein new technologies replace the existing (usually paper-based) system. The setting for most evaluation studies is within a hospital or a limited hospital to physician office interface (for example, enabling access to lab test results). Search of the literature did not detect a single study that describes the evaluation of a system with all four core components of an Electronic Health Record (EHR).

Reviewed research studies were of varying quality. Many studies lacked rigor with incomplete descriptions of the system under study; others provided detailed information about evaluation methods, instruments and findings which were useful in the development of the proposed evaluation framework. A pre-/post- implementation study design is the most widely agreed upon approach to evaluating health information systems. Many studies identified the use of randomized control trials (RCTs) as being problematic in the evaluation of complex health information systems.

No generic approach to evaluation was identified. Several models have been proposed to guide selected aspects of evaluation activities; most of these are grounded in discipline specific Conceptual frameworks. The evaluation framework utilized by the National Health Service in the UK to guide evaluation of electronic patient records (EPRs) and electronic health records (EHRs) proved to be the most relevant template for the development of the proposed evaluation framework for Canadian initiatives in EHRs.

A Proposed Evaluation Framework For Assessing EHR Initiatives Across Canada

As noted above, our review of the EHR initiatives in Canada indicates that there is little uniformity in the design and planned implementation of the identified core components of an EHR. Faced with a similar scenario in the National Health Service in the United Kingdom, evaluators such as Heathfield and colleagues (1999) chose to study the system in terms of form and functionality (i.e. the Electronic Health Record), as opposed to distinguishing between different versions of these systems for evaluation purposes. In the evaluation framework presented below, the system can refer to the full EHR in each jurisdiction, or one or more subcomponents of the EHR irrespective of the particular technology which was implemented to achieve the desired functionality.

Healthfield (1998) suggests that there are 3 general types of rationale for why evaluation is conducted in the field of health information systems: (1) to insure accountability for expenditure of resources; (2) to develop and strengthen performance of agencies, individuals and/or systems; and (3) to develop new knowledge. While each of these rationales for evaluation may consider evidence collected by a variety of approaches, both qualitative and quantitative, they carry with them: (1) assumptions about what evaluation can contribute; (2) orientation towards particular evaluation methods; (3) and requirements in terms of the timelines and resources necessary to address them. Each of these is explored in the evaluation framework document.

The proposed framework is designed to be a guide to designing an evaluation initiative which is useful to a wide range of stakeholders involved in the EHR initiatives across Canada, including those who fund the system development and implementation, policy makers, decision makers at all levels of the system, users of the system, and researchers. It is not an academic document, and it does not propose a conceptual model for understanding the design, implementation or impact of complex health information systems. As a guide, it is illustrative, not exhaustive.

The proposed framework presents time frames, core questions, indicators, data sources and potential study designs for evaluation. It is organized around seven steps, and is complemented with appendices which provide additional evaluation questions and indicators which have been used in previous studies or are proposed for use in future evaluation efforts. The seven steps, modeled on the PROBE document (NHS Authority, March 2001), include:

Step 1:

Identify Key Stakeholders in Each Jurisdiction

Step 2:

Orient Key Stakeholders To the E.H.R. Initiative and Reach Agreement on Why an Evaluation Is Needed

Step 3:

Agree on When To Evaluate

Step 4:

Agree on What To Evaluate

Step 5:

Agree on How To Evaluate

Step 6:

Analyze and Report

Step 7:

Agree on Recommendations and Forward Them to Key Stakeholders

Recommendations

In Canada today, we have a tremendous opportunity to collaborate across jurisdictions and stakeholder groups to develop: (1) a standardized approach to assessment of EHR initiatives and (2) a national inventory of evaluation protocols, instruments and evidence. If the work completed with this project serves as a springboard for discussion among key stakeholders regarding what is important to measure about EHR initiatives in Canada and how to measure it, then we will have laid a foundation upon which common evaluation priorities across jurisdictions can be identified and pursued. Our recommendations are as follows:

1. It is crucial that collaborative evaluation efforts around EHR initiatives focus on the functionality of the systems being introduced, as opposed to the specific form of the technology being employed. This is not to suggest that there is no need for evaluation of proprietary technology, but rather to urge the key stakeholders identified in this document to find ways to work together to answer the big picture questions around the implementation of EHRs.
2. Many important evaluation questions are too costly or complex for one jurisdiction to address on their own. Cross-jurisdictional collaboration in evaluation of EHR initiatives is required to strengthen the generalizeability of the findings and to insure that adequate resources (including research expertise and funding) is available to pursue key questions.
3. It is important to be clear about what you hope to learn from an evaluation, and your underlying assumptions about what evaluation can and cannot achieve. We recommend use of the accountability/performance enhancement/knowledge development classification of perspectives presented in the proposed evaluation framework to: (1) stimulate discussion among stakeholders about their primary rationale for expending scarce resources on evaluation activities and (b) aid the group to identify the types of tradeoffs which will be required as a consequence of the evaluation questions they pursue and the resources they can access to complete the study.
4. The process of assembling the Inventory of Electronic Health Records Initiatives Across Canada was extremely time consuming. This inventory should be updated regularly and used as a vehicle for information exchange among the different jurisdictions. Simultaneously, there is a need for an agreement between jurisdictions regarding what types of information they are willing to share and the level of detail that can be made available to other stakeholders. It would be very useful to develop an agreement among the jurisdictions which would see them participate in an annual update of their key initiatives in a standardized format.
5. This framework document and its companion documents will be widely circulated to : (1) participants in the project; (2) identified EHR initiatives across Canada; (3) and international colleagues for feedback. Comments and suggestions should be directed to:

   Doreen Neville, ScD
   Associate Professor,
   Health Policy and Health Care Delivery
   Faculty of Medicine
   Memorial University of Newfoundland

   Email: dneville@mun.ca
   Phone: 709-777-6215
   Fax: 709-777-7382

Introduction

Background

An electronic health record (EHR) provides each individual in Canada with a secure and private lifetime record of their key health history and care within the health system. The record is available electronically to authorized health care providers and the individual anywhere, anytime, in support of high quality care. Recognizing the importance of the EHR in improving the quality and efficiency of health care in Canada, the Federal government established Canada Health Infoway (Infoway) to support and accelerate the development and adoption of interoperable electronic health records solutions across Canada. Infoway's strategic plan for achieving this goal involves each region taking a lead role in at least one major initiative with elements and results which can be transferred to other Canadian jurisdictions. The Newfoundland and Labrador Centre for Health Information (NLCHI) has taken a lead role in the Infoway priority area of client registries.

Infoway and NLCHI have identified four core components of an EHR: (1) a unique personal identifier/client registry; (2) pharmacy network; (3) laboratory network; and (4) diagnostic services network. This research project "Towards an Evaluation Framework for Electronic Health Records Initiatives" was designed to complement the Infoway/NLCHI work and to advance knowledge in the field of EHR through the development of an evaluation framework which can be used to assess projects/initiatives across the four components of an EHR.

Traditionally, evaluations of components of medical or health information systems have focused on (1) technical and systems features that affect systems use; (2) cost- benefit analysis; (3) user acceptance and (4) patient outcomes. More recently, evaluation approaches have addressed the context and processes that contribute to the outcomes of the project, and have incorporated aspects of change and innovation diffusion theories into the evaluation model. While randomized clinical trials (RCTs) have been the design of choice for evaluation of technical and economic outcomes, qualitative and formative designs have more frequently employed to address issues such as acceptance of the new technology and the influence of the host organization/system on the adoption process. Many evaluation models have been proposed and applied in different sectors and settings. However, personnel responsible for advancing the electronic health information systems initiatives in Canada have not yet identified and agreed upon a core evaluation framework which could assess their respective initiatives and contribute to a core Canadian database on the impact of electronic health information systems projects or programs

Project Goals and Objectives

The goals of this project were (1) to conduct a review of the current approaches to evaluating the impact of health information systems, in particular electronic health records (EHRs); and (2) to develop a framework for future evaluation efforts of these initiatives which builds on (a) the identified best practices, and (b) feedback received from key stakeholders during the process of the review.

The project objectives included:

1. To identify and contact key personnel involved with the electronic health record initiative federally and in each province and territory and invite them to participate in the development of a core evaluation framework for Canadian EHR initiatives. This group would include those involved in planning and implementing the initiatives, policy makers who make the decisions re funding and implementation, researchers/evaluation personnel associated with the projects and key user groups, such as health care providers;
   
2. To identify the priority information needs of the key stakeholders with respect to evaluating the impact and success of health information systems projects (EHRs);
3. To perform a systematic search of the published literature and relevant program documents to identify the evaluation models which have been applied to health information projects, specifically those components which relate to use of a unique identifier, implementation of a client registry, development of real-time medication databases accessible to both physicians and pharmacists, use of imaging technology, and implementation of systems to provide electronic access to other diagnostic test. Telehealth was excluded, as the research team was award of other research proposals in this area which were being submitted for consideration;
   
4. To develop a protocol for assessment of the published literature and program reports which (a) addresses the identified information needs of the key stakeholders and (b) critically assesses the evaluation approaches which can be used to obtain the necessary information;
5. To identify best practices with respect to evaluation of health information system projects and synthesize them into a proposal for a comprehensive core evaluation framework for Canadian EHR initiatives;
6. To submit the proposed evaluation framework involved with the development of the health information system in the provinces, territories and federally for review and input;
7. To revise the proposed framework based on feedback received and to disseminate it to study participants and other identified key stakeholders

Research Team

The research team was lead by Doreen Neville, an associate professor of health policy and health care delivery at Memorial University Faculty of Medicine, and Stephen O'Reilly, Chief Executive Officer of the Newfoundland and Labrador Centre for Health Information (NLCHI). NLCHI was established in 1997, with the mandate to develop a comprehensive and integrated, patient oriented health information system, and is leading the provincial effort towards a full electronic health record for each individual in the province. Several other key personnel involved in the work of NLCHI around the development of EHRs were also members of the research team, including Mike Barron (the Health Information Network Project Leader); Donald MacDonald (NLCHI's Director of Research and Development and a graduate student working with Dr. Neville on the evaluation of the pharmacy component of an EHR); Kayla Gates (an analyst with the Research and Development Division of NLCHI and a graduate student working with Dr. Neville on the evaluation of the diagnostic testing component of the EHR); Lucy MacDonald (NLCHI's Director of Communications and Privacy); and Sandra Cotton (NLCHI's Director of Standards). Other team members included Dr. Ted Hoekman (Professor of Medical Informatics at Memorial University Faculty of Medicine); Dr. Gerard Farrell (Assistant Professor, Family Medicine, Memorial University Faculty of Medicine and the Technology Representative for the Newfoundland and Labrador Medical Association) and Dr. Stephen Bornstein (Professor, Political Science, and Director of the Newfoundland and Labrador Centre for Applied Health Services Research) and Montgomery Keough (research assistant on the project until August 31^st, 2003).

Scope of the Project

The main purpose of this project was to develop an evaluation framework for EHR initiatives in Canada which would meet the information needs of a variety of key stakeholders in policy and decision making roles, as well as users of the system, such as managers and clinical care providers. This framework was informed by a review of the current initiatives underway across Canada, a systematic review of the published literature and project reports, and consultation with key informants. It was not intended to be an academic document which proposes a conceptual model for understanding the design, implementation or impact of complex health information systems such as Electronic Health Records. Rather, the framework was developed to provide a practical guide to: (1) the types of evaluation questions which can be asked of the EHR initiatives; (2) the methodological options available to address these questions; and (3) the tradeoffs that occur when one or another approaches to evaluation is selected.

We hope that this framework will serve as a springboard and guide for discussions among key stakeholders regarding what is important to measure about the EHR initiatives in Canada, and how to feasibly address these questions in a rigorous manner. If the framework is used in several jurisdictions, then it will be possible to begin identifying common evaluation priorities, track and compare evaluation questions and methods, compile a national inventory of EHR evaluation projects, and identify opportunities for collaborative projects across jurisdictions and stakeholder groups

Target Audience

The target audience for this project includes key decision makers and researchers who are involved with the Electronic Health Record Initiatives in Canada at the federal and provincial/territorial levels. Examples include individuals who are involved in planning and implementing the initiatives, policy makers who make the decisions re funding and implementation, researchers/evaluation personnel associated with the projects and key user groups, such as health care providers.

Participants

One hundred and twenty five (125) people from across Canada were contacted regarding participating in this project, and 50 of those approached agreed to participate. Participants were secured from all provinces and included 9 Assistant Deputy Ministers of Health, 2 Secretaries to Treasury Board, 2 Assistant Secretaries to Treasury Board, and 37 individuals who held a variety of positions in government, health care institutions, research organizations, universities and information technology organizations.

Deliverables

Three deliverables were produced from the project and released as separate (but complementary) documents:

1. Towards an Evaluation Framework for Electronic Health Records: An Inventory Of Electronic Health Record Initiatives Across Canada
2. Towards an Evaluation Framework for Electronic Health Records:
   An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports;
3. Towards an Evaluation Framework for Electronic Health Records: A Proposed Evaluation Framework for Assessing Electronic Health Records Initiatives Across Canada.

An Inventory of Canadian Electronic Health Record Initiatives

Methods

The multi-step process used to compile the inventory of major Electronic Health Record (EHR) projects across Canada was as follows:

Step 1:

An initial search was conducted using an internet search engine, where each province/territory was used in combination with the term Electronic Health Record and variations thereof (e.g. Health Information System).

Step 2:

The Health Canada website was searched for information related to the development of an electronic health record for each province/territory.

Step 3:

Relevant websites identified through the initial internet search (e.g. Western Health Information Collaborative, Newfoundland and Labrador Centre for Health Information, etc.) were searched for documents related to health information system projects.

Step 4:

A list of key contacts for each province/territory was compiled based on knowledge of titles of individuals that are involved in the development of the provincial Health Information Network (HIN) in Newfoundland and Labrador. Key contacts included, but were not limited to, Assistant Deputy Minister of Health and Community Services equivalents, Secretary to Treasury Board equivalents, Assistant Secretary to Treasury Board equivalents and key individuals at various institutions (e.g. SHIN, NLCHI, etc.). The list of key contacts included some of the individuals that were initially identified as having an interest in the development of health information systems and invited to participate in the project. All individuals were contacted via e-mail (and in some instances telephone) and asked for their help in the development of an inventory of Electronic Health Record (EHR) projects across Canada. In some cases, the individual that was contacted suggested that they would not be helpful in this project and suggested another individual as a more appropriate contact. A revised list of key contacts was compiled.

Step 5:

Based on input from the key contacts and the multi-phase internet search, a draft document highlighting the major EHR projects across Canada was compiled. Each contact received his or her respective provincial component for review and revision.

Step 6:

Based on feedback received from key contacts, a second draft document was compiled. The draft document was sent to all key contacts with the information relevant to their province/territory highlighted. Each individual was asked to advise where information was missing or incorrect.

Step 7:

After final input was received from key contacts, the final document was completed. A special acknowledgement is due to Donald MacDonald and Kayla Gates, whose graduate work helped inform the review of the Pharmacy and Laboratory Networks initiatives.

Key Findings

A summary of the major Electronic Health Record initiatives is presented below. For more detailed descriptions, readers are referred to the full document entitled Towards an Evaluation Framework for Electronic Health Records: An Inventory Of Electronic Health Record Initiatives Across Canada

Unique Personal Identifier/Client Registry

The unique personal identifier/client registry has been recognized as the basic building block for the creation of province-wide electronic health records. While specific functions vary from project to project, ideal functionality includes the ability to cross-reference multiple, person-specific, medical record numbers from multiple health information systems (systems that contain information and data needed to compile a comprehensive electronic health record).

In 2001, the province of Newfoundland and Labrador implemented the first province-wide information system that links all regional health information systems into the initial phase of the provincial electronic health record; the Unique Person Identifier/Client Registry System. This system connects all health regions into a common master client index via a sophisticated HL7 compatible integration engine.

Canada Health Infoway recently announced a partnership with NLCHI to further develop the current UPI/Client Registry system into a model that can be adopted by other regional and provincial jurisdictions. The Newfoundland and Labrador client registry has been designated the lead provincial Common Client Registry System project and will be used to develop national standards and proof of concept projects related to the Pan-Canadian Electronic Health Record. This project is underway and is expected to be completed by early 2004.

Client registries are a major focus for the short-term investment strategy of Canada Health Infoway Another key initiative, at the sub-provincial level, is underway in the Capital Health Region, Edmonton Alberta. British Columbia, New Brunswick and Ontario are in varying stages of implementing systems very similar to models implemented in Newfoundland and Labrador and Alberta.

Pharmacy Networks

Information systems developed to capture data related to prescription medications are variable in functions across Canada. In the past, these systems were developed to process claims for government-funded drug programs. Technological advances in the last 10 years now allow for more enhanced functionality of medication systems (Pharmacy Networks). These systems now provide an opportunity to capture real time medication data, which can lead to health, economic and financial benefits for both governments and individual patients (Benefits Driven Business Case, NLCHI, 1998).

Nine of the ten provinces currently have some form of Pharmacy Network. Most of these networks provide an interface between community pharmacists and provincially funded drug programs, for adjudication purposes. The most advanced networks include the ability to provide complete drug profiles to pharmacists at the point of distribution. Such systems have been implemented in four provinces: Alberta (WellNet), Prince Edward Island (Pharmacy Network), British Columbia (PharmNet) and Manitoba (DPIN). Each of these provinces, to varying degrees, have incorporated the following functions in designing their Pharmacy Network: on-line real time adjudication, checks for duplication and double-doctoring, drug utilization reviews, checks for patient eligibility, drug profiles, connection to hospitals and physician offices, and electronic prescribing. A comparative listing of these functions is provided in Table 1, followed by a brief description the four networks.

Five provinces, Newfoundland and Labrador, Prince Edward Island, Manitoba, Alberta and British Columbia, have implemented (or are planning to implement) systems with more comprehensive functional capability. One of the major enhancements found in these systems is the ability to provide real-time patient drug profiles at the time the prescription is filled by the pharmacist.

Table 1 Functions of Selected Provincial Pharmacy Networks

Function	Alberta (2002)	British Columbia (1995)	Manitoba (1994)	PEI (1999)
On-line real time adjudication and transmission	x	x	x	x
Checks for duplication	x	x	x	x
Checks for double-doctoring	x	x	x	x
Provides full retrospective drug use evaluation/review on patient profile	x	x	x	x
Tracks patient's deductible on co-pay	x	x	x	x
Patient eligibility checked	x	x	x	x
Immediately identifies what is and is not a benefit	x	x	x	x
Pharmacare Status	x	x	x	x
Drug Profiles	x	x	x	x
Drug Profiles history on each patient	x	x	x	x
Records Rx dispensed for all or select group of patients	Will record all prescriptions	All prescriptions are recorded	All prescriptions are recorded. Note: not mandatory for aboriginals, but most are recorded.	Will report all prescription
Ability to record non-dispensing events	x
Connected with hospitals	x	x
Connected with physician offices/desk top prescribing	x
Other Notes	Currently in a 6 month pilot stage.		Five year plan.	In the development stage.

Source: Pharmacy Scoping Project Briefing Note, 2002, NLCHI

Manitoba

In 1994, the province of Manitoba implemented the Drug Programs Information Network (DPIN). The DPIN system was the first system in Canada that connected all community pharmacies. There are currently no linkages of community physicians to hospitals, although these connections are part of Manitoba's five-year business plan for the DPIN. The DPIN was originally developed to provide complete prescription profiles to pharmacists at the time of dispensing, as well as enhanced drug utilization reviews. The functions of the DPIN system are similar to other provinces with Pharmacy Networks (see Table 1), although in Manitoba it is not mandatory for pharmacists to record prescriptions filled by the treaty status/Registered Indian population (Kozyrskyj, Brown and Mustard, 1998).

British Columbia

The British Columbia PharmaNet initiative was implemented in 1995 in an attempt to contain escalating costs to the government drug program, and to improve the health of the population through the provision of drug therapy decision tools. The network allows for the exchange of medication information between pharmacists and hospital emergency rooms, however there are no linkages to community physicians.

An additional function of the PharmaNet system is the Pharmacare Trial Prescription Program. This module was developed to reduce expenditures for patients who are put on a new medication and for some reason must discontinue its use. A patient is given only a portion of the new drug, and their health care provider then monitors their progress. If for some reason the drug must be discontinued, the full prescription has not been wasted.

Alberta

The Pharmaceutical Information Network (PIN) is being developed in the Province of Alberta as part of the Alberta Wellnet initiative. The objective of the PIN project is to provide health care professionals with the information necessary to make optimal decisions on drug therapy. The network will not only provide adjudication functions for Alberta's government drug plan, it will also connect community pharmacists, physicians and hospitals to allow for the exchange of patient information. This will allow a physician to monitor a patient's current, as well as, historical drug profile, create/modify prescriptions through Computerized Physician Order Entry (CPOE), and access decisions support tools to assist in drug therapy decisions. The PIN project in Alberta was approved for implementation based on the estimated $69 million the province would save annually as a result of a reduction of adverse drug events (Pharmaceutical Information Network - Medication Information Strategy (White Paper), Western Health Information Collaborative, April 2002).

Prince Edward Island

Prince Edward Island is the latest province to begin developing a Pharmacy Network. At present this system is still in the implementation stage and much of the system development documentation is classified as proprietary. It is known that in 1997 the province implemented a Pharmaceutical Informatics Project (PhIP) system, which provided province-wide networking for the submission of pharmacists' claims to the government drug program. In 1999, this system was enhanced to allow fee-for-service physicians to submit medical claims to government for payment. Recently, the Province has started a process towards developing a Pharmacy Network that would enhance the role of the pharmacist by providing comprehensive functionality. It is not unreasonable to assume this system would include similar functions found in the three provinces with established Pharmacy Networks.

Laboratory Networks

Electronic sharing of laboratory information has been identified as high priority by several Canadian provinces and territories. Nine of the thirteen jurisdictions have indicated that they have plans to implement one or more aspects of an integrated laboratory network and five jurisdictions, including Newfoundland and Labrador, Manitoba, Saskatchewan, Alberta and British Columbia, have major implementation initiatives currently underway. In some jurisdictions, the initial focus is on electronic exchange of laboratory orders and results. Other jurisdictions are focusing on enabling immediate access to a patient's lab test history. For most jurisdictions, the ultimate goal is to have province wide integration of laboratory information that will support both the electronic exchange of orders and results between geographically dispersed sites as well as immediate access to a patient's longitudinal history of laboratory services from any site in the province. Direct Provider/Physician Order Entry (POE) has been identified as a long-term goal by some jurisdictions, but has not been the focus of any major initiatives to date.

Since the majority of the jurisdictions are in their planning and pilot implementation stages, detailed documentation is often unavailable. However, an overview of what is known about electronic exchange of laboratory information in Canada is provided below.

British Columbia

In 1998, the HealthNet/BC Project formed the Lab Test Standard Task Group (LTSTG) with representatives from HealthNet/BC working groups, the BC Health Information Standards Council and private sector and provincial labs. In 1999, the BC Lab Test Standard (LTS) was developed to enhance the quality of patient care through the timely exchange of consistent lab data, and to reduce the cost of managing the exchange of laboratory information. Version 1.3 of the Lab Test Standard (LTS) is now available. The standard defines the business and technical requirements for the electronic exchange of lab test data and accounts for all information exchanges that occur from the time an order is issued until the time a final result is received. The Lab Test Standard is based on a set of standard identifiers including the Personal Health Number (PHN), which is the provincial standard for personal identification; the Provider Data Standard (including Provider ID), BC Test Order Codes (BCTOC), the standard for test orders; and LOINC, the standard for reporting of test results. The LTS also provides a comprehensive set of rules regarding ordering lab tests, referring/redirecting orders, requesting order status, reporting results, accessing a patients' lab test history and privacy and confidentiality issues.

A number of BC laboratory systems are currently using the Lab Test Standard. Among these is a private sector province-wide initiative called PathNet. PathNet is a web-based electronic laboratory reporting system that integrates patient laboratory information from multiple participating laboratories, within and across regional boundaries. It allows physicians to access up-to-date laboratory test results, in real-time, for any patient that has had a test completed at any participating laboratory. In addition, PathNet will flag any abnormal test result(s) and allow access to a patient's laboratory test history. Other laboratory systems using the Lab Test Standard include BC Communicable Disease Control (BCCDC) in providing lab test results to the Population Health Information System (PHIS), the federal Canadian Integrated Public Health Surveillance initiative and the BC Cancer Agency (Western Health Information Collaborative, 2002).

Alberta

The primary focus for laboratory information exchange in Alberta has been on results reporting and providing a longitudinal history of lab test results. A joint initiative between the Capital Health Authority (CHA), the Physician Office System Program (POSP), Dynacare Kasper Medical Laboratories (DKML) and Alberta Wellnet has created a laboratory test results repository that will allow physicians to electronically receive and file lab results directly into a patient's record. Presently, the system is for reporting test results only and still requires paper forms to be used for lab requisitions. While physician order entry has been identified as a component of the provincial electronic health record initiative, there is still significant work remaining.

The Capital Health Authority Electronic Lab Results Reporting Project was launched in Northern Alberta in June, 2002 and is anticipated to expand to include other health authorities. Currently, physician office systems can electronically receive requested lab test results and a pilot implementation of the lab results history system is being prepared. With full implementation, it is proposed that access to a patient's lab test history will be available along with the Alberta Wellnet Pharmaceutical Information Network (PIN).

Manitoba

Manitoba is currently developing an integrated multi-site organization known as Diagnostic Services of Manitoba (DSM), which will undertake all provincial laboratory services. The goal of DSM is to avoid future costs through economies of scale in material costs and test utilization.

An integrated province-wide Laboratory and Rural and Northern Imaging Information System (LIS/RIS) is the key infrastructure component required for full functioning of the DSM. The system of interlocking laboratories will use a common set of standards for all associated laboratory procedures. With full implementation, specimens can be collected and prepared in one location, transported to another site for testing, and results will automatically be returned to the originating site in real-time. Only laboratory personnel will have authorized access to the LIS/RIS. All hospital and physician access will be through a data repository or a hospital results reporting capability (Western Health Information Collaborative , 2002).

The initial implementation phase of the DSM will not include automated computer order entry. As order entry capabilities become more available in the province, the LIS/RIS will be expanded to facilitate automated physician order entry. Laboratory and imaging results will be the initial building block for Manitoba's electronic health record. The initial phase of the DSM is expected to be complete by 2004 (Western Health Information Collaborative , 2002).

Saskatchewan

As a component of the Saskatchewan Health Information Network (SHIN), Saskatchewan is planning a province wide web-based capability for laboratory test orders and results reporting, based on the storage and extraction of laboratory data from a central repository. Since the fall of 2000, work has been underway on a multi-regional integrated clinical system project that will integrate applications from several areas including registration, lab, pharmacy and operating room scheduling. All regions involved in the project are implementing a common Laboratory Information System that will help automate the process of ordering, performing and reporting laboratory tests. Systems are being configured to generate HL7 messages to enable information flow between applications into a common view once it has been installed (Western Health Information Collaborative, 2002).

Newfoundland and Labrador

The laboratory information system (LIS) that is currently used by all clinical laboratories within Newfoundland and Labrador is the MEDITECH laboratory information system. The Meditech LIS is a computerized information system that manages laboratory test data throughout the testing process and generates laboratory reports. Within health care institutions in Newfoundland and Labrador, the Meditech LIS is interfaced with the Meditech Hospital Information System (HIS). Each of the Province's Institutional and Integrated Health Boards share a common laboratory coding system and LIS network. In addition, any clinic that has established a connection to the Meditech system within the region can access the laboratory information. However, laboratory information exchange across regions or between the regions and the provincial reference laboratories in St. Johns occurs by fax, telephone, postal service and courier.

Medinet is an interface that enables communication between two heterogeneous laboratory information systems (LIS) where one LIS must be a Meditech system. As a first step towards the province wide integration of laboratory information, the initial implementation of Medinet will enable real-time electronic exchange of laboratory test orders and results between all institutions within the Central East and Avalon regions and the provinces' two largest reference laboratories in St. John's. Successful implementation of Medinet in these regions is expected to lead to province-wide adoption of Medinet and enable real-time exchange of laboratory test orders and results between reference laboratories and health care institutions throughout the Province.

Diagnostic Imaging Networks

In recent years, the Canadian health care environment has witnessed significant investment in Picture Archiving and Communications Systems (PACS). Many institutions are implementing these systems as attachments to already installed diagnostic imaging systems. Diagnostic imaging systems provide the written report associated with images, whether images are electronic or film. Drivers for these systems include reduced cost of film and associated storage, as well as the ability to provide remote diagnosis on images taken from remote locations where imaging equipment is available but a radiologist is not.

With the exception of Manitoba, who indicated that they were in the process of creating an integrated, multi-site diagnostic services network, no other jurisdiction identified diagnostic imaging as the focus of a major initiative during the development of this document. One province, Newfoundland and Labrador, has identified on-line ordering and immediate access to a patient's longitudinal history of diagnostic services from any site in the Province as a later phase of the provincial electronic health record.

An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports

Methods

The project scope included a systematic assessment of the published literature and program reports to identify existing evaluation approaches and best practices in EHR evaluation. The need for a systematic assessment of the published literature was established subsequent to a thorough search being conducted for published systematic reviews. This search included a review of key databases and consultation with experts in the field of EHRs.

Search Strategy Utilized in the Literature Review

The selection process used in this study was a multi-stage process which involved a broad preliminary selection applied to the citations generated from electronic databases, including MEDLINE, CINAHL, PubMed and the Cochrane Collaborative. Reference lists of retrieved publications were also searched for relevant citations. Academic librarians were consulted at various stages of the search process.

The initial search included the following search strings:

Information and Communication or "Information and Communication"
Limits: Field - MeSH Major Topic, MeSH Terms fields. Language: English
Publication Periods: 1995-2001

Information Technolog* or "Information Technolog*"
Limits: Field - MeSH Major Topic, MeSH Terms. Language: English
Publication Period: 1995-2001

Communication* Technolog* or "Communication* Technolog*"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

Telehealth or "Telehealth"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

Telemedicine or "Telemedicine"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

Electronic Record or "Electronic Record"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

Information Manage* or "Information Manage*"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

Record* Manage* or "Record* Manage*"
Limits: Field - MeSH Major Topic, MeSH Terms. Language - English
Publication Period: 1995-2001

This initial search strategy was narrowed to exclude articles on telehealth. This decision was based upon the fact that there were numerous other initiatives focusing on telehealth which were ongoing at the time of this study. The search was further refined based upon consultation with the project advisory committee and refinement of the research questions. The following key search terms were identified to form the basis of searching during this phase of the literature review:

Health information system and evaluation
Electronic medical record and evaluation
Electronic patient record and evaluation
Electronic health record and evaluation
Health information systems
Electronic patient record
Electronic medical record
Pharmacy and Evaluation
Lab and Evaluation
Diagnostic imaging and Evaluation
Master Patient Index
Patient Master Index

Two Community Health graduate students became involved as the study progressed. Their search strategies were reviewed by the Principal Investigator and the following terms were included in this study:

Clinical notes
Clinical progress notes
Computerized physician order entry
Electronic Prescribing
Laboratory Information System
Laboratory AND electronic exchange
Laboratory AND electronic interchange
Pharmacy Network
Drug Information System

In later stages of the search process and based upon consultation with an academic librarian, a more strategic search strategy was developed using the "thesaurus" and "explode" features of MEDLINE. As an example, in one search, the term "evaluation" was entered and the "explode" feature used to yield numerous related articles. The same feature was used for the key term "medical informatics", which also yielded a number of relevant current articles. Next, the two searches were combined to produce a refined but comprehensive search. These searches resulted in the retrieval of 87 journal articles.

In addition to searching electronic bibliographic indexes, the Google search engine was used to search for evaluation reports of health information systems projects within Canada and internationally. The search was carried out using many of the same key terms used for searching the electronic bibliographic indexes.

The project research assistant also wrote to experts in the field (academics, health information systems' administrators and planners) to solicit additional information about any information (unpublished studies, projects, initiatives) they were aware of and then compared these to the list of journal articles retrieved through the database search. Most of those consulted indicated that there is very little information on the evaluation of geographically dispersed health information systems.

During the course of this project, the Principal Investigator, Dr. Doreen Neville, was a Canadian Associate in the Commonwealth Harkness Program in International Health Policy. Dr. Neville presented the work in progress on this grant in bimonthly forums attended by a number of international health policy experts and sought additional feedback from experts familiar with information system development throughout the year. These contacts were invaluable in helping to identify potentially relevant published and grey literature in this field. Key informants included Dr. Steven Schoenbaum, Vice President of the Commonwealth Fund, who has long standing involvement with health information system development in the US and Dr. Tim Scott, a Harkness Fellow from the UK who was working with Kaiser Pernamente in Hawai to evaluate their health information network. Part of the Harkness experience involved a field trip to the Veteran's Health Complex in Washington DC to learn first hand from key personnel about the experience of building, implementing and evaluating a very sophisticated and continually evolving computerized patient record system. Other Harkness Fellows also provided information about electronic health records initiatives in their respective countries (Australia, New Zealand and the U.K.) and identified additional sources of grey literature on this topic. At the final reporting seminar in Nashville, Tennessee in June 2003, a preliminary evaluation framework was presented and feedback received was incorporated into the framework. The Harkness Fund was helpful also in assisting Dr. Neville to identify a group of experts working in this area in the USA who would be willing to provide feedback on the project, including Drs. David Bates, Ted Shortcliff, and Rob Kolodner; these individuals will be sent a copy of the final report and invited to provide feedback.

Inclusion/Exclusion Criteria

After a comprehensive search of both the grey and published literature, a total of 219 articles and project reports were retrieved (textbooks are not included in this count and are referenced separately in the final report). All 219 articles/reports were independently assessed by the Principal Investigator and another member of the project team for inclusion in the annotated bibliography. Where a difference of opinion or uncertainty by either reviewer existed, final consensus was researched by joint review and discussion.

Selection of articles/reports for the annotated bibliography was carried out in two phases:

Phase I: An article was retained if it met each of the following inclusion criteria:

• English in Language
• Published after 1994 (due to the relative currency of the topic)
• Focuses on one (or more) of the four core components of an electronic health record - unique personal identifier/client registry, pharmacy network, laboratory network and diagnostic imaging network
• Describes the evaluation of a health information system project, including presentation of findings

An article was excluded if it:

• Had a technology focus (computer programs and language standards)
• Information system was limited to one location (e.g. within a hospital setting)
• Focused on Telehealth Applications (telehealth applications were beyond the scope of this project)

The first phase of the selection process revealed only 10 articles/project reports that met the inclusion/exclusion criteria.

Phase II: Due to the small number of articles/reports meeting all the inclusion/exclusion criteria, the 219 articles/project reports were re-evaluated. Building upon the work of Freidman and Wyatt (1997) around approaches to the evaluation of health information systems, a comprehensive quality assessment tool was developed to critically appraise articles across a range of criteria which were both objectivist and subjectivist in nature (see Appendix A in the complementary report titled Towards an Evaluation Framework for Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports). Key considerations in developing the quality assessment tool included:

1. (1) Relevance: How useful are the findings with respect to the task at hand i.e. do they contribute something new to our understanding of evaluation approaches for complex health information systems?
2. (2) Rigor: Has a thorough and appropriate approach been applied to key research methods in the study?
3. (3) Credibility: Are the findings well presented and meaningful?

An article/report was selected for inclusion in the annotated bibliography if it was considered valuable to the development a comprehensive evaluation framework for health information systems projects (i.e. relevant). An article was excluded if it was not able to contribute to the goals of the project or was outside the project scope.

Key Findings

Articles/Reports Included in the Annotated Bibliography

A total of 93 articles/reports were selected for inclusion in the annotated bibliography. Among those selected, 39 articles/reports were research studies (including systematic reviews) that focused on the evaluation of one or more aspects of an electronic health record. These were critically assessed using the complete Quality Assessment Checklist. A further 54 articles/reports discussed an approach to evaluating health information systems projects and therefore were also deemed relevant to the project goals. However, as they did not present the findings of a research study per se, the assessment did not include a review of the scientific rigor of the methodology or the credibility of the research findings.

Trends in the Literature

Assessment of the published literature and project reports revealed that there is a dearth of information regarding evaluation of geographically dispersed health information systems. Most evaluations of information systems in health care have dealt with relatively small scale initiatives, wherein new technologies replace the existing (usually paper-based) system. The setting for most evaluation studies is within a hospital or a limited hospital to physician office interface (for example, enabling access to lab test results). Search of the literature did not detect a single study that describes the evaluation of a system with all four core components of an Electronic Health Record (EHR).

Reviewed research studies were of varying quality. Many studies lacked rigor with incomplete descriptions of the system under study; others provided detailed information about evaluation methods, instruments and findings which were useful in the development of the proposed evaluation framework. A pre-/post- implementation study design is the most widely agreed upon approach to evaluating health information systems. Many studies identified the use of randomized control trials (RCTs) as being problematic in the evaluation of complex health information systems.

No generic approach to evaluation was identified. Several models have been proposed to guide selected aspects of evaluation activities; most of these are grounded in discipline specific conceptual frameworks. The evaluation framework utilized by the National Health Service in the UK to guide evaluation of electronic patient records (EPRs) and electronic health records (EHRs) provided a template for developing a evaluation framework for Canadian initiatives in EHRs.

A Proposed Evaluation Framework for Assessing Electronic Health Records Initiatives Across Canada

Methods

The following process was used to develop the proposed evaluation framework.

1. A comprehensive search of the literature concerning the evaluation of complex health information systems, particularly those most closely related to the development of an Electronic Health Record, was conducted and used to generate a synthesis of the literature around evaluation efforts in this field and to outline a preliminary draft of an evaluation framework.
2. A summary of the current (as of May 2003) Electronic Health Records Initiatives across Canada was produced and provided to key informants in each jurisdiction for verification and revision as required.
   
3. During the course of this project, the Principal Investigator, Dr. Doreen Neville, was a Canadian Associate in the Commonwealth Harkness Program in International Health Policy. Dr. Neville presented the work in progress on the evaluation framework in bimonthly forums attended by a number international health policy experts and sought additional feedback from experts familiar with information system development throughout the year. At the final reporting seminar in Nashville, Tennessee in June 2003, a preliminary evaluation framework was presented and feedback received was incorporated into the framework.
4. In July 2003, participants were provided with both the inventory of currrent Electronic Health Record Initiatives Across Canada and the preliminary evaluation framework and asked to rank their priority evaluation questions within the 3 time frames proposed by the model (pre-implementation, implementation process and implementation impact). A total of 20 participants across Canada provided feedback on the proposed evaluation framework.
   
5. The feedback received from from participants was used to further refine the proposed evaluation framework.
6. The framework document was then prefaced by a synopsis of the literature, which provides an overview of approaches to evaluation of electronic health records related projects, including perspectives on evaluation, evaluation frameworks or models commonly used, and a summary of the key messages regarding future initiatives in evaluation in this field. In addition, appendices were attached which provide a sample of evaluation questions and a menu of indicators which were used in previous studies or recommended for use in future evaluations.

Key Findings

Overview of Evaluation Approach

This framework is designed to be a guide to designing an evaluation initiative which is useful to a wide range of stakeholders involved in the EHR initiatives across Canada, including those who fund the system development and implementation, policy makers, decision makers at all levels of the system, users of the system, and researchers. It is not an academic document, and it does not propose a conceptual model for understanding the design, implementation or impact of complex health information systems. Rather, it seeks to provide a practical guide to the types of questions which can be asked of the EHR initiatives, the options available to address these questions, and some of the tradeoffs that will occur if one or another approach to evaluation is selected. As a guide, it is illustrative, not exhaustive.

Our review of the EHR initiatives in Canada indicates that there is little uniformity in the design and planned implementation of the identified core components of an EHR (Unique Personal Identifier/Client Registry; Pharmacy Network, Laboratory Network and Diagnostic Services Network), and each jurisdiction has a different configuration of legacy system upon which it is building its EHR. Faced with a similar scenario in the National Health Service in the United Kingdom, evaluators such as Heathfield and colleagues (1999) chose to study the system in terms of form and functionality (i.e. the Electronic Health Record), as opposed to distinguishing between different versions of these systems for evaluation purposes. In the evaluation framework presented below, the system can refer to the full EHR in each jurisdiction, or one or more subcomponents of the EHR irrespective of the particular technology which was implemented to achieve the desired functionality.

We hope that this framework will serve as a springboard and guide for discussions among key stakeholders regarding what is important to measure about the EHR initiatives in Canada, and how to feasibly address it in a rigorous manner. If the framework is used in several jurisdictions, then it will be possible to begin identifying common evaluation priorities, track and compare evaluation questions and methods, begin to compile a national inventory of EHR evaluation projects, and identify opportunities for collaborative projects across jurisdictions and stakeholder groups.

Steps Utilized in the Evaluation Framework

The framework is organized around several steps, as presented below, and is complemented with appendices which provide additional evaluation questions and indicators which have been used in previous studies or are proposed for use in future evaluation efforts.

Step 1: Identification of Key Stakeholders in Each Jurisdiction

We have identified several categories of stakeholders who would be considered core to an evaluation of the full EHR initiative in each jurisdiction. Representatives of a variety both national and provincial/territorial sectors are included in this list. Not only is it important that a wide range of stakeholders be involved in and apprised of the evaluation efforts within their own jurisdictions; it is also crucial that a number of individuals and organizations are aware of the initiatives across the country because it will improve the likelihood that (1) evaluation of EHR initiatives will get on and remain on the radar of these organizations as a strategic initiative and one which requires dedicated resources for input; (2) greater strategic alignment between the goals of the broader health system and the goals of the EHR initiatives will occur; (3) information exchange across jurisdictions will occur; (4) comparable evaluation approaches will be introduced across the country where feasible; (5) long term, stable champions for evaluation of EHR initiatives will be engaged at both the national and provincial/territorial levels.

Step 2: Orient Key Stakeholders to the EHR Initiative and Reach Agreement on WHY an Evaluation is Needed

It is important to orient key stakeholders to the EHR initiative and the evaluation process as early as possible, to determine their: (a) expectations of the EHR initiatives in their jurisdiction and (b) views on what an evaluation plan should address. A workshop format has proved useful for this type of stakeholder engagement, wherein an overview of the EHR initiative is presented; expectations documented and views on evaluation elicited.

Healthfield (1998) suggests that there are 3 general types of rationale for why evaluation is conducted in the field of health information systems: (1) to insure accountability for expenditure of resources; (2) to develop and strengthen performance of agencies, individuals and/or systems; and (3) to develop new knowledge. Given the diversity of key stakeholders involved with EHR initiatives, it is highly likely that they will identify different rationales for conducting evaluation. For example, one could expect that individuals/agencies responsible for the administration of public funds would highlight accountability as a major reason for evaluation; clinicians and administrators would be most interested in performance enhancements, and academics would likely value most highly the opportunity to gain new knowledge in their respective fields.

While each of these rationales for evaluation may consider evidence collected by a variety of approaches, both qualitative and quantitative, they carry with them: (1) assumptions about what evaluation can contribute; (2) orientation towards particular evaluation methods; (3) and requirements in terms of the timelines and resources necessary to address them. Each of these is explored in the evaluation framework document.

Step 3: Agree on When To Evaluate

Ideally, evaluation of complex information systems should involve longitudinal evaluation, that is, evaluation that occurs over time, and/or involves multiple data collection points. We recommend that whenever possible, the evaluation of EHR projects in Canada involve data collection at 3 or more points: (1) baseline (pre-system implementation); (2) during implementation and (3) post implementation (preferably multiple measures at 6, 12 and 18 months post implementation). We recognize that many jurisdictions have introduced (or are about to introduce) one of more components of a province wide EHR, and hence new baseline collection of data is not possible. However, pre-implementation data may be available from scoping exercises conducted prior to system implementation, or from separately conceived and completed evaluations of work flow, audits of patient charts, or research projects. Whenever pre-existing measures are available they should be noted so as to inform the design of any evaluation projects which are conducted during the implementation and post implementation phases.

Step 4: Agree on What to Evaluate

It is well recognized that there is virtually an endless number of research and evaluation questions which could be posed about complex health information systems such as the Canadian EHR initiatives. However, resources to pursue these issues are limited, in terms of funding and availability of personnel with expertise to conduct the evaluation. Therefore it is very important that each jurisdiction feels that it is gaining the maximum benefit it can from the investment of scarce resources in evaluation. A priority setting exercise with key stakeholders is one way to (a) identify the questions that it is important to answer (versus the questions that it is easy to answer) and (b) insure that all key stakeholders have an investment in the evaluation projects which are undertaken. If the evaluation framework proposed in this document experiences wide uptake across Canada, there will also be an opportunity to avoid duplication of effort where possible, or to strengthen the design of a project by conducting it simultaneously in more than one jurisdiction. One approach to priority setting would be build on the stakeholder identification of why an evaluation is important ( accountability, performance enhancement and/or knowledge development) and then identify core and optional questions within each category.

Step 5: Agree on How to Evaluate

As noted above, both the rationale for undertaking evaluation, and the particular questions which are important to each stakeholder have implications for the methods which can be used to conduct the evaluation. The evaluation framework document provides tables which illustrate Steps 4 and 5, with our recommended core questions for each evaluation period and category highlighted in bold type. The tables specify the time frame, sample and core questions, indicators which can be used to address the question, potential data sources and type of study design which would be feasible and appropriate.

A discussion of the most feasible methods for approaching the selected evaluation questions will involve consideration of the tradeoffs involved with the methods chosen. Each jurisdiction will need to consider the resources they have available to devote to the evaluation and determine the best use of those resources in term of evaluation questions addressed and methods used. In addition, we support the recommendations of Kaplan (1997) in that all jurisdictions undertake an evaluation which: (a) focuses on a variety of concerns; (b) uses multiple methods: (c) is modifiable; (d) is longitudinal; and (e) includes both formative and summative approaches (formative evaluation involves mostly process evaluation of a system during implementation; summative evaluation assesses a system once it has been implemented and operational for a period of time. Grant et al (2002) further suggest that the evaluation be timely, realistic, practical and endorsed by key stakeholders. The current thinking around evaluation of complex health information systems leans towards evaluation geared to performance enhancement and knowledge development, and away from accountability, particularly costing approaches to net benefits assessments. However, accountability remains a strong value in Canadian society in general and increasingly in the health and technology sector, and therefore we recommend that some type of accountability question be included in the evaluation approaches in each jurisdiction.

Step 6: Analyze and Report

Healthfield, Moehr, Lau etc (insert references) have noted that the task of consolidating the findings of a multi- method evaluation is perhaps the most difficult component of the study of complex health information systems. It is likely that most jurisdictions will select one or more evaluation questions to address, and the evaluation effort will consist of several sub components which are in fact separate evaluation projects, involving different methods and disciplines. We recommend that the findings from each evaluation project within the evaluation initiative be shared with those key stakeholders identified in Step 1, preferably in a workshop setting. This approach will permit fuller discussion of the interpretation and implications of the results obtained through different projects, or through the use of multiple methods within each project.

Step 7: Agree on Recommendations and Forward Them to Key Stakeholders

The network of key stakeholders attending the Workshop (Step 6) are also those who should be involved in generating the recommendations which arise from the findings of the evaluation. Those responsible for knowledge-generation oriented studies will have responsibilities to generate recommendations specific to their discipline/field of inquiry. These recommendations may prove to be relatively straightforward and not subject to much broad debate within the evaluation team ; the debates which occur in academic circles may be more contentious but of little direct impact on the evaluation team as a whole. Development oriented studies will face more discussion from the evaluation team and hence disagreements regarding recommendations may arise. Accountability-oriented studies, which impact on all evaluation team members (and on the users and funders of the information system initiatives), can anticipate more lively debate regarding interpretation of findings. Subsequent development of recommendations, particularly if the recommendations arising are negative in terms of continuation of the initiative may be challenging.

There is no guarantee that the process of engagement used to generate the evaluation questions and approaches will ensure a consistent interpretation of what recommendations can be supported by the results. There is however a greater likelihood that common stances on at least some of the key issues will be found if those involved are: (a) familiar with the main issues from the start; (b) aware of the different perspectives each team member brings to the discussion; and (c) comfortable that the variety of methods used in the evaluation produced the most unbiased results possible.

Recommendations

In Canada today, we have a tremendous opportunity to collaborate across jurisdictions and stakeholder groups to develop: (1) a standardized approach to assessment of EHR initiatives and (2) a national inventory of evaluation protocols, instruments and evidence. If the work completed with this project serves as a springboard for discussion among key stakeholders regarding what is important to measure about EHR initiatives in Canada and how to measure it, then we will have laid a foundation upon which common evaluation priorities across jurisdictions can be identified and pursued. Our recommendations are as follows:

• It is crucial that collaborative evaluation efforts around EHR initiatives focus on the functionality of the systems being introduced, as opposed to the specific form of the technology being employed. This is not to suggest that there is no need for evaluation of proprietary technology, but rather to urge the key stakeholders identified in this document to find ways to work together to answer the big picture questions around the implementation of EHRs.
  
• Many important evaluation questions are too costly or complex for one jurisdiction to address on their own. Cross-jurisdictional collaboration in evaluation of EHR initiatives is required to strengthen the generalizeability of the findings and to insure that adequate resources (including research expertise and funding) is available to pursue key questions.
  
• It is important to be clear about what you hope to learn from an evaluation, and your underlying assumptions about what evaluation can and cannot achieve. We recommend use of the accountability/performance enhancement/knowledge development classification of perspectives presented in the proposed evaluation framework to: (1) stimulate discussion among stakeholders about their primary rationale for expending scarce resources on evaluation activities and (b) aid the group to identify the types of tradeoffs which will be required as a consequence of the evaluation questions they pursue and the resources they can access to complete the study.
  
• The process of assembling the Inventory of Electronic Health Records Initiatives Across Canada was extremely time consuming. This inventory should be updated regularly and used as a vehicle for information exchange among the different jurisdictions. Simultaneously, there is a need for an agreement between jurisdictions regarding what types of information they are willing to share and the level of detail that can be made available to other stakeholders. It would be very useful to develop an agreement among the jurisdictions which would see them participate in an annual update of their key initiatives in a standardized format.
  
• This framework document and its companion documents will be widely circulated to : (1) participants in the project; (2) identified EHR initiatives across Canada; (3) and international colleagues for feedback. Comments and suggestions should be directed to:

  Doreen Neville, ScD
  Associate Professor,
  Health Policy and Health Care Delivery
  Faculty of Medicine
  Memorial University of Newfoundland
  
  Email: dneville@mun.ca
  Phone: 709-777-6215
  Fax: 709-777-7382

• Additional dissemination strategies which will be employed include: (1) posting the final report and the 3 deliverables on the NLCHI website; (2) presenting the study findings at local, national and international seminars; (3) submitting an article for publication in a peer-reviewed journal; and (4) submitting hard copies of the final report and deliverables to major health sciences libraries across Canada.

The research team would like to express its sincere appreciation to all the key informants who assisted with the completion of this project.

Bibliography

Adams WG, Mann AM, Bauchner HB. Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics 2003; 111:626-632.

Adragna L. Implementing the enterprise master patient index. Journal of AHIMA 1998; 69(9):46-52.

Aller RD. Creating integrated regional laboratory networks. Clinics in Laboratory Medicine 1999; 19(2): 299-316.

Alvarez R. The promise of e-health - a Canadian perspective. EHealth International 2002; 1(1):4.

Alvarez RC, Zelmer J. Standardization in health informatics in Canada. International Journal of Medical Informatics 1998; 48(1-3):13-18.

Amatayakul M. Critical success factors - steps to take to achieve a truly integrated information system. Health Management Technology 2000; 21(5):14-18.

Anderson J, Aydin C, Jay S. (Eds). Evaluating Health Care Information Systems: Methods and Applications. Thousand Oaks, California: Sage Publications, 1994.

Anderson J, Jay S, Anderson M, Hunt T. Evaluating the potential effectiveness of using computerized information systems to prevent adverse drug events. Proc AMIA Symp 1997; 228-232.

Anderson JG. Evaluation in health informatics: computer simulation. Computers in Biology and Medicine 2002; 32(3):151-164.

Anonymous. Computerized provider order entry systems. Health Devices 2001; 30(9-10):323-359.

Aspinall MB, Whittle J, Aspinall SL, Maher RLJ, Good CB. Improving adverse-drug-reaction reporting in ambulatory care clinics at a Veterans Affairs hospital. American Journal of Health System Pharmacy 2002; 59(9):841-845.

Bakker AR, Leguit FA. Evolution of an integrated HIS in the Netherlands. International Journal of Medical Informatics 1999; 54(3):209-224.

Balas EA. Information systems can prevent errors and improve quality. Journal of the American Medical Informatics Association 2001; 8(4):398-399.

Baldwin FD. Once is enough. Healthcare Informatics 2001(July); 30-33.

Bamford W, Rogers N, Kassam M, Rashbass J, Furness P. The development and evaluation of the UK national telepathology network. Histopathology 2003; 42:110-119.

Baorto DM, Cimino JJ, Parvin CA, Kahn MG. Using Logical Observation Identifier Names and Codes (LOINC) to exchange laboratory data among three academic hospitals. Proc AMIA Symp 1997; 96-100.

Bates D, Leape L, Cullen D, Laird N, Petersen L, Teich J et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Journal of the American Medical Association 1998; 280(15):1311-1316.

Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N et al. The impact of computerized physician order entry on medication error prevention. Journal of the American Medical Informatics Association 1999; 6(4):313-321.

Bates DW, Gawande AA. Improving safety with information technology. New England Journal of Medicine 2003; 348(25):2526-2534.

Bates DW, Pappius E, Kuperman GJ, Sittig D, Burstin H, Fairchild D et al. Using information systems to measure and improve quality. International Journal of Medical Informatics 1999; 53(2-3):115-124.

Bayegan E, Nytro O, Grimsmo A. Ranking of information in the computerized problem-oriented patient record. Medinfo 2001; 10(Pt 1):594-598.

Bayegan E, Nytro O. A problem-oriented, knowledge-based patient record system. Studies in Health Technology and Informatics, Health Data in Information Society (Volume 90), Proceedings of MIE2002 2002.

Bayegan E, Tu S. The helpful patient record system: problem-oriented and knowledged-based. Proc AMIA Symp 2002; 36-40.

Benjamin SD. The electronic clinical patient record. Practical Procedures and Aesthetic Dentistry 2001; 13(9):744-748.

Bingham A. Computerized patient records benefit physician offices. Healthcare Financial Management 1997; 51(9):68-70.

Birkmeyer CM, Bates DW, Birkmeyer JD. Will electronic order entry reduce health care costs? Effective Clinical Practice 2002; 5(2):67-74.

Blaine GJ, Cox JR, Jost RG. Networks for electronic radiology. Radiologic Clinics of North America 1996; 34(3):505-524.

Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? JAMA 2003; 290(2):259-264.

Bomba D, de Silva A. An Australian case study of patient attitudes towards the use of computerised medical records and unique identifiers. Medinfo 2001; 10(Pt 2):1430-1434.

Branger P, Duisterhout J. Electronic data interchange in medical care: an evaluation study. Proc Annu Symp Comput Appl Med Care 1991; 58-62.

Branger P, van der Wouden, Schudel B, Verboog E, Duisterhout J, van der Lei J et al. Electronic communication between providers of primary and secondary care. BMJ 1992; 305(6861):1068-1070.

Brennan S, Dodds B. The electronic patient record programme: a voyage of discovery. The British Journal of Healthcare Computing and Information Management 1997; 14:16-18.

Brown SH, Hardenbrook S, Herrick L, St.Onge J, Bailey K, Elkin PL. Usability evaluation of the progress note construction set. Proc AMIA Symp 2001; 76-80.

Buffone GJ, Petermann CA, Bobroff RB, Moore DM, Dargahi R, Moreau DR et al. A proposed architecture for ambulatory systems development. Medinfo 1995; 8(Pt 1):363-366.

Burgess B, Wager KA, Lee FW, Glorioso R, Bergstrom L. Clinics go electronic: two stories from the field. Journal of the American Health Informatics Management Association 1999; 70(6):42-46.

Burkle T, Ammenwerth E, Prokosch H, Dudeck J. Evaluation of clinical information systems. What can be evaluated and what cannot? Journal of Evaluation in Clinical Practice 2001; 7(4):373-385.

Bush J. Computers: looking for a good electronic medical record system? Family Practice Management 2002; 9(1):50-51.

Canfield K. Clinical resource auditing and decision support for computerized patient record systems: a mediated architecture approach. Journal of Medical Systems 1994; 18(3):139-150.

Carine F, Parrent N. Improving patient identification data on the patient master index. Health Information Management 1999; 29(1):14-17.

Chin HL, McClure P. Evaluating a comprehensive outpatient clinical information system: A case study and model for system evaluation. Proc Annu Symp Comput Appl Med Care 1995; 717-721.

Chin HL, Krall MA. Successful implementation of a comprehensive computer-based patient record system in Kaiser Permanente Northwest: strategy and experience. Effective Clinical Practice 1998; 1(2):51-60.

Chin HL. Embedding guidelines into direct physician order entry: simple methods, powerful results. Proc AMIA Symp 1999; 221-225.

Chronbach LJ. Designing Evaluations of Educational and Social Programs. San Francisco: Jessey-Bass, 1982.

Cimino JJ, Li J, Mendonca EA, Sungupta S, Patel VL, Kushniruk AW. An evaluation of patient access to their electronic medical records via the world wide web. Proc AMIA Symp 2000; 151-155.

Claflin N. Computerized interdisciplinary assessment. Journal for Healthcare Quality 2000; 22(2):25-33.

Connelly DP. Integrating integrated laboratory information into health care delivery systems. Clinics in Laboratory Medicine 1999; 19(2):277-297.

Cook TD, Campbell DT. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Company, 1979.

Covvey HD. IT capabilities for the realization of the laboratory without walls. Proc AMIA Symp 1996; 613-617.

Creighton C. A literature review on communication between picture archiving and communication systems and radiology information systems and/or hospital information systems. Journal of Digital Imaging 1999; 12(3):138-143.

Cupito M. How to find who. Health Management Technology 1998; 19(6):32-37.

Darbyshire P. User-friendliness of computerized information systems. Computers in Nursing 2000; 18(2):93-99.

Dayhoff RE, Kuzmak PM, Frank SA, Kirin G. Extending the multimedia patient record across the wide area network. Proc AMIA Symp 1996; 653-657.

DeLone W, McLean E. Information systems success: the quest for the dependent variable. Information Systems Research 1992; 3(1):60-95.

DeLone W, McLean E. The DeLone and McLean Model of Information Systems Success: A ten-year update. Journal of Management Information Systems 2003; 19(4):9-30.

Donaldson LJ. From black bag to black box: will computers improve the NHS? BMJ 1996; 312(7043):1371-1372.

Doran B, DePalma JA. Plan to assess the value of computerized documentation system: adaptation for an emergency department. Topics in Emergency Medicine 1996; 18(1):63-73.

Doupnik AM. An overview of electronic document management system product offerings. Topics in Health Information Management 2002; 23(1):62-73.

Drazen E. Why don't we have computer-based patient records? Journal of the American Health Informatics Management Association 1996; 67(6):56-60.

Drazen EL, Little AD. Beyond Cost Benefit: An assessment approach for the 90's. AMIA 1992: 113-17

Drazen E, Waegemann CP. Point counterpoint - computer-based patient record. Healthcare Informatics 1998; 15(5):84-96.

Drazen E. Is this the year of the computer-based patient record? Healthcare Informatics 2001; 18(2):94-98.

Effler P, Ching-Lee M, Bogard A, Ieong MC, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from clinical laboratories. Journal of the American Medical Association 1999; 282(19):1845-1850.

Elson RB, & Connelly DP. Computerized patient records in primary care: Their role in mediating guideline-driven physician behaviour change. Archives of Family Medicine 1995; 4(8):698-705.

Endoh A, Minato K, Komori M, Inoue Y, Nagata S, Takahashi T. Quantitative comparison of human computer interaction for direct prescription entry systems. Medinfo 1995; 8(Pt 2):1101-1105.

Fletcher RD, Dayhoff, R.E., Frank S, Jones R, Wu C et al. The integrated multimedia electronic patient record. 1999.

Fletcher RD, Dayhoff RE, Wu CM, Graves A, Jones RE. Computerized medical records in the Department of Veterans Affairs. Cancer 2001; 91(8 suppl):1603-1606.

Forsythe DE, Buchanan B. Broadening our approach to evaluating medical information systems. Proc Annu Symp Comput Appl Med Care 1991; 8-12.

Francis L, Hebert M. Experiences from Health Information System Implementation Projects Reported in Canada Between 1991 and 1997. Journal of End User Computing 2001; 13(4):17-25.

Freriks G. Identification in healthcare: Is there a place for unique patient identifiers? Is there a place for the master patient index? 2000.

Gadd CS, Penrod LE. Assessing physician attitudes regarding use of an outpatient EMR: A longitudinal, multi-practice study. Proc AMIA Symp 2001; 194-198.

Gadd CS, Friedman CP, Douglas G, Miller DJ. Information resources assessment of a healthcare integrated delivery system. Proc AMIA Symp 1999; 525-529.

Gamm L, Barsukiewiez C, Dansky K, Vasey J. Pre- and post- control model research on end-users' satisfaction with an electronic medical record: preliminary results. Proc AMIA Symp 1998; 225-229.

Gardner J. VA leads the way. Modern Healthcare 1997; 27(48):24.

Gates K. Evaluation of a system for electronic exchange of laboratory information: A pre-implementation study. Master's Thesis, Memorial University Faculty of Medicine, March 2004.

Goddard BL. Termination of a contract to implement an enterprise electronic medical record system. Journal of the American Medical Informatics Association 2000; 7(6):564-568.

Golob R, Quinn J. Goals & roles: integrated delivery systems & the master patient index. Healthcare Informatics 1994; 11(11):68-72.

Grant A, Plante I, Leblanc F. The TEAM methodology for the evaluation of information systems in biomedicine. Computers in Biology and Medicine 2002; 32(3):195-207.

Green CJ, Moehr JR. Performance evaluation frameworks for vertically integrated health care systems: shifting paradigms. Proc AMIA Symp 2000; 315-319.

Greenes RA, Peleg M, Boxwala A, Tu S, Patel V, Shortliffe EH. Sharable computer-based clinical practice guidelines: rationale, obstacles, approaches, and prospects. Medinfo 2001; 10(Pt 1):201-205.

Gritzalis DA. Enhancing security and improving interoperability in healthcare information systems. Medical Informatics (Lond) 1998; 23(4):309-323.

Gustafson DH, Hawkins RP, Boberg EW, Bricker E, Pingree S, & Chan CL. The use and impact of a computer-based support system for people living with AIDS and HIV infection. Proc Annu Symp on Comput Appl Med Care 1994; 604-608.

Gustafson DH, Hawkins R, Boberg E, Pingree S, Serlin RE, Graziano F et al. Impact of a patient-centered, computer-based health information/support system. American Journal of Preventive Medicine 1999; 16(1):1-9.

Hammond WE, Hales JW, Lobach DF, Straube MJ. Integration of a computer-based patient record system into the primary care setting. Computers in Nursing 1997; 152(2 Suppl):S61-S68.

Hanmer L. Criteria for the evaluation of district health information systems. International Journal of Medical Informatics 1999; 56(1-3):161-168.

Hassey A, Gerret D, Wilson A. A survey of validity and utility of electronic patient records in a general practice. BMJ 2001; 322(7299):1401-1405.

Hawkins F. Evaluation of clinical documentation before and after EMR implementation. IT Health Care Strategist 2000; 2(12):8-11.

Hawkins HH, Hawkins RW, Johnson E. A computerized physician order entry system for the promotion of ordering compliance and appropriate test utilization. Journal of Healthcare Information Management 1999; 13(3):63-72.

Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316(7149):1959-1961.

Heathfield H, Hudson P, Kay S, Mackay L, Marley T, Nicholson L et al. Issues in the multi-disciplinary assessment of healthcare information systems. Journal of Information Technology and People 1999; 12(3):253-275.

Heathfield HA, Buchan IE. Current evaluations of information technology in health care are often Inadequate. BMJ 1996; 313(7063):1008-1009.

Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T et al. Evaluating large scale health information systems: from practice towards theory. Proc AMIA Ann Symp 1997; 116-120.

Heathfield HA, Pitty D. Evaluation as a tool to increase knowledge in healthcare informatics. Medinfo 1998; 9(Pt 2):879-883.

Herbst K, Littlejohns P, Rawlinson J, Collinson M, Wyatt JC. Evaluating computerized health information systems: hardware, software, and human-ware: experiences from the Northern Province, South Africa. Journal of Public Health Medicine 1999; 21(3):305-310.

Hersh WR, Patterson PK, Kraemer DF. Telehealth: the need for evaluation redux. Journal AMIA 2002; 9(1):89-91.

Hippisley-Cox J, Pringle M, Cater R, Wynn A, Hammersley V, Coupland C et al. The electronic patient record in primary care - regression or progression? A cross sectional study. British Medical Journal 2003; 326(7404):1439-1443.

Hocking J, Brown G, Malyuk D, Ensom R. Computerized integration of pharmacy and laboratory data: a prototype model. The Canadian Journal of Hospital Pharmacy 1993; 46(5):212-214.

Hripcsak G, Wilcox A. Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. Journal of the American Medical Informatics Association 2002; 9(1):1-15.

Jerant AF, Hill DB. Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? Journal of Family Practice 2000; 49(4):349-357.

Johnson SB, Haug P, Curtis C, Defa T, Davoren B, Kolodner R et al. Where are they now? CPR leaders assess their progress. Journal of AHIMA 2000; 71(8):35-39.

Kaplan B. Initial Impact of a clinical laboratory computer system: themes common to expectations and actualities. Journal of Medical Systems 1987; 11(2/3):137-147.

Kaplan B. An evaluation model for clinical information systems: clinical imaging systems. Medinfo 1995; 8(Pt 2):1087.

Kaplan B. Addressing organizational issues into the evaluation of medical systems. Journal of the American Medical Informatics Association 1997; 4(2):94-101.

Kaplan B. Social Interactionist framework for information systems studies: the 4C's. Proc of the IFIP WG 8 2 and 8 6 Joint Working Conference on Information Systems: Current Issues and Future Changes 1998; 327-339.

Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Towards an informatics research agenda. Journal of the American Medical Informatics Association 2001; 8(3):235-241.

Kaplan B, Lundsgaarde HP. Toward an evaluation of an integrated clinical imaging system: identifying clinical benefits. Methods of Information in Medicine 1996; 35(3):221-229.

Kaplan R, Norton D. The balanced scorecard - measures that drive performance. Harvard Business Review. 1992; 70(1):71-79.

Karmel M. The electronic medical record: good-bye paper charts, hello better patient care. Minnesota Medicine 2002; 85(3): 57-59.

Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework. Computers in Biology and Medicine 2002; 32(3):165-177.

Keshavjee K, Troyan S, VanderMolen D. Measuring the success of electronic medical record implementation using electronic and survey data. Proc AMIA Symp 2001; 309-312.

King JA, Morris LL, Fitz-Gibbon CT. How to Asses Program Implementation. Thousand Oaks, California: Sage Publications, 1987.

Kozyrskyj A, Brown T, Mustard C. Community pharmacist perceptions of a provincial drug utilization database. Canadian Pharmaceutical Journal 1998; 131:24-29.

Kozyrskyj AL, Mustard CA. Validation of an electronic, population-based prescription database. Annals of Pharmacotherapy 1998; 32(11):1152-1157.

Krall MA. Acceptance and performance by clinicians using an ambulatory electronic medical record in an HMO. Proc Annu Symp Comput Appl Med Care 1995; 708-711.

Kuhn KA, Giuse DA. From hospital information systems to health information systems: problems, challenges, perspectives. Methods of Information in Medicine 2001; 40(4):275-287.

Kukafka R, O'Carroll PW, Gerberding JL, Shortliffe EH, Aliferis C, Lumpkin JR et al. Issues and opportunities in public health informatics: a panel discussion. Public Health Management Practice 2001; 7(6):31-42.

Kushniruk A, Patel V, Cimino JJ, Barrows RA. Cognitive evaluation of the user interface and vocabulary of an outpatient information system. Proc AMIA Symp 1996; 22-26.

Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Computers in Biology and Medicine 2002; 32(3):141-149.

Kushniruk AW, Kaufman DR, Patel VL, Levesque Y, Lottin P. Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology. M D Computing 1996; 13(5):406-415.

Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems. Proc AMIA Annu Symp 1997; 218-222.

Kushniruk AW, Patel VL. Cognitive evaluation of decision making process and assessment of information technology in medicine. International Journal of Medical Informatics 1998; 51(2-3):83-90.

Kushniruk AW, Patel VL, Cimino JJ. Evaluation of web-based patient information resources: application in the assessment of a patient clinical information system. Proc AMIA Symp 2000; 443-447.

Kushniruk AW, Patel C, Patel VL, Cimino JJ. 'Televaluation' of clinical information systems: an integrative approach to assessing web-based systems. International Journal of Medical Informatics 2001; 61(1):45-70.

Larrabee JH, Boldreghini S, Elder-Sorrells K, Turner Z, Wender RG, Hart JM et al. Evaluation of documentation before and after implementation of a nursing information system in an acute care hospital. Computers in Nursing 2001; 19(2):56-65.

Lau F. Towards a framework for action research in information system studies. Information Technology and People 1999; 12 (2): 148-175.

Lau F, Hebert M. Experiences from health information system implementation projects reported in Canada between 1991 and 1997. Journal of End User Computing 2001: 13 (4): 17-25.

Lenson CM. Building a successful enterprise master patient index: a case study. Topics in Health Information Management 1998; 19(1):66-71.

Lim P. MediNet: Singapore's nationwide medical network. Annals of the Academy of Medicine, Singapore 1990; 19(5):656-661.

Lincoln MJ, Weir C, Moreshead G, Kolodner R, Williamson J. Creating and evaluating the Department of Veteran Affairs electronic medical record and national clinical lexicon. Proc Annu Symp Comput Appl Med Care 1994; 1047.

Littlejohns P, Cluzeau F. Guidelines for evaluation. Family Practice 2000; 17(Suppl 1):S3-S6.

Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lesson still to be learnt. BMJ 2003; 326:860-863.

Litzelman D, Dittus R, Miller M, Tierney, W. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine 1993; 8(6):311-317.

Liu Z, Sakurai T, Orii T, Iga T, Kaihara S. Evaluations of the prescription order entry system for outpatient clinics by physicians in the 80 university hospitals in Japan. Medical Informatics and the Internet in Medicine 2000; 25(2):123-132.

Lock C. What value do computers provide to NHS hospitals? BMJ 1996; 312(7043):1407-1410.

Lovis C, Payne TH. Extending the VA CPRS electronic patient record order entry system using natural language processing techniques. Proc AMIA Symp 2000; 517-521.

MacDonald D. Pharmacist's Expectations of a Pharmacy Network: A Baseline Evaluation. Master's Thesis. Faculty of Medicine, Memorial University of Newfoundland, March 2004.

Malone PM, Young WW, Malesker MA. Wide-area network connecting a hospital drug informatics center with a university. American Journal of Health System Pharmacy 1998; 55(11):1146-1150.

Marshall PD, Chin HL. The effects of an electronic medical record on patient care: clinician attitudes in a large HMO. Proc AMIA Symp 1998;150-154.

Martin-Baranera M, Planas I, Palau J, Miralles M, Sancho J, Sanz F. Assessing physician's expectations and attitudes toward hospital information systems: The IMASIS experience. M D Computing 1999; 16(1):73-76.

Mast CG, Caruso MA, Gadd CS, Lowe HJ. Evaluation of a filmless radiology pilot - a preliminary report. Proc AMIA Symp 2001; 443-447.

Mathews KA. Evaluation of clinical information systems. Nursing Management 1993; 24(7):104-105.

Mattern WD, Scott S. A fully integrated clinical information system to support management of end-stage renal disease. Dis Manage Health Outcomes 2001; 9(11):619-629.

Meyer J. Action Research. In N Fulop, P Allen, A Clarke, N Black (eds) Studying the Organization and Delivery of Health Services. New York, NY: Routledge, 2001.

Mbananga N, Madale R, Becker P. Evaluation of hospital information system in the Northern Province in South Africa. Report prepared for the Health Systems Trust, 2002, Medical Research Council of South Africa.

McDaniel JG. Simulation studies of a wide area health care network. Proc Annu Symp Comput Appl Med Care 1994; 438-444.

Medical Records Institute, 2003. Overview of the MRI fifth annual survey of EHR trends and usage.

Meyers JS. Electronic medical records: 10 questions I didn't know to ask. American Academy of Family Physicians 2001; 8(3):29-32.

Miller R. Reference standards in evaluating system performance. Journal of the American Medical Informatics Association 2002; 9(1):87-88.

Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980 - 1997. BMJ 2001; 322(7281):279-282.

Moczygemba J, Biedermann S. MPIs (master patient index) in healthcare: current trends and practices. Journal of AHIMA 2000; 71(4):55-60.

Modai I, Sigler M, Kurs R. The computerized lab alert system for patient management in clinical care. Psychiatric Services 1999; 50(7):869-885.

Moehr JR. Evaluation: salvation or nemesis of medical informatics? Computers in Biology and Medicine 2002; 32(3):113-125.

Monane M, Matthias D, Nagle B, Kelly M. Improving prescribing patterns for the elderly through online drug utilization review intervention. JAMA 1998; 280(14):1249-1252.

Murff HJ, Kannry J. Physician satisfaction with two order-entry systems. Journal of the American Medical Informatics Association 2001; 8(5):499-509.

Nazi KM. The journey to e-health: VA healthcare network upstate New York (VISN 2). Journal of Medical Systems 2003; 27(1):35-45.

Neame RL, Olson M. Measures implemented to project personal privacy for an on-line national patient index: a case study. Topics in Health Information Management 1996; 17(2):18-25.

Neville D, Keough M, Barron M, MacDonald D, Gates K, Tucker S, Cotton S, Farrell G, Hoekman T, Bornstein S, O'Reilly S. Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Across Canada. March 2004.

Neville D, Gates G, Tucker S, Keough M, MacDonald D, Barron M, Cotton S, Farrell G, Hoekman T, Bornstein S, O'Reilly S. Towards an Evaluation Framework For Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Program Reports. February 2004.

Neville D, Gates G, MacDonald D Barron M Tucker S, Keough M, Cotton S, Farrell G, Hoekman T, Bornstein S, O'Reilly S. Towards an Evaluation Framework For Electronic Health Records: A Proposal For An Evaluation Framework. March 2004.

NHS Information Authority, 2001. PROBE: Project review and objective evaluation for electronic patient and health record projects. Prepared by the UK Institute of Health Informatics for ERDIP.

NHS Information Authority, 2001. Evaluation of electronic patient and health record projects. Prepared by the UK Institute of Health Informatics for the NHS Information Authority.

NHS Information Authority, March 2003. Electronic Record Development and Implementation Programme update. www.nhsia.nhs.uk/erdip/pages/publications/ERDIPUpdateJan03_5.pdf, Accessed September 2003.

Nielsen PE, Thomson BA, Jackson RB, Kosman K, Kiley KC. Standard obstetric record charting system: evaluation of a new electronic medical record. Obstetrics and Gynecology 2000; 96(6):1003-1008.

Ohmann C, Boy O, Yang Q. A systematic approach to the assessment of user satisfaction with health care systems: constructs, models and instruments. Studies in Health technology and Informatics 1997; 43(Pt B):781-785.

Ornstein SM, Jenkins RG, Edsall RL. Computerized patient systems: a survey of 28 vendors. Family Practice Management 1997; 4(10):45-59.

Ornstein SM. Electronic medical records in family practice: the time is now. The Journal of Family Practice 1997; 44(1):45-48.

Ornstein SM, Jenkins RG, MacFarlane LL, Glaser A, Snyder K, Gundrum T. Electronic medical records as tools for quality improvement in ambulatory practice: theory and a case study. Topics in Health Information Management 1998; 19(2):35-43.

Ornstein SM, MacFarlane LL, Jenkins RG, Pan Q, Wager KA. Medication cost information in a computer-based patient record system. Archives of Family Medicine 1999; 8(2):118-121.

Ornstein SM, Garr DR, Jenkins RG, Musham C, Hamadeh G, Lancaster C. Implementation and evaluation of a computer-based preventive services system. Family Medicine 1995; 27(4):260-266.

Osada M, Nishihara E. Implementation and evaluation of workflow based on hospital information system/radiology information system/picture archiving and communication systems. Journal of Digital Imaging 1999; 12(2 (Suppl 1)):103-105.

Ostbye T, Moen A, Erikssen G, Hurlen P. Introducing a module for laboratory test order entry and reporting of results at a hospital ward: an evaluation study using a multi-method approach. Journal of Medical Systems 1997; 21(2):107-117.

Patel VL, Kaufman DR, Allen VG, Shortliffe EH, Cimino JJ, Greenes RA. Toward a framework for computer-mediated collaborative design in medical informatics. Methods of Information in Medicine 1999; 38(3):158-176.

Patel VL, Kushniruk AW, Yang S, Yale J. Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. Journal of the American Medical Informatics Association 2000; 7(6):569-585.

Patel VL, Arocha JF, Kushniruk AW. Patients' and physicians' understanding of health and biomedical concepts: relationship of the design of EMR systems. Journal of Biomedical Informatics 2002; 35(1):8-16.

Poon EG, Kuperman GJ, Fiskio J, Bates DW. Real-time notification of laboratory data requested by users through alphanumeric pagers. Journal of the American Medical Informatics Association 2002; 9(3):217-222.

Powsner SM, Wyatt JC, Wright P. Opportunities for and challenges of computerisation. Lancet 1998; 352(9140):1617-1622.

Protti D, Peel V. Critical success factors for evolving a hospital toward an electronic patient record system: a case study of two different sites. Journal of Healthcare Information Management 1998; 12(4):29-38.

Protti D. A proposal to use a balanced scorecard to evaluate information for health: an information strategy for the modern NHS (1998 - 2005). Computers in Biology and Medicine 2002; 32(3):221-236.

Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthcare Management Forum 2002; 15(2):33-35.

Protti D. The power of principles and premises: using them to help define the EHR. Healthcare Management Forum 2002; 15(3):46-48.

Raschke RA, Gollihare B, Wunderlich TA, Guidry JR, Leibowitz AI, Peirce JC et al. A computer alert system to prevent injury from adverse drug events. Journal of the American Medical Informatics Association 1998; 280(15):1317-1320.

Rehm S, Kraft S. Electronic medical records: the FPM vendor survey. Family Practice Management 2001; 8(1):45-54.

Rigby M, Robins S. Practical success of an electronic patient record system in community care - a manifestation of the vision and discussion of the issues. International Journal of Bio-Medical Computing 1996; 42:117-122.

Rivkin S. Opportunities and challenges of electronic physician prescribing technology. Medical Interface 1997; 10(8):77-83.

Robbins J. The Northern Territory Client Master Index. Health Information Management 1999; 29(1): 35-37.

Robert G, Gabby J, Stevens A. Which are the best information sources for identifying emerging health care technologies? An international Delphi survey. International Journal of Technology Assessment in Health Care 1998; 14(4): 636-643.

Rogers E. Diffusion of Innovation (3rd edition). New York: The Free Press, 1993

Rossi PH, Freeman HE. Evaluation: A Systematic Approach 5. Newbury Park, California: Sage Publications, 1993.

Rothschild J, Lee T, Bae T, Bates D. Clinician use of a palmtop drug reference guide. Journal American Medical Informatics Association 2002; 9(3):223-229.

Sabo D. Clinical information systems: a gateway to the 21st century. Nursing Administration Quarterly 1997; 21(3):68-75.

Sado AS. Electronic medical records in the intensive care unit. Critical Care Clinics 1999; 15(3):499-522.

Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet 1995; 346(8971):341-346.

Safran C, Jones PC, Rind D, Bush B, Cytryn KN, Patel VL. Electronic communication and collaboration in a health care practice. Artificial Intelligence in Medicine 1998; 12(2):137-151.

Sailors RM, East TD. Clinical informatics: 2000 and beyond. Proc-AMIA-Symp 1999; 609-613.

Sarr MG. The electronic environment: how has it, how will it, and how should it affect us? Journal of Gastrointestinal Surgery 2001; 5(6):572-582.

Schiff GD, Rucker TD. Beyond structure-process-outcome: Donabedian's seven pillars and eleven buttresses of quality. Journal on Quality Improvement 2003; 27(3):169-174.

Schiff GD, Klass D, Peterson J, Shah G, Bates DW. Linking laboratory and Pharmacy: opportunities for reducing errors and improving care. Archives of Internal Medicine 2003; 163(8):893-900.

Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA 1998; 279(13):1024-1029.

Schuerenberg K. Electronic records find long-term use: Remote access to patient records enables Denver physicians to provide better services to long-tem care patients. Health Data management February 2003.

Shortliffe EH. The evolution of health-care records in the era of the internet. Medinfo 1998; 9(Pt 1):8-14.

Shortliffe EH. Clinical information systems in the era of managed care. Transactions of the American Clinical and Climatological Association 1993; 105:203-215.

Shortliffe EH, Bleich HL, Caine CG, Masys DR, Simborg DW. The federal role in the health information infrastructure: a debate of the pros and cons of government intervention. Journal of the American Medical Informatics Association 1996; 3(4):249-257.

Shortliffe EH, Barnett GO, Cimino JJ, Greenes RA, Huff SM, Patel VL. Collaborative medical informatics research using the internet and the world wide web. Proc AMIA Annu Symp 1996; 125-129.

Shortliffe EH. The next generation internet and health care: a civics lesson for the informatics community. Proc AMIA Symp 1998; 8-14.

Shortliffe EH, Wiederhold G, Fagan LM. Medical Informatics: Computer Applications in Health Care. Reading, Massachusetts: Addison-Wesley Publishing Company, 1990.

Sittig DF, Kuperman GJ, Fiskio J. Evaluating physician satisfaction regarding user interactions with an electronic medical record system. Proc AMIA Symp 1999; 400-404.

Smith PD. Implementing an EMR system: one clinic's experience. Family Practice Management 2003; 10(5):37-42.

Snaedal J. The ethics of health sector databases. EHealth International 2002; 1(1):6-8.

Soliman F, Soar J. Physician clinical communication systems--an Australian perspective. Journal of Medical Systems 1997; 21(2):99-106.

Soper WD. Why I Love my EMR. Family Practice Management 2002; 9(9):35-38.

Stevens CA, Morris A, Sargent G. Internet health information sources. The Electronic Library 1996; 14(2):135-147.

Talmon J, Enning J, Castaneda G, Eurlings F, Hoyer D, Nykanen P et al. The VATAM guidelines. International Journal of Medical Informatics 1999; 56(1-3):107-115.

Tan LT. National patient master index in Singapore. International Journal of Bio-Medical Computing 1995; 40(2):89-93.

Tang P, LaRosa M, Gorden S. Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions. Journal of the American Medical Informatics Association 1999; 6(3):245-251.

Tang PC, Fafchamps D, Shortliffe EH. Traditional medical records as a source of clinical data in outpatient setting. Proc Annu Symp Comput Appl Med Care 1994; 575-579.

Teich JM. Clinical information systems for integrated healthcare networks. Proc AMIA Symp 1998; 19-28.

Thiru K, Hassey A, Sullivan F. Systematic review of scope and quality of electronic patient record data in primary care. BMJ 2003; 326(7398):1070-1075.

Treweek S, Flottorp S. Using electronic medical records to evaluate healthcare interventions. Health Informatics Journal 2001; 7(2):96-102.

Twair AA, Torreggiani WC, Mahmud SM, Ramesh N, Hogan B. Significant savings in radiologic report turnaround time after implementation of a complete picture archiving and communication system (PACS). Journal of Digital Imaging 2000; 13(4):175-177.

van der Loo RP, van Gennip EMSJ, Bakker AR, Hasman A, Rutten FFH. Evaluation of automated information systems in health care: an approach to classifying evaluative studies. Computer Methods and Programs in Biomedicine 1995; 48(1-2):45-52.

Van der Meijden MJ, Tange HJ, Hasman TA. Determinants of success of inpatient clinical information systems: a literature review. Journal of the American Medical Informatics Association 2003; 20(3):235-243.

Villella R. Lab connections: building a case for a web-based lab results reporting system. Healthcare Informatics 2000; 17(10):119-120.

Waegemann CP. The five levels of the ultimate electronic health record. Healthcare Informatics 1995; 12(11):26-35.

Wager KA, Heda S, Austin CJ. Developing a health information network within an integrated delivery system: a case study. Topics in Health Information Management 1997; 17(3):20-31.

Wager KA, Ornstein SM, Jenkins RG. Perceived value of computer-based patient records among clinical users. M D Computing 1997; 14(5):334-340.

Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. Impact of an Electronic Medical Record System on Community-based Primary Care Practices. The Journal of the American Board of Family Practice 2000; 13(5):338-348.

Walker A. South Australia: best practice guidelines for patient master index maintenance. Health Information Management 1999; 29(1):43-45.

Wang D, Peleg M, Tu S, Shortliffe EH, Greenes RA. Representation of clinical practice guidelines for computer-based implementations. Medinfo 2001; 10(Pt 1):285-289.

Weir C, Lincoln MJ, Roscoe D, Turner C, Moreshead G. Dimensions associated with successful implementation of a hospital based integrated order entry system. Proc Annu Symp on Comput Appl Med Care 1994; 653-657.

Weir CR. Linking information needs with evaluation: the role of task identification. Proc AMIA Symp 19998; 310-314.

Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods of Information in Medicine 2003; 42(1):61-67.

Wenzel GR. Creating an interactive interdisciplinary electronic assessment. Computers, Informatics, Nursing 2002; 20(6):251-260.

Wills S. The 21st century laboratory: information technology and health care. Clinical Leadership Management Review 2000; 14(6):289-291.

Wolfe H. Cost-benefit of laboratory computer systems. Journal of Medical Systems 1986; 10(1):1-9.

Wu SC, Smith JW, Swan JE. Pilot study on the effects of a computer-based medical image system. Proc AMIA Symp 1996; 674-678.

Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada

See also:

• Toward an Evaluation Framework for Electronic Health Records Initiatives: A Proposal For an Evaluation Framework
• Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects

Prepared as a component of the study:

Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects

Doreen Neville, Sc.D.
Montgomery Keough, B.Sc. (Hons.)
Michael Barron, MBA
Donald MacDonald, B.A.
Kayla Gates, B.Sc.
Sheila Tucker, BA (Conj.), B.A. (Hons.), B.Ed., MLIS, CPAD
Sandra Cotton. B.A.
Gerard Farrell, M.D.
Theodore Hoekman, Ph.D.
Stephen Bornstein, Ph.D.
Stephen O'Reilly, MBA

Introduction

Background

An electronic health record (EHR) provides each individual with a secure and private lifetime record of their key health history and care within a health system. The record is available electronically to authorized health care providers and the individual anywhere, anytime, in support of high quality care. Recognizing the importance of the EHR in improving the quality and efficiency of health care, the federal government of Canada, in 2001, established Canada Health Infoway to support and accelerate the development and adoption of interoperable electronic health records solutions across the country. Four core components have been identified as the key building blocks of an EHR by Infoway and the Newfoundland and Labrador Centre for Health Information (NLCHI): (1) a unique personal identifier/client registry; (2) a pharmacy network; (3) a laboratory network; and (4) a diagnostic imaging network.

Towards an Evaluation Framework for Electronic Health Records Initiatives: A Review and Assessment of Methods used to Measure the Impact of Health Information Systems Projects, a project funded by Health Canada, Office of Health and the Information Highway, was carried out between May 2002 and December 2003. The goals of the project were to: (a) review current approaches to evaluating the impact of health information systems (particularly those leading to an EHR); and (b) develop an evaluation framework which addresses the information needs of key stakeholders and the identified best practices in the evaluation of such initiatives. Three deliverables were produced from the project and released as separate (but complementary) documents:

1. Towards an Evaluation Framework for Electronic Health Records: An Inventory of Electronic Health Records Initiatives Across Canada;
2. Towards an Evaluation Framework for Electronic Health Records: An Annotated Bibliography and Systematic Assessment of the Published Literature and Project Reports;
3. Towards an Evaluation Framework for Electronic Health Records: A Proposal for an Evaluation Framework.

This report presents the Inventory of Electronic Health Records Initiatives Across Canada as of August 2003. The project was guided by an advisory committee comprised of key personnel who are leading the work of NLCHI around the development of EHRs, including the Chief Executive Officer, the Health Information Network Project Leader, the Director of Research and Development, the Director of Standards Development, the Director of Communications and Privacy, and the project's principal research investigator.

Process

The multi-step process used to compile the inventory of major Electronic Health Record (EHR) projects across Canada was as follows:

Step 1:

An initial search was conducted using an internet search engine, where each province/territory was used in combination with the term Electronic Health Record and variations thereof (e.g. Health Information System).

Step 2:

The Health Canada website was searched for information related to the development of an electronic health record for each province/territory.

Step 3:

Relevant websites identified through the initial internet search (e.g. Western Health Information Collaborative, Newfoundland and Labrador Centre for Health Information, etc.) were searched for documents related to health information system projects.

Step 4:

A list of key contacts for each province/territory was compiled based on knowledge of titles of individuals that are involved in the development of the provincial Health Information Network (HIN) in Newfoundland and Labrador. Key contacts included, but were not limited to, Assistant Deputy Minister of Health and Community Services equivalents, Secretary to Treasury Board equivalents, Assistant Secretary to Treasury Board equivalents and key individuals at various institutions (e.g. SHIN, NLCHI, etc.). The list of key contacts included some of the individuals that were initially identified as having an interest in the development of health information systems and invited to participate in the project. All individuals were contacted via e-mail (and in some instances telephone) and asked for their help in the development of an inventory of Electronic Health Record (EHR) projects across Canada. In some cases, the individual that was contacted suggested that they would not be helpful in this project and suggested another individual as a more appropriate contact. A revised list of key contacts was compiled.

Step 5:

Based on input from the key contacts and the multi-phase internet search, a draft document highlighting the major EHR projects across Canada was compiled. Each contact received his or her respective provincial component for review and revision.

Step 6:

Based on feedback received from key contacts, a second draft document was compiled. The draft document was sent to all key contacts with the information relevant to their province/territory highlighted. Each individual was asked to advise where information was missing or incorrect.

Step 7:

After final input was received from key contacts, the final document was completed. A special acknowledgement is due to Donald MacDonald and Kayla Gates, whose graduate work helped inform the review of the Pharmacy and Laboratory Networks initiatives.

Summary of Major Electronic Health Record (EHR) Initiatives Across Canada

Unique Personal Identifier/Client Registry

The unique personal identifier/client registry has been recognized as the basic building block for the creation of province-wide electronic health records. While specific functions vary from project to project, ideal functionality includes the ability to cross-reference multiple, person-specific, medical record numbers from multiple health information systems (systems that contain information and data needed to compile a comprehensive electronic health record).

In 2001, the province of Newfoundland and Labrador implemented the first province-wide information system that links all regional health information systems into the initial phase of the provincial electronic health record; the Unique Person Identifier/Client Registry System. This system connects all health regions into a common master client index via a sophisticated HL7 compatible integration engine. Canada Health Infoway recently announced a partnership with NLCHI to further develop the current UPI/Client Registry system into a model that can be adopted by other regional and provincial jurisdictions. The Newfoundland and Labrador client registry has been designated the lead provincial Common Client Registry System project and will be used to develop national standards and proof of concept projects related to the Pan-Canadian Electronic Health Record. This project is underway and is expected to be completed by early 2004.

Client registries are a major focus for the short-term investment strategy of Canada Health Infoway Another key initiative, at the sub-provincial level, is underway in the Capital Health Region, Edmonton Alberta. British Columbia, New Brunswick and Ontario are in varying stages of implementing systems very similar to models implemented in Newfoundland and Labrador and Alberta.

Pharmacy Networks

Information systems developed to capture data related to prescription medications are variable in functions across Canada. In the past, these systems were developed to process claims for government-funded drug programs. Technological advances in the last 10 years now allow for more enhanced functionality of medication systems (Pharmacy Networks). These systems now provide an opportunity to capture real time medication data, which can lead to health, economic and financial benefits for both governments and individual patients (Benefits Driven Business Case, NLCHI, 1998).

Nine of the ten provinces currently have some form of Pharmacy Network. Most of these networks provide an interface between community pharmacists and provincially funded drug programs, for adjudication purposes. The most advanced networks include the ability to provide complete drug profiles to pharmacists at the point of distribution. Such systems have been implemented in four provinces: Alberta (WellNet), Prince Edward Island (Pharmacy Network), British Columbia (PharmNet) and Manitoba (DPIN). Each of these provinces, to varying degrees, have incorporated the following functions in designing their Pharmacy Network: on-line real time adjudication, checks for duplication and double-doctoring, drug utilization reviews, checks for patient eligibility, drug profiles, connection to hospitals and physician offices, and electronic prescribing. A comparative listing of these functions is provided in Table 1, followed by a brief description the four networks.

Five provinces, Newfoundland and Labrador, Prince Edward Island, Manitoba, Alberta and British Columbia, have implemented (or are planning to implement) systems with more comprehensive functional capability. One of the major enhancements found in these systems is the ability to provide real-time patient drug profiles at the time the prescription is filled by the pharmacist.

Table 1 Functions of Selected Provincial Pharmacy Networks

Function	Alberta (2002)	British Columbia (1995)	Manitoba (1994)	PEI (1999)
On-line real time adjudication and transmission	x	x	x	x
Checks for duplication	x	x	x	x
Checks for double-doctoring	x	x	x	x
Provides full retrospective drug use evaluation/review on patient profile	x	x	x	x
Tracks patient's deductible on co-pay	x	x	x	x
Patient eligibility checked	x	x	x	x
Immediately identifies what is and is not a benefit	x	x	x	x
Pharmacare Status	x	x	x	x
Drug Profiles	x	x	x	x
Drug Profiles history on each patient	x	x	x	x
Records Rx dispensed for all or select group of patients	Will record all prescriptions	All prescriptions are recorded	All prescriptions are recorded. Note: not mandatory for aboriginals, but most are recorded.	Will report all prescription
Ability to record non-dispensing events	x
Connected with hospitals	x	x
Connected with physician offices/desk top prescribing	x
Other Notes	Currently in a 6 month pilot stage.		Five year plan.	In the development stage.

Source: Pharmacy Scoping Project Briefing Note, 2002, Newfoundland and Labrador Centre for Health Information

Manitoba

In 1994, the province of Manitoba implemented the Drug Programs Information Network (DPIN). The DPIN system was the first system in Canada that connected all community pharmacies. There are currently no linkages of community physicians to hospitals, although these connections are part of Manitoba's five-year business plan for the DPIN. The DPIN was originally developed to provide complete prescription profiles to pharmacists at the time of dispensing, as well as enhanced drug utilization reviews. The functions of the DPIN system are similar to other provinces with Pharmacy Networks (see Table 1), although in Manitoba it is not mandatory for pharmacists to record prescriptions filled by the treaty status/Registered Indian population (Kozyrskyj, Brown and Mustard, 1998).

British Columbia

The British Columbia PharmaNet initiative was implemented in 1995 in an attempt to contain escalating costs to the government drug program, and to improve the health of the population through the provision of drug therapy decision tools. The network allows for the exchange of medication information between pharmacists and hospital emergency rooms, however there are no linkages to community physicians.

An additional function of the PharmaNet system is the Pharmacare Trial Prescription Program. This module was developed to reduce expenditures for patients who are put on a new medication and for some reason must discontinue its use. A patient is given only a portion of the new drug, and their health care provider then monitors their progress. If for some reason the drug must be discontinued, the full prescription has not been wasted.

Alberta

The Pharmaceutical Information Network (PIN) is being developed in the Province of Alberta as part of the Alberta Wellnet initiative. The objective of the PIN project is to provide health care professionals with the information necessary to make optimal decisions on drug therapy. The network will not only provide adjudication functions for Alberta's government drug plan, it will also connect community pharmacists, physicians and hospitals to allow for the exchange of patient information. This will allow a physician to monitor a patient's current, as well as, historical drug profile, create/modify prescriptions through Computerized Physician Order Entry (CPOE), and access decisions support tools to assist in drug therapy decisions. The PIN project in Alberta was approved for implementation based on the estimated $69 million the province would save annually as a result of a reduction of adverse drug events (Pharmaceutical Information Network - Medication Information Strategy (White Paper), Western Health Information Collaborative, April 2002).

Prince Edward Island

Prince Edward Island is the latest province to begin developing a Pharmacy Network. At present this system is still in the implementation stage and much of the system development documentation is classified as proprietary. It is known that in 1997 the province implemented a Pharmaceutical Informatics Project (PhIP) system, which provided province-wide networking for the submission of pharmacists' claims to the government drug program. In 1999, this system was enhanced to allow fee-for-service physicians to submit medical claims to government for payment. Recently, the Province has started a process towards developing a Pharmacy Network that would enhance the role of the pharmacist by providing comprehensive functionality. It is not unreasonable to assume this system would include similar functions found in the three provinces with established Pharmacy Networks.

Laboratory Networks

Electronic sharing of laboratory information has been identified as high priority by several Canadian provinces and territories. Nine of the thirteen jurisdictions have indicated that they have plans to implement one or more aspects of an integrated laboratory network and five jurisdictions, including Newfoundland and Labrador, Manitoba, Saskatchewan, Alberta and British Columbia, have major implementation initiatives currently underway. In some jurisdictions, the initial focus is on electronic exchange of laboratory orders and results. Other jurisdictions are focusing on enabling immediate access to a patient's lab test history. For most jurisdictions, the ultimate goal is to have province wide integration of laboratory information that will support both the electronic exchange of orders and results between geographically dispersed sites as well as immediate access to a patient's longitudinal history of laboratory services from any site in the province. Direct Provider/Physician Order Entry (POE) has been identified as a long-term goal by some jurisdictions, but has not been the focus of any major initiatives to date.

Since the majority of the jurisdictions are in their planning and pilot implementation stages, detailed documentation is often unavailable. However, an overview of what is known about electronic exchange of laboratory information in Canada is provided below.

British Columbia

In 1998, the HealthNet/BC Project formed the Lab Test Standard Task Group (LTSTG) with representatives from HealthNet/BC working groups, the BC Health Information Standards Council and private sector and provincial labs. In 1999, the BC Lab Test Standard (LTS) was developed to enhance the quality of patient care through the timely exchange of consistent lab data, and to reduce the cost of managing the exchange of laboratory information. Version 1.3 of the Lab Test Standard (LTS) is now available. The standard defines the business and technical requirements for the electronic exchange of lab test data and accounts for all information exchanges that occur from the time an order is issued until the time a final result is received. The Lab Test Standard is based on a set of standard identifiers including the Personal Health Number (PHN), which is the provincial standard for personal identification; the Provider Data Standard (including Provider ID), BC Test Order Codes (BCTOC), the standard for test orders; and LOINC, the standard for reporting of test results. The LTS also provides a comprehensive set of rules regarding ordering lab tests, referring/redirecting orders, requesting order status, reporting results, accessing a patients' lab test history and privacy and confidentiality issues.

A number of BC laboratory systems are currently using the Lab Test Standard. Among these is a private sector province-wide initiative called PathNet. PathNet is a web-based electronic laboratory reporting system that integrates patient laboratory information from multiple participating laboratories, within and across regional boundaries. It allows physicians to access up-to-date laboratory test results, in real-time, for any patient that has had a test completed at any participating laboratory. In addition, PathNet will flag any abnormal test result(s) and allow access to a patient's laboratory test history. Other laboratory systems using the Lab Test Standard include BC Communicable Disease Control (BCCDC) in providing lab test results to the Population Health Information System (PHIS), the federal Canadian Integrated Public Health Surveillance initiative and the BC Cancer Agency (Western Health Information Collaborative, 2002).

Alberta

The primary focus for laboratory information exchange in Alberta has been on results reporting and providing a longitudinal history of lab test results. A joint initiative between the Capital Health Authority (CHA), the Physician Office System Program (POSP), Dynacare Kasper Medical Laboratories (DKML) and Alberta Wellnet has created a laboratory test results repository that will allow physicians to electronically receive and file lab results directly into a patient's record. Presently, the system is for reporting test results only and still requires paper forms to be used for lab requisitions. While physician order entry has been identified as a component of the provincial electronic health record initiative, there is still significant work remaining.

The Capital Health Authority Electronic Lab Results Reporting Project was launched in Northern Alberta in June, 2002 and is anticipated to expand to include other health authorities. Currently, physician office systems can electronically receive requested lab test results and a pilot implementation of the lab results history system is being prepared. With full implementation, it is proposed that access to a patient's lab test history will be available along with the Alberta Wellnet Pharmaceutical Information Network (PIN).

Manitoba

Manitoba is currently developing an integrated multi-site organization known as Diagnostic Services of Manitoba (DSM), which will undertake all provincial laboratory services. The goal of DSM is to avoid future costs through economies of scale in material costs and test utilization.

An integrated province-wide Laboratory and Rural and Northern Imaging Information System (LIS/RIS) is the key infrastructure component required for full functioning of the DSM. The system of interlocking laboratories will use a common set of standards for all associated laboratory procedures. With full implementation, specimens can be collected and prepared in one location, transported to another site for testing, and results will automatically be returned to the originating site in real-time. Only laboratory personnel will have authorized access to the LIS/RIS. All hospital and physician access will be through a data repository or a hospital results reporting capability (Western Health Information Collaborative, 2002).

The initial implementation phase of the DSM will not include automated computer order entry. As order entry capabilities become more available in the province, the LIS/RIS will be expanded to facilitate automated physician order entry. Laboratory and imaging results will be the initial building block for Manitoba's electronic health record. The initial phase of the DSM is expected to be complete by 2004 (Western Health Information Collaborative , 2002).

Saskatchewan

As a component of the Saskatchewan Health Information Network (SHIN), Saskatchewan is planning a province wide web-based capability for laboratory test orders and results reporting, based on the storage and extraction of laboratory data from a central repository. Since the fall of 2000, work has been underway on a multi-regional integrated clinical system project that will integrate applications from several areas including registration, lab, pharmacy and operating room scheduling. All regions involved in the project are implementing a common Laboratory Information System that will help automate the process of ordering, performing and reporting laboratory tests. Systems are being configured to generate HL7 messages to enable information flow between applications into a common view once it has been installed (Western Health Information Collaborative, 2002).

Newfoundland and Labrador

The laboratory information system (LIS) that is currently used by all clinical laboratories within Newfoundland and Labrador is the MEDITECH laboratory information system. The Meditech LIS is a computerized information system that manages laboratory test data throughout the testing process and generates laboratory reports. Within health care institutions in Newfoundland and Labrador, the Meditech LIS is interfaced with the Meditech Hospital Information System (HIS). Each of the Province's Institutional and Integrated Health Boards share a common laboratory coding system and LIS network. In addition, any clinic that has established a connection to the Meditech system within the region can access the laboratory information. However, laboratory information exchange across regions or between the regions and the provincial reference laboratories in St. Johns occurs by fax, telephone, postal service and courier.

Medinet is an interface that enables communication between two heterogeneous laboratory information systems (LIS) where one LIS must be a Meditech system. As a first step towards the province wide integration of laboratory information, the initial implementation of Medinet will enable real-time electronic exchange of laboratory test orders and results between all institutions within the Central East and Avalon regions and the provinces' two largest reference laboratories in St. John's. Successful implementation of Medinet in these regions is expected to lead to province-wide adoption of Medinet and enable real-time exchange of laboratory test orders and results between reference laboratories and health care institutions throughout the Province.

Diagnostic Imaging Networks

In recent years, the Canadian health care environment has witnessed significant investment in Picture Archiving and Communications Systems (PACS). Many institutions are implementing these systems as attachments to already installed diagnostic imaging systems. Diagnostic imaging systems provide the written report associated with images, whether images are electronic or film. Drivers for these systems include reduced cost of film and associated storage, as well as the ability to provide remote diagnosis on images taken from remote locations where imaging equipment is available but a radiologist is not.

With the exception of Manitoba, who indicated that they were in the process of creating an integrated, multi-site diagnostic services network, no other jurisdiction identified diagnostic imaging as the focus of a major initiative during the development of this document. One province, Newfoundland and Labrador, has identified on-line ordering and immediate access to a patient's longitudinal history of diagnostic services from any site in the Province as a later phase of the provincial electronic health record.

Overview of Major Initiatives

Newfoundland & Labrador

Unique Person Identifier/Common Client Registry

The basic building block for the development a provincial electronic health record is a system that enables the collection of information from installed feeder and legacy systems. The province of Newfoundland and Labrador has implemented the first province-wide information system that links all regional health information systems into the first phase of the provincial electronic health record; he Unique Person Identifier/Client Registry System.

The Newfoundland and Labrador Centre for Health Information (NLCHI) recently completed the implementation of a province wide client registry system. This project began in early 2000 and went live in December of that same year. This system connects all health regions into a common master client index via a sophisticated HL7 compatible integration engine. NLCHI sees further development of this system in the short run as a key to the consolidation and provision of clinical information that already exists electronically to health providers throughout the province.

The functionality of the registry includes, but is not limited to:

1. Directory of all people who receive services delivered through the health system (by a registered organization and by a registered provider).
2. The registry provides services:
   1. a. Across multiple care settings
   2. b. Across multiple locations
   3. c. In real time.
3. Uniquely identifies an individual in order to:
   1. a. Provide consistent and accurate identification of a unique person at point of contact with the health system;
   2. b. Across multiple locations
   3. c. Help authenticate a service recipient at the appropriate jurisdiction/organization;
   4. d. Share identification data across jurisdictions/organizations; and
   5. e. Facilitates the sharing of personal health information among authorized organizations and stakeholders.
4. Governs and coordinates the information (i.e. message) flow among the jurisdictional and organizational Client Registries through a detailed message specification in order to:
   1. a. Synchronize and reconcile data with the envisioned Pan-Canada EHR and other organizational/jurisdictional registries;
   2. b. Integrate with a Pan-Canadian EHR.
5. The Registry has the following basic required functionality:
   1. a. Search and display client information;
   2. b. Addition of new clients;
   3. c. Verification of client information;
   4. d. Update client information;
   5. e. Management of duplicate records; and
   6. f. Communication of changes to stakeholders.

Canada Health Infoway recently announced a partnership with NLCHI to further develop the current UPI/Client Registry system into a model that can be adopted by other regional and provincial jurisdictions. The Newfoundland and Labrador client registry has been designated the lead provincial Common Client Registry System project and will be used to develop national standards and proof of concept projects related to the Pan-Canadian Electronic Health Record. This project is underway and is expected to be completed by early 2004. Once complete, the province will be ideally positioned to consolidate patient information from multiple regional systems for the purposes of compiling the provincial electronic health record.

Pharmacy Network

NLCHI has completed a detailed project scope for the development of a province-wide comprehensive pharmacy network. This system is phase two of the provincial electronic health record. The Unique Person Identifier/Client Registry was the first phase of this initiative.

The Newfoundland and Labrador Pharmacy Network (Pharmacy Network) will offer on-line, comprehensive, active medication profiles, as well as drug information and drug interaction databases. The network will provide tools and processes to support prescribing, dispensing, compliance monitoring, research and the formulation of policy regarding prescription medications. Increased access to appropriate medication information will enhance the quality of care, facilitate accountability, and promote cost effective usage of medications.

Functionality

The NPN will link doctors, hospitals, pharmacists and other authorized health stakeholders together into an electronic data network. The proposed functionality includes:

• On-line comprehensive active medication profile
• Prescription monitoring program supported on-line
• Adverse drug reaction recording
• On-line provider lookup
• Compliance monitoring;
• Recording of non-dispensing events
• Electronic Prescribing
• Drug interaction check
• Contra-indication check
• Best choice drug suggestions
• On-line drug monographs
• On-line patient lookup

Stakeholder Input Process

The NLCHI Pharmacy Team consulted with the following stakeholders:

• Pharmacists
• Nurse Practitioners
• Consumer Health Groups
• Department of Health and Community Services
• Newfoundland and Labrador Prescription Drug Program
• Pharmacy software vendors
• National Organizations (i.e. CPA)
• Physicians
• Dentists
• Health Boards
• Department of Human Resources and Employment
• Third party payers
• Research Community

Project Scope Deliverables

The project scope is now complete and project deliverables were presented to the Project Steering Committee of NLCHI in late March, 2003. These deliverables and work products will support the development of the Pharmacy Network and are:

• Conceptual Solution
• Report on other jurisdictions
• Newfoundland and Labrador Prescription Drug Program Analysis
• Legislative review
• Standards review
• Review of benefits
• Architectural Model
• Operations and Maintenance Strategy
• Governance Model
• Implementation Strategy
• Workflow report
• Change Management Strategy
• Privacy Considerations
• Cost Estimates
• Funding Approach
• Request for Information (RFI) findings

Laboratory and Diagnostic Imaging

In Newfoundland and Labrador there are a total of eight institutional and/or integrated Health Boards. Each Board has implemented a regional Meditech Hospital Information System (HIS) which includes a regional Laboratory Information System (LIS). In January 2002, the Department of Health and Community Services implemented the Meditech Laboratory Information System (LIS) at the Public Health Laboratory, a Provincial Reference Laboratory, which completes the computerization of clinical laboratories province-wide.

There is an initiative currently underway that will enable the seamless flow of laboratory information between the regional laboratory information systems and the Province's two main reference laboratories - the laboratory department at the Health Care Corporation of St. John's and the Public Health Laboratory. In this province, the conventional system for exchanging laboratory information between ordering site and reference laboratory forces duplicate entry of information at each site and is dependant on courier and/or postal service for results delivery. By enabling real-time electronic exchange of laboratory orders and results between sites, the new system will eliminate duplicate data entry and will no longer depend on the courier/postal service for results delivery. This initiative is the first step towards province-wide integration of laboratory information. The initial connection was established between the Public Health Laboratory and the provinces largest institutional health board in May 2003.

In a later phase of the provincial EHR, the Unique Personal Identifier (UPI) will enable further consolidation of laboratory information at a provincial level. As a component of the provincial EHR, physicians will be able to electronically order laboratory and other diagnostic services on-line, receive online decision support at the time of request and have immediate access to a patient's longitudinal history of diagnostic services from any site in the Province.

Nova Scotia

Unique Patient Identifier

In Nova Scotia, a Unique Provincial Health Identifier (UPHI) has been established using the provincial Health Card Number (HCN). The HCN is collected at the time of service registration for all health service recipients. For non-eligible individuals (e.g., RCMP, out-of-province, etc.) a new HCN is assigned. Implementation of the UPHI has started with the acute care sector by providing online access to the provincial HCN Master File for hospital registrations. Specifically, the Nova Scotia Hospital Information System (NShIS), which serves all hospitals outside the Capital District Health Authority (CDHA), has HCN access integrated into the registration process.

Common Client Registry

A Common Client Registry is being planned which will:

• build on the UPHI infostructure, and its integration with the provincial NShIS;
• utilize work done on standards and interoperability for the NShIS and Canadian Health infostructure Partnership Program (CHIPP) projects; and
• fully integrate with acute care registration (NShIS, CDHA and IWK Health Centre) and other health service registration points.

Pharmacy Network

A pharmacy/medication order-entry-results system with clinical decision support within hospitals has been identified as a common opportunity with other Atlantic Provinces to:

• conduct an environmental scan;
• do a technology assessment;
• define common requirements;
• develop a stakeholder communication strategy;
• develop a stakeholder engagement strategy;
• develop common information, technology and data exchange standards;
• create a governance structure; and
• recognise policy issues surrounding requirements.

Laboratory

An end-to-end, all-in lab information system is planned for Nova Scotia hospitals (all labs in Nova Scotia are in-hospital). The laboratory information system will include scheduling and order entry/results reporting and will be available within institutions and physician offices.

New Brunswick

Client Registry

In New Brunswick, the Client Service Delivery System (CSDS) is a strategic initiative supporting the management and delivery of services to clients across a large number of community based program areas offered by the Department of Health and Community Services (newly restructured into two Departments; the Department of Health and Wellness, and the Department of Family and Community Services.)

The CSDS is a comprehensive, integrated software system shared by Family and Community Services, Mental Health and Public Health, Financial Services and Partners for administering and efficiently delivering services to clients. It will also provide professional workers with a sophisticated system for assessing client needs and eligibility, and for planning and delivering services and ensuring regular follow up. Client services can be managed across programs, divisions and partner organizations. The standards incorporated into the CSDS allows for future and additional sharing of information across multiple data stores.

Scope

The CSDS supports case management activities of front line professionals in assessing client needs and planning, providing and monitoring services delivered to clients in various programs. Incorporated into the CSDS is the Common Individual Registry which provides a unique identifier and basic demographic data on all residents of New Brunswick. The Common Individual Registry is available to all professionals accessing the CSDS.

The case management processes supported by CSDS include:

• Common Individual Registry
• Case and Service Planning and Court/Legal Activities
• Financial Management activities
• Intake and Assessment activities
• Resource Management activities

The programs supported by CSDS include but are not limited to: Early Childhood Initiative, Immunization Programs, Well Child Clinics, (Public Health), Child Protection, Child Care, Adoption, Community Based Services for Disabled Children, Adult Protection, Long Term Care Program and Family and Community Social Services and Mental Health's Acute and Long Term Programs for Adults and Children. The CSDS will provide a common automated infrastructure for the various programs and can be tailored to satisfy the unique requirements of the various service delivery areas. This system will be bilingual and available 24/7. It will provide on line access to program standards, policies, procedures, and help features. It will also provide portable assessment tools through the use of laptop computers.

Current / Prior Initiatives

The restructuring of New Brunswick's Department of Health and Community Services may impact the remaining development and implementation of the CSDS. The primary stakeholders will now reside in two different departments.

Existing Infrastructure

To support this initiative, new infrastructures were implemented province-wide to be used by approximately 2,000 workers.

Key Stakeholders

Family and Community Social Services, Public Health Services, Mental Health Services and Financial Service Divisions are the primary stakeholders for this system. The CSDS will also be used by VON nurses and partners from Extra-Mural Services.

Constraints

Restructuring of New Brunswick's Departments with the Family and Community Social Services moving to another new Department may affect the continuation and the scope of this project.

Business Case

The CSDS was one of the Corporate Technology Initiatives identified in the Department's Strategic Information Plan completed a few years ago. The purpose of this project was to develop an integrated computer system that would support the case management activities of all departmental professionals providing services to clients, regardless of the program area. It will also replace five legacy systems having different technologies and data stores.

Current State of Readiness

The CSDS Project is developed and implemented by components called releases. Eleven releases are planned. Nine releases are currently developed and partially implemented. Development of the last two releases is underway.

Effect on Patient Care

This new system will provide a single source of information for the registration of clients and provide faster response to clients at risk. It will reduce the amount of time to conduct an in-home assessment and reduce inefficiencies and redundancy in the collection of data. It will also provide improved coordination of service planning and delivery to clients across programs and between partners. It will provide improved matching of client needs to available services. Overall, this system will provide the ability to coordinate and track client activities and services over time as well as across program areas.

Prince Edward Island

Unique Personal Identifier

In Prince Edward Island (PEI), the Provincial Health Number (PHN) uniquely identifies a person entitled to receive health and social services and is assigned under the authority of the Minister of Health and Social Services. PEI has enabling health number legislation - the Provincial Health Number Act. Under the terms of this Act, no person shall require the production of another persons health card, collect or use another person's health number; except, where authorized by the Minister or for administration, planning or research as prescribed in the regulations for the health and community services Ministry. The PHN has been implemented across the province.

Common Client Registry

The Common Client Registry (CCR), a foundation piece for the electronic health record, was implemented in April 2002. The CCR uses the PHN as the key for the registry. The CCR is a central registry of identification, contact, demographic, certain filtered eligibilities and certain filtered encounter information for the clients/patients of the health system, including:

• A PHN (unique identifier), for each individual resident and non-resident client/patient;
• Legal name and names by repute;
• Gender;
• Date of birth, age (calculated), date of death;
• Other ID numbers;
• Addresses (multiple current addresses if applicable), address history;
• Home and business mailing address, phone, fax;
• The household of which the client/patient is a member and name of the primary contact for the household;
• Program eligibility (health services programs, eligibility indicators, eligibility/ineligibility dates, full history);
• Encounter history (health services/locations, encounter dates); and
• Access controls, access audit trails, user profiles.

Business Case

The business objectives of the CCR are:

• identify clients uniquely throughout the health information systems by generating a unique PHN;
• provide cross-referencing capability between health information systems;
• provide a history of client encounters with the health system;
• establish a centrally maintained database of basic client demographic information; and
• reduce redundant data maintained by various information systems within the health system.

Scope

The CCR is a database of common client/patient information used by the following health and social services applications:

• Vital Statistics System
• Claims Processing System (CPS)
• Admission/Discharge/Transfer System (ADT)
• Pharmaceutical Informatics Project (PhIP)
• Case Management System (CM)

These applications may have other applications linked to the CCR through them (e.g. ADT links to other hospital systems). The CPS will remain responsible for all information related to Medicare eligibility. The Vital Statistics System will remain the legal repository of all information covered by their current legislation relating to births, legal names and marriage registration. All birth identities of adopted persons within this system will be filtered and only accessible with designated persons with Adoption Services and Vital Statistics with access rights to the encrypted cross references. The CCR will be the central data repository for demographic data on all clients receiving service from the PEI Health and Social Services system. Other applications/data repositories will be integrated over time.

Current / Prior Initiatives

The Client Services Delivery Network (CSDN), developed for Veterans Affairs Canada by Electronic Data Systems (EDS Canada), was used as a basis for the CCR in PEI.

Existing Infrastructure

The CCR was initially populated with converted data from the CPS and then further populated with data from the Vital Statistics System.

Key Stakeholders

The PEI Department of Health and Social Services, EDS Canada and DeltaWare have signed a contract for the adaptation of the CSDN. Other stakeholders include the stakeholders of the above health and social services applications.

Current State of Readiness

This application went live April 2002.

Effect on Patient Care

It is anticipated that the CCR will improve health and social services management in PEI by:

• elimination of data variances/errors in common data among satellite systems;
• integration of the satellite systems via shared eligibility and encounter data, resulting in more efficient and effective operation of each satellite application and the overall system; and
• establishing a unique identifier for each individual client/patient, resulting in improved operations and helping eliminate the errors (and potential fraud) resulting from multiple identities for the same person.

Pharmacy Network

The Pharmaceutical Informatics Project (PhIP) will link physician offices, pharmacies and other health stakeholders into an electronic data network. The PhIP will provide the tools and processes to support prescribing, dispensing, compliance monitoring, research and the formulation of policy regarding prescription medications, thus capturing all drugs for all people. All patient information would be instantly updated as the system will access the CCR.

Scope

This project will be implemented in three phases:

Phase 1: implemented in 1997, a Province-wide Pharmaceutical Informatics Project system to connect all retail pharmacies for the electronic submission of claims administered through the Provincial Pharmacy Program to a central repository.

Phase II: currently underway, will be implemented in spring 2003, and involves replacing the current technology with an enhanced claims processing system. The new system will also perform drug utilization reviews, access the CCR and include functionality to receive claims for 'All Drugs-All People", and will include the availability of the drug profile through an URL.

Phase III: will include the deployment of the URL to hospital pharmacies, emergency rooms, retail pharmacies and physician offices. This phase will include the implementation of "All Drugs - All People" capturing all drugs dispensed by all pharmacies within the province. It will also include the integration with physician and pharmacy vendor products. This system will form the basis for electronic transmissions of prescriptions from the physician to the pharmacist. The physician will have the ability to register prescriptions from their offices and retail pharmacies will provide dispensing information of all prescriptions filled.

Current / Prior Initiatives

In 1999, The Department of Health and Social Services implemented a generic web-based Claims Processing System (CPS) with the Medicare Office for physician's claims being the first program to use the CPS System.

Functionality

The PEI Pharmacy Network, once fully implemented, will link physician offices, emergency rooms, acute care and retail pharmacies. The specific functionality will include:

• Electronic submission of prescriptions to pharmacies
• Drug utilization reviews
• Drug to drug interactions
• Drug allergy interactions
• Step therapy
• Co-ordination of benefits
• Network enabled electronic fax submission of prescriptions to pharmacies
• Physician access to on-line drug profile for patients from office/community
• Retail pharmacist access to on-line drug profile for patients
• Hospital access to on-line drug profile for patients
• Pharmacist and physician oriented decision support

Key Stakeholders

The main stakeholder groups are patients, retail and hospital pharmacists, pharmacy vendors, physicians and the Medical Programs Division of the Department of Health and Social Services.

Effect on Patient Care

The Pharmaceutical Informatics Project system will benefit clients and providers by:

• assisting pharmacists and physicians in monitoring patients with multiple prescriptions;
• alerting pharmacists and physicians to potential drug interactions;
• preventing over prescribing of drugs;
• facilitating quick access to information required in emergency situations;
• improving access to government drug benefits plans; and
• improving access to information on the medication use of residents and the effects of these medications on their health.

Quebec

Client Register

A unique identifier for each citizen of Quebec is provided by the Régie de l'assurance maladie du Québec (RAMQ). Established in 1969, the RAMQ was assigned the task of implementing the provinces Health Insurance Plan, administering it, and providing universal access to healthcare services. Over the years RAMQ has evolved considerably and today, administers many different programs in the area of health insurance. Under the authority of the Minister of Health and Social Services (Sante et Services Sociaux), the RAMQ administers the Health Insurance Plan, the Public Prescription Drug Insurance Plan, and over 40 programs entrusted to it by the government. In addition, the RAMQ's databases contain a wealth of information on health and social services programs. All public institutions in the health and social domain are connected to the Reseau de télécommunication sociosanitaire (RTSS), a dedicated and secured network that allows the exchange of information between participating institutions.

Pharmacy Network

Every pharmacy is connected to a dedicated network linked to the RAMQ.

Laboratory

A computerized system is being implemented in every public laboratory.

Ontario

Background

The Ontario Health Insurance Plan (OHIP) is an individual-based registration system. A unique ten-digit health number is assigned to an insured person (within the meaning of the Health Insurance Act, i.e., a health number is only issued to an individual that is registered for provincial health insurance.

The Ministry of Health and Long-Term Care's (MOHLTC) Registration and Claims Branch manages the client registry. This registry contains personal health registration information such as the individual's legal name, date of birth, gender, address, telephone number, language preference and Canadian citizenship/immigration status. The Registration and Claims Branch also processes claims for insured services.

Ontario has recognized that integrated care can only be fully supported through an integrated, electronic health information system. This system must be able to match information from multiple care environments to the correct patient. Because an individual receives a range of health care services from multiple providers, multiple identifiers such as hospital cards are often generated. These facts, together with the need for relevant, integrated information, form part of the compelling argument for a unique patient identifier (UPI) common to the health care system.

The government established the Smart Systems for Health Agency (SSHA) in 2002. One aspect of SSHA's mandate is to provide the information technology infrastructure that will permit the approved sharing of information, such as patient health care data, in a secure and confidential environment. It is expected that the Smart Systems for Health (SSH) will be Ontario's health information network.

Unique Patient Identification

Unique patient identification is a cornerstone of Ontario's e-Health initiatives. Examples of these initiatives in Ontario include: SSHA secure infrastructure, Ontario Family Health Network and e-Physician Project, voluntary Emergency Health Record, Ontario Laboratory Information System, Community Care Connects, Integrated Services for Children, Ontario Drug Network, Telehealth/Telemedicine and Point-of-Service Verification.

The MOHLTC is considering a unique patient identification system for use across the continuum of care. The diagram below illustrates the conceptual model of the Ontario UPI.

The MOHLTC is looking at design considerations that would be required to complete the development phase for the UPI. These include:

• reviewing business processes and supporting information technology infrastructures for the creation, assignment and maintenance of a provincial unique patient identifier, regardless of funding/payment models (i.e. direct patient payment or health insurance plan);
• reviewing current policy and legislative requirements for a provincial unique patient identifier and recommending the preferred approach to achieving the necessary policy and legislative framework. Privacy and consent will be important aspects of this framework;
• collaborating, communicating and coordinating with key stakeholders, both internal and external to the Ministry, who have an interest in, or are impacted by, a provincial UPI infrastructure;
• determining best practices, levering existing opportunities and building on what currently exists (including re-use) will be important operating principles; and
• recommending conceptual and logical designs for a UPI, a governance and management structure, a policy and legislative framework, and a development and implementation plan.

Provider Registry

Provider Identification

Each health care provider registered with the MOHLTC in Ontario is assigned a unique identifier called a Stakeholder Number (SN). The SN is a ten digit number and is randomly generated by the provider registration system. A SN is assigned to an individual stakeholder only once; this is of particular importance given that an individual may be licensed in more than one health profession (e.g., medicine and dentistry), and the SN is the means by which two licenses within the same SN are linked.

A billing number for the submission of claims or service encounter data for payment purposes may be given to a stakeholder (either an individual or organization) upon registration on the corporate registration database. An individual stakeholder can have more than one billing number.

Corporate Provider Database

The Provider Registry is a relational integrated repository of provider information. The registry contains information about Ontario health care providers and is used for operational and planning purposes. These providers include, but are not limited to, physicians, physiotherapists, optometrists, chiropractors, nurse practitioners, midwives, private medical laboratories and hospital groups. The database includes providers from fee-for-service practitioners, family health networks/groups and alternate payment programs.

Pharmacy Network

The Ontario Drug Benefit (ODB) Health Network System (HNS) is a province-wide network that connects prescribing physicians, retail pharmacies, hospital outpatient dispensaries and selected long-term care dispensaries. The ODB network allows for drug utilization reviews and review of drug-to-drug interactions, with its primary purpose being the adjudication of claims for eligible recipients.

Physicians and pharmacists are assigned a unique identifier. Providers' license numbers are also recorded on the system. Dispensing locations are uniquely identified through known addresses and cross-referenced to system identifiers. Eligible recipients are identified by their health number or by a temporary eligibility number in the case of some social assistance recipients.

Claim record histories are maintained on the HNS and are used for claim adjudication and reporting purposes. Access to claim information and data extracts are governed by the Freedom of Information and Protection of Privacy Act (FIPPA). Pharmacists only have access to claims information submitted at their own pharmacy. The collection of ODB claim information is administered under the Ontario Drug Benefit Act.

Currently, only prescribing physicians have access to the ODB Network.

At present, the ODB network operates on an X.25 network. Data are stored in an MOHLTC data centre. With the implementation of SSHA, this application will move to the secure managed SSHA network. The technical operation of the network will convert to the SSHA-managed private network, while the application will remain under the control of the MOHLTC.

Over the next 18 months the ODB network is being converted to use the SSHA Transfer Control Protocol/Internet Protocol (TCP/IP) managed network. This will improve reliability, availability and security. It also puts the ODB application onto a network that will be connected to 150,000 health care providers over the next 5 years.

Ontario anticipates deploying a Primary Care IT Program directed at family physicians in a primary care renewal group (e.g. Family Health Networks) later this year. These physicians will likely be future users of the information contained on the ODB network. The target in the next two years is to have 300 Family Health Networks, averaging 10 physicians each, plus other primary care providers (e.g., nurse practitioners) participating. Family Health Networks will be located across the province.

Another important group of users will be physicians working in hospital emergency rooms. These hospitals are currently connected to the secure SSHA network and access to ODB recipient drug histories has been identified as a top priority for rollout to these locations. A principal benefit is enhanced patient care and safety.

Over the longer term, Ontario's strategic directions include expanding the ODB network to include all prescription drugs for all clients receiving services in Ontario.

Laboratory

The Ontario Laboratories Information System (OLIS) project is sponsored by the Laboratories Branch within the MOHLTC and SSHA. The objective of OLIS is to implement a single integrated provincial laboratory information system that would allow all laboratory information to be electronically exchanged between practitioners and laboratory providers and provide the Ministry with program management information.

Since the OLIS is one of several SSHA initiatives to link all health care providers, it will use SSHA infrastructure such as central data centres, secure network, public key infrastructure (PKI), registration, and Tier-1 help desk services.

Once fully implemented, practitioners (e.g., physicians, nurse practitioners, midwives and dentists) will be able to order laboratory tests electronically for patients. Specimen collection centres will be able to retrieve orders and enter specimen information. Laboratory service providers (e.g., community labs, hospital labs, public health labs and practitioners) will be able to enter test results information and ordering practitioners will be able to retrieve results electronically. With a patient's explicit consent, a practitioner will also be able to view test results ordered by other practitioners.

Through central repositories, the OLIS will capture and make available information about test orders, specimen information and test results to authorized practitioners and laboratories. This data will be communicated to and from the OLIS by authorized client systems using HL7 format standard OLIS transactions. The OLIS standards for HL7 will be compatible with versions 2.X and 3.X of HL7.

OLIS will create interface specifications that developers of external client software will use to interface their systems with the OLIS. These specifications shall describe the HL7 messages for ordering tests, entering specimen and results information and for inquiring about results. It will also include specifications regarding connectivity to the OLIS using the secure SSHA network.

It is expected that practitioners and laboratory service providers will use either their CMS or LIS software to communicate with the OLIS using an OLTP interface or a web-based interface that will be developed.

The OLIS will support all business functions related to exchanging laboratory test information, including:

1. Order Information - The ability of a practitioner, specimen collection centre, or a laboratory to enter, amend, cancel, view, or retrieve an order. Data required for this includes patient and practitioner identification information, and tests requested.
2. Consent Information - The ability for a practitioner to indicate a patient's consent to view results of tests ordered by other practitioners.
3. Specimen Information - The ability of a practitioner, specimen collection centre, or a laboratory to enter, amend, view or retrieve specimen information. Data required for this includes the collection date and time, and destination lab information.
4. Results Information - The ability of a laboratory service provider or practitioner to enter and view information about a test result.
5. Results Retrieval - The ability of a practitioner to retrieve results of completed tests. A practitioner may retrieve results for a single patient or all results for tests ordered over a period of time.
6. Inquiries - The ability for a stakeholder to inquire about information on an order, patient, or tests. Examples of inquires that a practitioner may submit include: all results for a patient, a specific test ordered or status of tests submitted for all patients. Examples of inquiries that a laboratory may submit include: list of tests to be completed and results not yet retrieved by submitting practitioners.

The OLIS will develop a web based facility for exchanging health knowledge and education information. This includes code table updates (e.g. LOINC, order nomenclature), information and educational material. To make effective use of the OLIS and to avoid unnecessary error messages, code table updates will also be available for use and reference by practitioners and laboratories.

Manitoba

Unique Identifier

Person Identifiers

Manitoba implemented the Personal Health Identification Number (PHIN) in 1984. The PHIN is the personal health identification number assigned to an individual by the Ministry of Health to uniquely identify an individual for health care purposes. At the present time PHINs are only issued to those who are eligible for health care benefits. The collection, use and disclosure of the PHIN and other personal health information is regulated by Manitoba's Personal Health Information Act.

Manitoba Health's Information Systems Branch has been participating in the Canadian Institute for Health Information's (CIHI) Unique Identifier Roadmap Project, which encompasses client, facility and jurisdiction unique identifiers. In the context of this initiative, a jurisdiction refers to the 19 organizations that uniquely identify clients of health services in Canada. These include the ten provinces, three territories and the six groups whose public health care responsibilities fall under the federal government.

Provider Identifiers

Manitoba is participating in the Western Health Information Collaborative (WHIC) Provider Registry System project, a standards-based repository of core provider data that is supplied by regulatory or recognized health care organizations and made available to authorized consumers to facilitate controlled exchange of health information.

Location (Delivery Site) Identifiers

A key infrastructure component of an electronic health record will be the ability to uniquely identify the place or places which a client received, is receiving, or is planning to receive health services. As a first step to the development of a Delivery Site Registry, Manitoba Health participates in the Delivery Site Identifiers Initiative being lead by the CIHI. This initiative is looking at a standards-based approach which will result in the classification, definition and unique identification of health services "places".

Comments

Manitoba Health provides extensive feedback to CIHI on this concept and requirements. Based on information distributed by CIHI for review and subsequent feedback, it is anticipated that the Unique Identifiers Initiative will encompass the classification, definition and identification of all types of places that provide health services.

Client Registry

Manitoba Health's Registration System is used to maintain a registry of all Manitobans who may be eligible for health insurance benefits, and some non-Manitoba residents who may have received health services in Manitoba. This registry is extensively used by various claim-processing systems to verify eligibility requirements. In this registry, every family in Manitoba is assigned a family number and every resident a personal health identification number. As more and more claim processing and health information systems go on-line, this registry is being extensively used as the "hub" of patient identification across these systems and thus helps insure health systems integration.

No regional client registries exist at present. Manitoba is planning a comprehensive provincial client registry for use at the regional and provincial level, which will include:

• Direct access with validation
• Full enterprise master patient index
• Centralised identifier
• Registration, admission/transfer/discharge
• Integration capability
• Messaging exchange
• Link to legacy systems

This project is planned to proceed to development and implementation this year.

Pharmacy Network

Overview

The Drug Programs Information Network (DPIN) was introduced in 1994 to provide better service and to connect all pharmacies in Manitoba to a central database. Pharmacies record information on prescription drugs dispensed in all Manitoba pharmacies by sending the information to the DPIN. The DPIN processes all Pharmacare claims and provides pharmacists with an instant calculation of the payment required by the patient.

The DPIN is connected through a virtual private network (VPN) to over 275 retail pharmacies throughout the province. Pharmacy software management systems communicate with the DPIN via an expanded CPhA version 3 message standard. Though not legislated in Manitoba, 95% of all retail prescription drug medications are entered into DPIN for Manitoba residents. At the time of filling the prescription in the pharmacy, the pharmacist is made aware of adverse drug event information and whether the individual or the drug program is responsible for payment on the prescription. The DPIN process approximately 15 million transactions per year.

Current Status

Manitoba has four Drug Programs in operation within the DPIN:

i.) Pharmacare Drug Plan -- for all Manitoba residents, income based. The family or individual must complete an application form and provide all income information. This drug plan pays 100% of the cost on formulary drugs after reaching the deductible.

ii.) Personal Care Home Residents Drug Plan -- 100% coverage on formulary drugs.

iii.) Family Services Social Assistance Drug Plan -- 100% coverage on formulary drugs.

iv.) Home Palliative Care Drug Plan -- 100 % coverage on extended drug formulary. This plan started in December, 2002.

Drug Programs Information Network - Emergency Rooms (DPIN-ER):

Based on the prescription drug data from the DPIN application, Manitoba Health provides patient drug medication views to hospital emergency rooms (DPIN-ER) (and also to a small number of hospital general admissions areas). DPIN-ER went live in 1997 and is a web-based interactive information system connected through the Provincial Data Network (PDN) to 81 hospitals throughout the province. User access is authenticated through a security database and all transactions are encrypted. At the time of patient presentation to the hospital emergency room, authorized hospital personnel use the system to verify patient drug utilization. DPIN-ER processes approximately 231,000 transactions per year.

Provincial Data Network (PDN):

The PDN is a single Wide Area Network (WAN) connecting Government Departments, Agencies and the Health Care Sector. It is a hybrid of frame relay, broadband, and satellite communication technologies and is operated as a service by Manitoba Telecommunications & Network Services department. This WAN provides the capacity for existing systems and is designed with expansion capability as systems are added. This is an IP network that provides each site with an access device (i.e.: router/switch), some fault tolerance, redundancy and network management.

WHIC Pharmaceutical Management Strategy:

Manitoba is participating in the WHIC Pharmaceutical Management Strategy which aims to improve quality of patient care for clinical pharmacy management by adding clinical pharmaceutical management functions -- dispensing (and eventually prescribing) to existing claims adjudication and administration.

Manitoba's existing Drug Program Information Network currently provides dispensing and notification of adverse drug interactions at the retail pharmacies including claims adjudication and administration. Manitoba will explore opportunities for collaborative efforts on the WHIC Pharmaceutical Management Strategy and the potential to leverage the order entry/prescribing component of the Alberta DPIN project as part of an overall integrated provider solutions strategy for primary health care physicians.

Provider Access Demonstration Projects:

Manitoba is looking to pilot patient drug medication viewing in physicians offices as part of the Provider Access Demonstration Projects which are being undertaken through the Primary Health Care Transition Program.

Laboratory and Diagnostic Imaging

Overview

The Laboratory Information System (LIS) is a comprehensive, electronic information system that computerizes clinical, clerical and administrative functions within the Lab site. The Radiology Information System (RIS) is a comprehensive electronic information system that computerizes clinical, clerical and administrative functions within the imaging (e.g., X-ray, CT scan, MRI) site.

Both the LIS and RIS are integral parts of a comprehensive electronic health record. The vision for these systems is that they will be integrated with the Admission-Discharge-Transfer (ADT) systems in Manitoba, as well as with other core registries (e.g., Provider, Location and Client). Eventually, the information contained in LIS/RIS systems will be available to providers from all levels of care, allowing for seamless healthcare delivery as patients receive services from a variety of service channels throughout their lifetime. The longitudinal aggregation of the information contained in these systems will form the basis for decision support systems that support evidence based policy decisions concerning utilization of healthcare services, resource allocation and other aspects of health system planning.

Current Status

Manitoba is in the process of creating a single integrated, multi-site, publicly owned organisation to be known as Diagnostic Services of Manitoba (DSM). This organisation will undertake all provincial laboratory services, including rural and northern imaging services. DSM will be created from elements of existing public sector laboratories and related systems. It will include all Winnipeg public hospital laboratories, Brandon Westman Laboratory, all Rural and Northern laboratory and imaging services and Cadham Provincial Laboratory Services. The organization will consolidate operations and staff, establish the necessary facilities and reorganize equipment to run the operation. The goal is to avoid future costs through economies of scale in material costs and test utilization.

A system of interlocking laboratories will be formed throughout the Province. These laboratories will use one set of standard operating procedures, common analyzer platforms, integrated test ranges, a province-wide integrated transportation system and a province-wide integrated Laboratory and Rural and Northern Imaging Information System. High-volume and complex tests performed manually will be carried out at three main sites (two in Winnipeg and one in Brandon). Second-level laboratories will operate in the Winnipeg community hospitals and at the larger rural sites, while a base-level laboratory system will be in place at all hospitals in the Province.

The presence (or absence) of a type of testing at a particular site will continue to be determined by the Regional Health Authority's (RHAs) when they set their clinical requirements. As a matter of principle, DSM will not unilaterally change the availability of clinical services by removing access to testing.

An integrated Manitoba-wide laboratory and rural and Northern Imaging Information System (LIS/RIS) is the key infrastructure component required to enable the formation and function of DSM.

Although Manitoba will be well down the road to having standard results reporting within their Provincial LIS, there is still the issue of private labs that needs to be accommodated within the local scene. Further, the vision of DSM is that access to the LIS repository of information will be limited to laboratory personnel only. Physician and hospital access will be directed to a data repository or through a hospital results reporting capability. The most promising tool for individual physician offices is web access to that repository.

Privacy and confidentiality are of utmost importance and Manitoba will currently be relying on audit and logging capabilities within the software applications as well as encryption, user profiles associated with registries of authorized individuals, etc.

In the initial days of the LIS, results will be transmitted by any available means. This will include auto faxing, secure e-mail, system-to-system reporting through an interface with a hospital information system, telephone or mail. This only applies to initial results reporting from the lab to the requestor and those designated on the test requisition. All others will be directed to the data repository.

It should be noted that the first iteration of the LIS will not include automated order entry. At present none of the acute care facilities in the province utilize automated order entry for lab testing. When this capability becomes available, the LIS will be interfaced to facilitate this capability. The standards that are being developed for test nomenclature will form the basis for these interfaces.

Saskatchewan

The Saskatchewan Health Information Network (SHIN) is a Treasury Board crown corporation that works closely with Saskatchewan Health and the 12 Saskatchewan Health regions, but also operates at arms length from government.

SHIN was created in 1997 to facilitate the flow of patient information between authorized health providers and to develop an electronic patient record. Since then, SHIN has worked with health regions on identifying, purchasing and implementing application specific feeder systems required to build the electronic health record.

As currently envisioned, patient health information would continue to be "owned" by the regions. Therefore the electronic patient records would be regionally based. At some point in the future small key pieces of these records could be pulled into a provincial electronic health record. There are many issues (e.g., privacy legislation, ownership of data in a provincially-based record etc.) that will need to be addressed prior to any such development taking place.

Unique Patient Identifier/Client Registry

SHIN is currently working with five of Saskatchewan's mid-sized regions on a project called the Integrated Clinical Systems (ICS) project. The project envisions the deployment of home care, central patient index/registration and laboratory and pharmacy systems for these five health regions. In addition, a Common View system will be deployed in these districts which will enable a patient's information to be viewed in one file. This is the first phase in the deployment of an electronic health record for health regions. These systems will upgrade existing systems and in some cases, will automate manual processes. The new systems will be hosted at SHIN's data centre and connected to users through CommunityNet (CNET). CNET is a broadband, high-speed, province-wide telecommunications network that will connect more than 800 educational facilities, 310 health facilities and 256 government offices in 366 Saskatchewan communities. Working Groups made up of subject matter experts in each area are working on developing a common configuration for the specific system and the applicable health information. In the final step of this project, all of these applications will feed into a common view application which will enable authorized health providers to view information from these three applications in a secure manner.

Eventually, the integrated system will result in more patient information being available to authorized care providers when and where they need it. It will also mean that within a health region, patients moving across the continuum of care will only need to give their registration information once, rather than having to register with each system individually.

Pharmacy Network

Drug Plan Network

More than 360 urban pharmacies are now connected to Saskatchewan Health's Drug Plan application via the SHIN network. Legislation was recently passed authorizing the collection by Saskatchewan Health of data on all prescriptions, not just those insured by the Department. Work is now proceeding on a systems solution to collect data on all prescriptions provided by pharmacies in order to support improved clinical decision-making.

To date, the claims side is in place and the clinical dispensing side is anticipated within 12 months. SHIN is working closely with pharmacies and pharmacy vendors on this component of the electronic health record.

Drug Utilization Review (DUR) Hosting

SHIN is hosting this program database for the College of Physicians and Surgeons (formerly called the Triplicate Drug Plan). This database is only accessible by the College and is used to scan for evidence of patients receiving concurrent narcotic or controlled drugs from three different physicians over a 30 day period. Currently, Saskatchewan Health can only scan a small portion of the claims. Once the Drug Plan Network system is fully implemented, the ability of the College of Physicians and Surgeons to scan for drug misuse and improve patient safety will be greatly enhanced.

Laboratory

The Laboratory Information System will help to automate the process of ordering, performing and reporting on tests. Currently, test orders from physicians are sent to the lab in a paper format. This paper test order is then given to a lab technician, who performs the test. That individual prints out the results of the test and gives the paper back to the receptionist who compiles the results with the orders and mails them back to the physician. The new system will allow lab personnel to receive patient demographic information electronically and manage test result data and generate reports electronically. Initially, the reports will be mailed back to physicians; eventually physicians will be able to receive these reports electronically.

The system allows lab results from areas including core lab, haematology, urinalysis, microbiology, blood bank and others to be transmitted into other departmental systems. This means that a patient's lab results could be transmitted into a system that captures a range of patient information (e.g., lab, home care, pharmacy, radiology etc.). This is the first step in the creation of an electronic patient file. The system was implemented in Cypress Regional Health Authority in 2002/2003, and is poised to begin operating in Prince Albert-Parkland early in the 2003/2004 fiscal year.

All regions involved in the project have agreed to a common implementation of the application. As the systems are implemented they are being configured to generate HL7 messages that will enable information flow between the applications and into a common view application once it has been installed.

Alberta

Unique Identifier

The Alberta Provincial Personal Health Identifier (PPHI) initiative is a key development in providing timely and accurate personal demographic information and current Alberta Health Care Insurance Plan (AHCIP) eligibility information to all parts of the health system. The initiative promotes the use of a standard identifier which is the Unique Lifetime Identifier (ULI) assigned by Alberta Health and Wellness. The initiative also promotes the sharing of consistent Person Information (PI) data and is a key foundation in the development of an electronic health record.

The PPHI will be implemented in two releases:

1. Person Directory Release 1: provides web browser access to search and display individually identifiable registration information (including current basic AHCIP eligibility information) as well as displaying, adding and maintaining consent information. The personal information displayed in the Person Directory (PD) application is a replica of registration information in the Central Stakeholder Registry (CSR). CSR contains demographic information on individuals who have a vested interest in the Alberta health system, and as such is the foundation for province-wide unique identification.
   
2. Person Directory Release 2: provides a web browser and system to system messaging interface to support, add and update functionality to the CSR. Release 2 Functionality will include the ability to:
   • Search for a person
   • Display personal demographic information
   • Add a person (e.g. new-born, new Alberta resident, visitor)
   • Add and maintain personal demographic information
   • Display consent information
   • Record and maintain consent decision
   • View current and historical basic AHCIP eligibility

Strict security safeguards are in place as per the Alberta WellNet security administration. Each user will have individual user identification and access will be restricted based on role. All users will be logged and audited.

Recently, the Capital Health Region in Edmonton, in partnership with Canada Health Infoway, commenced development of an Enterprise Master Person Index (EMPI) that will support the unique identification and linking of client information from disparate sites. It will also support the identification of clients who are from other jurisdictions.

Pharmacy Network

The Pharmaceutical Information Network (PIN) aims to improve the quality and cost-effectiveness of drug therapies delivered to Albertans. It links together physicians in the community, pharmacists, hospitals and other authorized health care providers, giving them confidential access to patient medication histories, equipping them with decision-support tools for prescribing and dispensing and enabling electronic prescriptions. Specific objectives include:

• to support communication among doctors, pharmacists and patients to help improve the overall quality of care for patients;
• to reduce hospitalisations and admissions to long-term care facilities by reducing adverse drug reactions and improving patient compliance with prescribed drugs;
• to enable health service providers to make better drug choices based on the best available evidence and access to patient medication profiles; and
• to protect the privacy, confidentiality and rights of the patient.

The PIN also provides tools and processes to support prescribing, dispensing, compliance monitoring, research and the formulation of legislation and policy regarding prescription medications.

Laboratory

The purpose of the laboratory component of the initiative is to deliver a history of a patient's laboratory test results, in a secure manner, to health care providers who are making care decisions regarding that patient. This will enable providers to more quickly and accurately diagnose and treat a patient and avoid additional tests. This project is a joint initiative between Capital Health Authority, Dynacare Kasper Medical Laboratories (DKML) and Alberta Wellnet, and will provide significant coverage for residents of northern Alberta. Similar initiatives for southern Alberta labs are under discussion.

Progress to Date

• Capital Health Authority completed the development of the repository of lab data and started to load historical data which has been in storage.
• Development of the web solution for user access to the lab repository has been completed.

Next Steps

• Capital Health will continue to populate the lab repository.
  
• Development of user training and implementation materials will be completed.
  
• Preparation for a pilot implementation of the Lab Result History system will commence.

British Columbia

Unique Identifier/Client Registry

In British Columbia (BC), the Personal Health Number (PHN) is a unique alphanumeric lifetime identifier used in the specific identification of an individual client or patient who has an interaction with the health system. It is assigned only to one person and may not be assigned to any other person at any time. As well as existing as an independent standard, the PHN is also an Embedded Standard, contained within the Health Registry Standard. PHN is used as the primary means of person identification on the Health Registry. The Personal Health Number became mandatory for all inpatient and day care surgical admissions commencing April 1, 1991.

Provider Registry

The Provider Registry is a standards-based repository of core provider data supplied by authorized sources that will facilitate the formal exchange of health information and is available to authorized consumers. Uniquely identifying providers will facilitate the transmission of health information between participating organizations and is one of the fundamental building blocks towards the realization of the pan-Canadian Electronic Health Record (EHR). The project is a Western Health Information Collaborative (WHIC) initiative, led by the BC Ministries of Health Planning and Health Services.

The Provider Registry System (PRS) has been designed with the potential to be implemented by any Canadian province or territory and expanded as a model for national data standards. Each participating province (British Columbia, Alberta, Saskatchewan and Manitoba) is implementing the common Provider Registry product within their existing technical infrastructure (e.g. servers, message routing) and initially populating it with data from their respective key Colleges. The PRS will ensure the security of patient health information by employing or enabling proven security and privacy techniques.

This project has successfully demonstrated that collaboration across regional and provincial domains can work and yield benefits. From the initial discussions to the development/build stage, the Provider Registry project has led the way for other collaborative projects in Canada. As a fundamental building block, the Provider Registry will serve as a valuable model towards the evolution of the Electronic Health Record.

Pharmacy Network

PharmaNet is a province-wide network linking all pharmacies to a central set of data systems. These systems significantly improve data and services in support of drug dispensing, drug monitoring and claims processing. PharmaNet connects all community pharmacies, with additional clients including emergency departments, hospital admitting, medical practice offices and clinics, the College of Pharmacists and the College of Physicians and Surgeons.

Scope

A network linking provincial pharmacies, many emergency departments and the College of Pharmacists has been in place since 1996. It will eventually be expanded to include physicians' offices. A pilot with 100 physician offices has been underway for some time and evaluated regarding expansion. Planning for expansion is currently underway.

Technology

High availability RISC/6000 computing complex HL/7 messaging via a TCP/IP secure network. Capacity to replace/enhance existing procedures or technologies has replaced claims submission batch processing, with an on-line point of sale system. IBM Canada and Systems Xcellence are the suppliers.

Impact on Patient Care

PharmaNet is reducing the number of negative drug interactions by keeping medical profiles of clients. The network is also helping to reduce prescription fraud and drug abuse by giving pharmacists better access to information.

Collaboration/Stakeholders

B.C. Ministry of Health Services, College of Pharmacists of British Columbia, College of Physicians and Surgeons of British Columbia and the BC Pharmacy Association.

Laboratory

Lab Test Standard (LTS)

The BC Lab Test Standard defines the business and technical requirements for the electronic exchange of lab test data. This standard expands beyond the traditional point-to-point exchange of information and accounts for all information exchanges from the time an order is issued until the time a final result is received. Completion of a lab test may involve a number of participants. Participants in this standard are public and private labs, agencies which perform lab tests and authorized providers.

The purpose of this provincial standard for the transmission of lab test data is to:

• facilitate the exchange of lab test information between a "network" of participants (potentially all licensed individual providers and all laboratories);
• enhance the quality of patient care through the exchange of consistent lab test information in a more timely and secure manner;
• reduce the cost of managing the exchange of lab test information; and
• reduce the costs associated with developing proprietary solutions.

The standard will be used in health system facilities where licensed individual providers communicate electronically with public and private labs that provide laboratory services to patients in BC. The standard is used by:

• authorized providers and employees in practice, public labs and private labs; and
• any Software Support Organization (SSO) intending to provide LTS software to participants. SSOs include commercial software vendors and organizations that perform their own software development.

The standard has been adopted and implemented by PathNET, a commercial organization established by the major laboratory companies in BC to distribute lab test results to physicians.

Yukon

Unique Identifier

Yukon has a unique identifier for each person registered on the Yukon Health Care Insurance Plan. This has been in place for more than 20 years.

Client Registry

At present, all persons registered in the health care system are included on the registration database. However, due to limitations in the software, this database cannot be used to link to other components of the health care system at present. The Yukon will be developing a client registry that can be linked to other health care information systems within the next one to two years.

Pharmacy Network

Yukon will be developing a pharmacy network within the next one to two years.

Laboratory

There are currently no private labs within Yukon. The hospital lab uses Meditech software to track results. Neither physician offices nor nursing stations are linked to this system at present, but the feasibility of linking will be investigated.

Northwest Territories

Unique Patient Identifier / Client Registry

There is no Unique Patient Identifier/Client Registry available in the Northwest Territories (NWT). The system currently in place is a resident registry component of the Northern Health Information Management (NHIM) system. The resident registry relies on the health care number of the individual. If, however, an individual enters a health care facility in the NWT without a health care number, that individual will be assigned a temporary NWT health care number which will identify the individual within that hospital. This process will be repeated if the same individual goes to a different hospital within the NWT (i.e. he or she will be assigned a new temporary NWT health care number).

The intention of the Government of the Northwest Territories is to inter-connect the registry modules of operational and central registry systems so that a person can be positively identified by using the person's healthcare number. This will be implemented through the new central registry module of a new application system to replace the aging NHIM system. Deltaware Systems was recently awarded this replacement project. It includes a Vital Statistics module and a Medical Travel module.

The registry of the new system will become the trusted-source of registration information (central registry) for clients, providers and facilities. It is targeted to be operational in the summer of 2004. Starting in late 2004 or early 2005, the registry modules of various feeder systems will be integrated to the central registry of the new system. Furthermore, each registrant will be assigned a NWT unique lifetime number (ULN) in the new central registry system. This ULN will be forwarded to each operational system for the purposes of positively synchronizing records. This number will also be used for extracting and reporting person-level information, for example, extracting to a data warehouse or reporting to CIHI.

Pharmacy Network

There are currently three different pharmacy systems in place to support the internal pharmacies within the four NWT hospitals. The existing systems are at different levels of sophistication. In 2004/2005, the Government of the Northwest Territories plans to upgrade or replace these pharmacy systems with MediPharm, a module of the MediPatient Plus system from MediSolution. The MediPatient Plus system is used by all the NWT hospitals for patient registration and admission, discharge and transfer. A number of the hospitals have additional MediPatient Plus modules in operation. Utilizing the same supplier for pharmacy will permit an easier interaction between systems that will help facilitate:

• easier updating of, and access to, a patient's health record,
• support for electronic orders within the hospital,
• support for drug interaction assessments,
• interaction between hospital and health centre pharmacies which, in turn, will help make possible:
  1. a. more global management of drug and prescription dispersal, and
  2. b. access from remote community sites to manage local inventories.

The new pharmacy system will not link with retail pharmacy settings. A review is underway to assess options on how best to link the retail pharmacies for the purposes of order management, drug interactions, prescription management and updating the patient's health record.

Laboratory

At present, the laboratory system in place at each of the four NWT hospitals utilizes older software from TripleG. Starting in 2004/2005, the Government of the NWT will be replacing this older software with an updated software version from the supplier. Starting in 2005/2006, it is proposed that the current multiple lab systems be inter-connected to form one logical system.

Goals include:

1) to be able to enter local laboratory results for access by all authorized users, including those outside the local area;

2) to grant access directly from the patient's health record; and

3) to facilitate the electronic capture of lab results from third party laboratories. The new system will also permit broader access for authorized health care providers such as doctors and nurses.

Nunavut

Client Registry

Several components of a client registry have been introduced.

These include:

• a Community Health Reporting System which collects information on services provided on a patient basis in the Community Health Centers is currently being tested.
  
• a Communicable Disease Module for tracking of communicable diseases and producing statistical reports was implemented in March 2001.
  
• a new homecare module which collects information on home services by client type was implemented in October 2002.
  
• a Pap database is being started.
  
• the Cancer registry is presently being updated.
  
• a new vital statistics database is planned for implementation April 2003.
  
• a new medical travel database is under development and due for implementation later in 2003.

Contact Persons for Each Jurisdiction

The following individuals were consulted during the development of this document:

Newfoundland & Labrador

Mike Barron, Project Leader, Newfoundland and Labrador Centre for Health Information (NLCHI)

Nova Scotia

Chief Information Officer Branch, Nova Scotia Department of Health

New Brunswick

David Cowperthwaite, Director, Information Systems, New Brunswick Department of Health and Wellness

Prince Edward Island

Faye Campbell, Manager, Information Resource Management, Health Informatics, PEI Dept of Health and Social Services

Quebec

Gilles Cantin, Directeur adjoint des technologies de l'information et des tèlè communications MinistPre de la santè et des services sociaux

Ontario

Lorelle Taylor, Chief Information Officer, Human Services I&IT Cluster, Ministry of Health and Long-Term Care (MOHLTC)

David McCutcheon, Assistant Deputy Minister, Health Services Division (HSD), MOHLTC

Jim Jack, Director, Registration and Claims, HSD, MOHLTC

Susan Fitzpatrick, Director, Provider Services, HSD, MOHLTC

Dawn Ogram, Director, Laboratories, HSD, MOHLTC

Susan Paetkau, Director, Drug Programs, HSD, MOHLTC

Bud MacDonald, Head, Health Services Programs I&IT, Human Services I&IT Cluster, MOHLTC

Manitoba

Ken Browne, Senior Business Analyst, Manitoba Health Information Systems Branch

Saskatchewan

Phil Moleski, Director, IT Development and Operations

Virginia Wilkinson, Director of Communications and Business Development, Saskatchewan Health Information Network

Alberta

Linda Miller, Director, Information Management, Health Accountability

British Columbia

John Schinbein, Chief Information Officer, Ministry of Health Services, Ministry of Health Planning

Stuart Frampton, Senior Client Manager, Regional Services, Health

Anita Malovec, Business Manager Client/Provider Registries, IMG - HealthNet, Ministries of Health Planning and Services

Yukon

Joanne Fairlie, Assistant Deputy Minister, Health Services

Northwest Territories

Kees Hamming, Chief Information Officer, Department of Health and Social Services

Nunavut

Patrick Ridgeley, Manager, Information Technology, Department of Health & Social Services

Related Documents and Reports

The following documents were retrieved and utilized during the development of this report:

British Columbia Centre for Disease Control (BCCDC) Information Management, October 2000. PHIS: Public Health Information System Overview. Prepared by Corinne Cook.

British Columbia Health Chief Information Officers Council. Framework for an Electronic Health Record for British Columbians, January 2003.

Buckeridge, D. Health Informatics in Canada: Definitions, Education, and the Path Ahead. November 1999.

Canadian Institute for Health Information. Navigating the Swirl: An Overview of Canadian Health Informatics Initiatives, October 1998.

EU -- Canada Expert Meeting on E-health within the Northern Dimension, July 2001. (accessed 12/12/2002).

Gates, K. Evaluation of a System for Electronic Exchange of Laboratory Information: A Pre-Implementation Study. A Thesis submitted to the School of Graduate Studies in partial fulfillment of the requirements of the degree of Master of Science, Faculty of Medicine, Memorial University of Newfoundland, March 2004.

Health Canada, Office of Health and the Information Highway, January 2001. Toward Electronic Health Records. Toward Electronic Health Records (accessed 08/17/2003).

Healthnet/BC Provider Data Standard. British Columbia Ministry of Health and Ministry Responsible for Seniors, October 2001.

Kozyrskyj A, Brown T, Mustard C. Community Pharmacist perceptions of a provincial drug utilization database. Canadian Pharmaceutical Journal 1998; 131:24-29

Lau, F. and Hebert, M. Experiences from health information system implementation projects reported in Canada between 1991 and 1997. Journal of End User Computing 2001; 13(4): 17-25

MacDonald, D. Pharmacists' Expectations of a Pharmacy Network: A Baseline Evaluation. A Thesis submitted to the School of Graduate Studies in partial fulfillment of the requirements of the degree of Master of Science, Faculty of Medicine, Memorial University of Newfoundland, March 2004.

National Steering Committee on Patient Safety. Building a Safer System. A National Integrated Strategy for Improving Patient Safety in Canadian Health Care, September 2002.

Newfoundland and Labrador Centre for Health Information, Information Systems Strategic Plan, 1998

Newfoundland and Labrador Health Information Network, NLCHI Benefits Driven Business Case (BDBC), November 1998.

Newfoundland and Labrador Centre for Health Information, UPI/Client Registry Project Scope. February 2000.

Newfoundland and Labrador Centre for Health Information. Pharmacy Scoping Project Briefing Note, 2002.

Ontario Hospital eHealth Association (March 2002). eHealth Update, March 2002.

Report to Health Infostructure Atlantic. Report on Common Vision and Workshop Findings, July 2000.

Saskatchewan Pharmaceutical Association. A Framework for a Comprehensive and Integrated Drug use Management Strategy for Saskatchewan (Discussion Paper), June 1999.

Western Health Information Collaborative. Client Registries (White Paper), March 2002.

Western Health Information Collaborative. Laboratory Information Strategies and Standards (White Paper), March 2002.

Western Health Information Collaborative. Provider Registry (White Paper), April 2002.

Western Health Information Collaborative. Pharmaceutical Information Network. Medication Information Strategy (White Paper), May 2002.