Evaluating Telehealth 'Solutions' A Review and Synthesis of the Telehealth Evaluation Literature

Office of Health and the Information Highway, Health Canada March 2000

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Table of Contents

• Executive summary
• Acknowledgements
• Introductions
• The Institute of Medicine (IOM) evaluation framework
• Alternative frameworks and perspectives
• Conclusions
• Recommendations
  • Appendix I - Literature search - primary citations
  • Appendix II - An overall conceptual model
  • Appendix III - Which telehealth solutions require evaluation?
  • Appendix IV - At what stage of telehealth development is evaluation required?

Executive Summary

Telehealth is broadly defined as the application of telecommunications and information technology to the delivery of health care and health-related services and information over large and small distances. Through telehealth, these become more accessible to health care providers and consumers in rural, remote, and under-served areas.

In recent years, demonstration and pilot projects have been valuable in confirming basic feasibility and safety of telehealth but, similar to health care in general, most have not been guided by a systematic framework for evaluation. Such information is crucial if policy-makers and decision-makers are to be convinced of the usefulness of continued investment in this technology, particularly given competition for scarce resources. This lack of evidence also contributes to the continued low level of acceptance of telehealth among practitioners and clients.

Telehealth evaluations should permit informed decisions to be made by at institutional, regional, provincial, and federal levels. But to do so it is desirable that consistency be developed in the nature of evaluation activities. In reality, such consistency -- if achievable -- will not occur in the short term. As a consequence, there are two goals: first, to inform about desirable components of an evaluation, so that shortcomings in past, current, or future evaluations can be recognised, and second, to identify a suitable evaluation framework as the basis for a standardised, and thereby consistent, approach to evaluation of all future telehealth applications.

Routine application of such a framework will also reduce variation due to omissions and errors, and facilitate comparison of evaluations between jurisdictions, enabling evidence-based decisions to be made.

As a result of a thorough literature review, one source was identified as the most significant and complete contribution to telehealth evaluation at this time. This reference, published by the Institute of Medicine, focuses on the categories of quality, accessibility, cost and acceptability, each being an issue that lies at the core of most health services research and technology assessments. Quality of care was defined as "the degree to which health care services for individuals and populations increases the likelihood of desired health outcomes and is consistent with current professional knowledge". Access referred to the "timely receipt of appropriate care", and cost to the "economic value of the resource use associated with the pursuit of defined objectives or outcomes". Finally, acceptability was the degree to which "patients, clinicians, or others are satisfied with a service or willing to use it".

While the IOM offers a generic framework that can be used as a standardised tool, an inherent limitation was recognised. In providing broadly relevant evaluation 'theme questions' concerning the issues of quality, accessibility, cost and acceptability of clinical telehealth, it does not address application-specific outcome measures and criteria. As a result, appropriate application-specific questions must be determined by investigators, and superimposed upon the IOM framework.

It is crucial that a leadership direction be adopted that will permit consistent and comparable evaluation of all Canadian telehealth applications. Such a consistent approach could be instituted through a policy requirement that the evaluation framework of the IOM be required as the sector standard, with application-specific measures superimposed upon this model. Significant benefits would accrue to both individual telehealth applications and to the telehealth industry within Canada as a result of such a policy approach.

Acknowledgements

A panel was established that represented a broad spectrum of telehealth and social science skills. Their task was to review and critique the tertiary references provided to them following the literature search and preliminary evaluations. Members of the expert panel were :

The support of the Office of Health and the Information Highway, Health Canada, through contract # M-420005 is gratefully acknowledged.

Structure of the report

This current report identifies, synthesises, and critiques the current literature on telehealth evaluation frameworks. The goal of the report is to present key evaluation elements and to discuss them within a unified telehealth evaluation plan whose routine implementation would ensure greater consistency and comparability as Canada moves forward towards broad-based application of telehealth 'solutions'. Appendices provide additional detail regarding citations and additional perspective.

Expected Benefits

This report will assist in creating permanent change in the manner in which telehealth applications are supported and assessed by potential funders, in scholarly writings related to telehealth evaluation, and in informing policy and decision-makers of salient issues to consider.

A secondary result anticipated through dissemination of the report is a more dominant role of qualified researchers supporting and disseminating carefully researched evaluations of all future telehealth initiatives.

Significant benefits would accrue to both individual telehealth applications and to the telehealth industry within Canada as a result of adoption of the reports recommendations.

Introduction

Definition of telehealth

In many literature references the term 'telemedicine' is used. To some, this implies use of telecommunications technology only for medical (i.e., physician) intervention. However, it is universally accepted that such technology has much broader applicability (e.g., use by nurses and many other health care professionals; use for additional activities such as education, research, and health care administration). To address this broader utility, the term 'telehealth' is used throughout this report, except where a quote is provided or direct reference to a paper is made.

Bashshur et al., defined telemedicine as "a system of care that employs telecommunication and computer technology to substitute for face-to-face interaction between patients, physicians, and / or non-physician providers in various combinations". This definition shares many features with that provided by the Institute of Medicine (IOM), where telemedicine was defined through "the use of electronic information and communication technologies to provide and support health care when distance separates the participants". Together these definitions embrace the elements of information and telecommunication technologies, distance between participants, and health or medical uses.

For the purposes of this report, telehealth is broadly defined as the application of telecommunications and information technology to delivery of health care and health-related services and information over large and small distances.

Growth of Telehealth

Since its origins 30 or so years ago, the growth of telehealth in Canada and the United States has been very slow. The industry, for the most part, comprised short term pilot projects funded through government subsidies and research and development grants to researchers and clinical practitioners. Telehealth activity at that time was restricted to remote medicine provision through the use of communication and information technologies. Today telehealth's spectre has broadened to include the provision of health care and health care services. Although the primary rationale for the continued development of telehealth remains the same, it is also conceivable that telehealth in the future will dramatically reshape the 'face to face' medical model of health care delivery as we know it, that has existed for generations.

Over the past few years growth in telehealth networks has been rapid. A study of telehealth centers commissioned by the Office of Rural Health Policy in the USA determined that 29% of the rural health hospitals already use or plan to implement a local area network (LAN). The Telemedicine Information Exchange currently lists 148 projects, of which 6 are Canadian. Globally, 168 and 183 telehealth sites in over 35 countries were registered in 1996 and 1997, respectively. The Association of Telemedicine Service Provider's (ATSP) data on 96 active telehealth sites in over 40 US states indicated that the number of telehealth consults in 1996 (21,000) tripled that of 1995, and in the first quarter of 1997 over 11,000 consults in over 40 medical specialities had been registered.

In Canada, 300 companies are currently active in the telehealth sector. Large companies serve as computer manufacturers and telecommunication providers and a host of smaller companies function as software developers, service providers, consultants and are active in R & D. The combined annual revenue of these companies in 1997 was $330 million, a figure which is expected to reach $1 billion per annum by 2000. In 1999, telehealth activity in Canada was valued at an estimated $500 million. Homecare, the most important growth component in telehealth, has increased 15% annually over the past 5 years. In this sector alone in 1996, the Canadian provincial governments spent $1.5 billion out of the $72.5 billion allocated to health care.

The breadth of clinical applications in telehealth is wide. Among others it includes routine telephone calls to a family physician, emergency 911 calls, filmless teleradiology services, interactive video based consultations, and more exotic services such as telecardiology and remote, robotics-based, telesurgery. Some applications require 'real time' diagnostic capabilities using video, audio and specialised data transmissions, while others entail less expensive 'store and forward' technologies, or just plain ordinary telephone lines (POTS).

Telehealth 'Solutions'?

Proponents maintain that telehealth 'solutions' will resolve the seemingly intractable problems of health care; ensuring universal access to health care while at the same time cutting costs and maintaining quality. The evidence base for such a claim, however, does not exist, and the reasons are several fold. First, they stem historically from the restricted funding time frames of many of the first generation telehealth projects, such that long term evaluation data of a 'steady-state' telehealth systems could not be captured. The early demise of many of these projects, therefore, was not the result of any failure to reach stated objectives, but simply the consequence of 'pulling the plug' long before decisions had, or could have, been made on intended goals. Second, it resulted from the limited feasibility and technical focus of many of these early pilot projects. More often than not, the ensuing rapid technological change limited evaluation solely towards the technological aspects of the application. While this information was important in determining the minimally acceptable clinical standards, it did little to address the broader questions concerning the effects of telehealth on the health care system. Moreover, there is a general consensus that it is probably futile and uninformative to continue to evaluate telehealth in this manner given the pace at which information and computer technology renders obsolete the assumptions made about the capabilities of current technology.

At first glance, one would think that telehealth would only derive support from technological progress. A wide variety of telehealth initiatives should be supported through advances in computer hardware, communications bandwidth (now available at affordable costs), improved high capacity, high speed inter-operating technologies supporting telehealth integration, and the backing of the private sector to facilitate technology transfer and commercialization. Yet this is not the case, as technological growth in telehealth seems fragmented as well, with over 80% of the industry members in Canada maintaining that it is the very 'lack of common technological standards that hampers the development of telehealth networks'. In 1997, the G7 Ministers of Industry and Telecommunications met to discuss these very concerns and pushed for a global development and implementation strategy for a common telecommunications network.

Telehealth Evaluation

After 30 years of telehealth experience, knowledge of the actual effects of telehealth on the health care system still lags far behind the rush to establish telehealth systems in various parts of the country. In this competitive environment, telehealth clearly has not demonstrated 'solutions' to redress the problems of health care delivery to under-served populations. The overriding question still remains; does telehealth address the tripartite goals of health care of "assuring universal access to care while at the same time limiting increases in cost and maintaining quality"? If this indeed is the case, evaluation data should exist to support it; but it does not. In fact, few if any concerted efforts have been made to seek standardisation with regards to an evaluation framework for telehealth. Perednia has delivered a very sobering thought on this issue, "thus, there is probably not a single telehealth project in the United States that is, by itself, currently capable of providing meaningful data on medical efficacy, clinical utility or cost effectiveness."

Need for Evaluation

Unfortunately, telehealth shares with other clinical health care services in lacking clear evidence of its effectiveness and cost-effectiveness. What is clearly missing from the telehealth agenda is an evaluation of patient care applications. Although a variety of demonstration and pilot projects have been valuable in demonstrating basic feasibility and safety, most have not been guided by a systematic framework for evaluating the effects of clinical telehealth on the quality, accessibility, cost or acceptability of health care. It is this information that is now required by managers of health plans and decision makers in health care as a prerequisite for new or continued investment in this technology, particularly given the competition for resources in this era of budgetary restraint. Moreover, it is this very lack of evidence that underscores telehealth's lack of acceptance by both clinicians and patients.

Search Strategy

The literature search strategy included the clinical and health informatics literature for articles, abstracts and conference proceedings encompassing all aspects of telehealth and telemedicine evaluation. It included the following databases, over the following time periods.

HealthSTAR : Health Services, Technology, Administration, and Research database. Covers clinical (evaluation of patient outcomes and the effectiveness of procedures, programs, products, services and processes) and non-clinical (health care administration and planning). Includes journal articles, technical and government reports, meeting papers, abstracts, books, and book chapters.Updated weekly; ~ 17,000 citations added per month. Primarily english language, but international in scope.

MEDLINE : MEDlars onLINE database. Covers all languages. Updated weekly; ~ 31,000 citations added per month; contains over 9.2 million records. Covers biomedicine; including the fields of medicine, nursing, and pre-clinical sciences. Contains articles from more than 3,800 international biomedical journals.

CINAHL : Cumulative Index to Nursing and Allied Health Literature database. Over 1,000 journals reviewed and indexed.

Cochrane Library. A regularly updated electronic library providing a comprehensive source of literature on evidence-based health care.

These sources were viewed as sufficient to identify the major peer reviewed papers dealing with evaluation of telemedicine / telehealth applications. It should be noted that this search strategy did not include documents prepared by or on behalf of governments, since these sources will already have been presented to, and assimilated by, the Office of Health and the Information Highway. However, the formal search was supplemented with Internet searches using Altavista, Netscape, Excite, and Yahoo, and experience gained through contacts in G7 / 8 countries.

From the initial search of over 400 abstracts and conference proceedings, 90 papers were selected as primary references for review (Appendix I). A hardcopy of each paper identified was obtained, and each reference was reviewed and summarised by the project research associate. From this original list of primary references, 33 articles were selected and forwarded to a three-member panel for final selection of key references. By focusing on themes, and identifying research groups within the literature, 15 articles were chosen as key references for presentation, discussion and critique by a 9 member panel. The panel, comprised of medical professionals and researchers, psychologists, sociologists, economists and experts in the Canadian telehealth community, met over a 2-day period to initiate the final report. This report emanates from those proceedings.

After the detailed review and summary of the primary references, the existence of an evaluation document, developed and published by the Institute of Medicine (IOM) in response to a request by the National Library of Medicine's (NLM) for a "broad framework for evaluating clinical telemedicine", was noted. This document represents the most comprehensive discussion of clinical telehealth evaluation developed to date : it encompasses many of the key evaluation features advanced in the literature; reflects the contributions of many of the leading proponents of clinical telehealth; is broad in scope (although failing to address application-specific provisional and outcome measures and criteria); and develops broadly relevant evaluation questions concerning the issues of quality, accessibility, cost and acceptability of clinical telehealth.

This report, therefore, focuses on the IOM evaluation framework and attempts to encompass other evaluation frameworks, concepts and discussion papers, within it. The evaluation framework presented below serves as a comprehensive introduction to the telehealth evaluation plan forwarded by the IOM. This information is then discussed and critiqued within the context of additional evaluation features identified in the literature.

The Institute of Medicine (IOM) Evaluation Framework

The National Library of Medicine (NLM) commissioned the Institute of Medicine to "develop a broad framework for evaluating clinical telehealth". The 15-member committee comprised experts in telemedicine, medical informatics, health care delivery, health services research, quality assurance, economics and public policy analysis. They defined telemedicine, categorised its clinical applications, and then identified working principles, to guide their evaluation plan. A comprehensive summary of their approach and evaluation framework is provided below, but must be viewed as only a cursory review of their book.

Definition of Telemedicine

The IOM defined telemedicine through "the use of electronic information and communication technologies to provide and support health care when distance separates the participants". This definition embraced the elements of information and telecommunication technologies, distance between participants, and health or medical uses. It also inferred that the majority of benefits of telemedicine derive from substitutions, made possible only through technology, which include the site of care, the type of service provider, and the content and process of care.

Although not the focus of their report, the IOM also recognised the many non-clinical applications of telemedicine such as professional and patient education, research, public health, and health care administration.

Categories of Clinical Telemedicine

In telehealth, the variety of clinical applications is endless and includes routine telephone calls to a family physician, emergency 911 numbers, filmless teleradiology, and also extends to include interactive video based consultations and more exotic services such as telecardiology and remote, robotics-based telesurgery. Some applications require 'real time' diagnostic capabilities using video, audio and specialised data transmissions while others entail less expensive 'store and forward' technologies. Most clinical telehealth applications can be categorised using the scheme presented below as modified by the IOM.

• Initial urgent evaluation of patients for triage, stabilisation, and transfer decisions
• Supervision of primary care by non-physician providers when a physician is not available locally
• One-time or continuing provision of specialty care when a specialist is not available locally
• Monitoring and tracking of patient as part of follow-up care or management of chronic problems
• Use of remote information and decision analysis resources to support or guide care for specific patients.

This classification scheme also included additional information related to the clinical problem (urgency, complexity, pathophysiology, and persistence), the processes of care (type of care, source of care, source of clinical information), and to the kind of information contained within a typical telehealth application (aural, visual, textual).

Working Principles Relating to Health Care

The principles outlined in Figure 1 reflect what the IOM describe as 'the complicated and volatile nature of the health care system'. These principles focus on the ways and means of monitoring telehealth and on some underlying assumptions that may guide these activities.

Working Principals Relating to Evaluation

The evaluation framework is guided by four basic principles, outlined in Figure 2, which the IOM felt formed the foundation for health services research and evaluation research generally, and which in turn are related to the special challenges brought fourth by an evaluation of telehealth. Similar to the target audience for this report, the IOM's target audience was health care policymakers, clinicians, patients, and managers.

Elements of the IOM Evaluation Framework

In concert with these basic principles the IOM proposed an evaluation framework, the purpose of which was to strengthen individual telehealth evaluations and to promote the adoption of co-ordinated evaluation strategies across projects and organisations. The essential elements of their evaluation plan are presented in Figure 3.

Distinctive features of the evaluation plan were identified by the IOM. These include delineating how the technical, clinical, and administrative components intended to work versus their actual implementation. This information was believed to be crucial in order to distinguish the failure of an application from the failure of an implementation. With this information, telehealth applications could then be adopted as is, redesigned and reassessed, or discontinued. Moreover, broadening the evaluation focus beyond the technical features of particular technologies - to include the clinical, organisational and social domains - was also endorsed in their program.

The IOM was also cognisant of the technological imperative often present in telehealth applications where a 'solution' is looking for a problem. The solution to this technological dilemma lies in a needs assessment where the clinical and health care goals of the telehealth application are clearly identified (see Figure 3, first bullet). Once this has taken place, organisational structures can be put in place to further capitalise on technological advances while at the same time recognising the need for a 'best fit' technological solution. The second side to this technological imperative suggests that technology often develops a life of its own long before its role in telehealth and health care system can be fully understood.

Another aspect recognised by the IOM was the fast pace of change in the telehealth environment. To address this, a sensitivity analysis keyed to a business plan was recommended, so that decision makers could also adjust intended conclusions to changes that might occur in the future to key variables or assumptions.

Finally, the IOM also supported the development of rigorous experimental evaluations and peer reviewed publication in an area where non-experimental designs and anecdotal reports have prevailed.

Categories within the IOM Evaluation Framework : Quality, Accessibility, Cost, and Acceptability

Evaluation, according to the IOM represented a "variety of methods and strategies for identifying the effects and assessing the value, feasibility, or other qualities of a technology, program or policy". Their evaluation criterion sought "measures, indicators, and standards, for describing outcomes or making judgements" and were provided for by a series of general questions. The evaluation questions were not linked to specific clinical telehealth applications but probed general issues. They focussed on the categories of quality, accessibility, cost and acceptability, issues that the IOM felt lie at the core of most health services research and technology assessments.

Citing previous IOM documents, the committee advanced definitions for each of these evaluation categories. Quality of care, was defined as the "degree to which health care services for individuals and populations increases the likelihood of desired health outcomes and is consistent with current professional knowledge". Access, referred to the "timely receipt of appropriate care" and its costs to the "economic value of the resource use associated with the pursuit of defined objectives or outcomes." Finally, acceptability alluded to the degree to which "patients, clinicians, or others are satisfied with a service or willing to use it," and represented an extension of 'patient only' satisfaction data that prevailed in the literature and which the IOM considered too limiting. Each of these evaluation categories included a series of basic theme questions. These are presented and discussed below. These questions probed the concepts of quality, access, cost, and acceptability and also attempted to determine their interactions and interrelationships.

A. QUALITY - Evaluating Quality of Care and Health Outcomes

Theme questions :

• What were the effects of the telehealth application on the clinical process of care compared to the alternative(s)?
• What were the effects of the telehealth application on immediate, intermediate, or long-term health outcomes compared to the alternative(s)

Quality of Care

The IOM indicated that the ultimate purpose of any medical care was to maintain and improve quality of care (quality of care being defined as "the degree to which the health care services for individuals and populations increases the likelihood of desired health outcomes and are consistent with current professional knowledge"). This definition encompassed both individuals and populations and reflected how different clinical interventions could be used to the greatest social advantage. Moreover, given that the evidence base for what does and does not work in health care is still modest, the IOM recognised the importance of professional knowledge (experience and judgement) with biomedical and clinical research (evidence-based medicine) in delivering health care. This definition also described the link that exists between the processes of care and the outcomes of care, although the latter measure has received most of the attention in the evidence-based health care environment of recent years. This definition purposely avoided confounding quality of care with resource constraints on the grounds that what is judged as being high quality or not, should not be based on available, or limits in, resources.

In order to properly assess quality of care, a definition of appropriate care has been adopted by the IOM for their evaluation framework. Appropriate care is defined as "the expected health benefit, which exceeds the expected negative consequences by a sufficient margin" that the care is worth providing. Determining what is 'worth' providing against health risks and benefits is recognised as being somewhat controversial and is usually addressed, more or less, by a combination of both subjective and objective measures. Paradoxically, the objective measures are often based on what is considered to be standard practice, despite the fact that these very standards may not be evidence based, and, therefore, may be suspect.

Evaluating the clinical effects of telehealth can occur at a number of different levels and may involve both the processes and outcomes of care. It is important to realise that many of the process measures proposed by the IOM do not cover "routine but important quality assurance procedures" specific to one telehealth application, but serve to identify basic questions concerning the quality of care. It is left up to the individual evaluators to then devise detailed questions concerning their specific telehealth applications. In many examples of telehealth, process measure are utilised as proxies for health outcome measures, when data on the latter is unavailable. The choice of proxy measures should engage research data demonstrating a link of the proxy to the desired health outcome.

In the literature, three broad types of 'quality' problems in telehealth have been differentiated. They are the overuse of care, the underuse of care, and the poor technical or interpersonal performance of equipment. All may have a serious impact on telehealth and should be evaluated separately. For example, unnecessary use of telehealth consultations would constitute overuse, failure to refer a patient for a telehealth consultation would represent an underuse, and failure to read an X-ray correctly would comprise a technical failure.

Questions developed by the IOM to evaluate the clinical process of care in telehealth versus the alternative(s) are shown in Figure 4.

The value of process measures notwithstanding, decision-makers, clinicians and patients increasingly demand information on health outcomes. This incessant demand for direct measures of health outcomes is the direct result of a shift in society towards evidence-based medicine. Patient outcomes may include endpoints of health but may also incorporate a range of immediate and intermediate results. Many of these measures should be directly connected to an element of a clinical intervention, but establishing this link becomes problematic in longer-term episodes of care. With respect to telehealth, valid and reliable health outcome instruments may already be available or will have to be constructed. In any case, they should fit the "patients, settings, services, and desired outcomes" of the telehealth application.

The evaluation questions proposed by the IOM in order to analyse health outcomes versus the alternative(s) are presented in Figure 5.

Other Quality of Care Issues in Telehealth

The IOM also identified a number of evaluation issues surrounding quality of care that should form a component of any systematic evaluation of telehealth. Specifically, questions should be included as to what extent clinical applications of telehealth bestow a learning effect on clinicians. If so, do these changes in knowledge or practice produce demonstrable differences in short-, long-, or longer-term health outcomes or in health status of patients? Is there a learning curve related to the volume of telehealth use and if so, does it exist for certain telehealth skills and applications, and not for others? Should one then be concerned about the quality of care in telehealth sites demonstrating low volumes? If consultations are then directed to higher volume hubs, or 'centers of excellence', what effect might this have on the expertise of the local specialist?

B. Access - Evaluating Access to Care and Health Outcomes

Theme questions :

• Did telehealth affect the use of services or the level or appropriateness of care compared to the alternative(s)?
• Did the application affect the timeliness of care or burden of obtaining care compared to the alternative(s)?

Access to Care

The promise that has surrounded the introduction of a number of pilot and demonstration projects was that telehealth would improve access to health care services for those people residing in rural or remote areas where clinical services were uncommon or non-existent. In recent years, however, this promise has been extended to the poor, the socially disadvantaged, the institutionalised, and more recently to the urban and suburban housebound.

The IOM defined access as the "timely receipt of appropriate care" or "the ease or difficulty of obtaining care", or more simply as the "availability of the right care at the right time without undue burden". These definitions invoke the notions of appropriate care, timeliness of care, availability of care, and burden / ease of care. Availability of care suggests that the service is ready for use, if and when needed. Undue burden of care indicates that the difficulty of obtaining a service should be an integral part of its evaluation. In measuring access to care, the IOM recommended evaluating access to appropriate care and not to all services, whether appropriate or not. In the past, measures of physical resources such as hospital beds and physicians per 1,000 population were used as indicators of access to health care, as if to suggest that the mere presence of a physical resource could be linked to increased health benefits. This clearly need not be the case. The IOM suggested that more operational measures should be used to measure access to health care.

Barriers to Access

In evaluating access to health care, the IOM also ascertained that barriers other than geographical need to be considered in a telehealth evaluation framework. These included :

• significant distance from primary, secondary and tertiary medical services;
• poor transportation services;
• inadequate financial resources, particularly insurance coverage or directly subsidised services;
• delivery system characteristics, including poor coordination of care, long waiting times for appointments, inadequate numbers or kinds of specialists, and bureaucratic obstacles to services; and,
• gaps in our knowledge about how these factors interact to affect the use of services and what can be done to overcome or eliminate barriers to access.

Other barriers identified include restrictive reimbursement policies for telehealth (i.e., for any medical service that is not conducted 'face to face'), restrictions on interstate (inter-provincial) licensure, product liability issues, malpractice issues, inadequate security, reluctance (for teleconsulting and telereferrals) on the part of mainstream medicine. Of all the issues impinging on professional acceptance and dissemination of telehealth, cost-effectiveness appears to be one of the most critical. Others have put forth the technical, clinical, organisational and behavioural obstacles that may exist preventing telehealth's widespread adoption. Uncertainties related to licensure and medical liability also continues to hamper access to telehealth.

Facilitators to Access

Whereas barriers have been discussed, others have identified facilitators to telehealth. Many feel that the telehealth community must develop and adhere to a code of ethics emphasising professional conduct and clinical norms of practice and practice guidelines. Quality assurance criteria and standards should be developed for assessing quality and monitoring progress.

Other Access Issues in Telehealth

Parallel effects of telehealth may have an impact on the retention and recruitment of clinicians in under-served areas through reduced social isolation and increased educational opportunities. This may occur through telehealth links to specialists and CME programs and, ultimately, may eliminate many pre-existing barriers to health care access.

As a result of their deliberations, the IOM proposed a number of questions to guide examination of access, which are presented in Figure 6 below.

Population-based Measures of Access to Care

Access to health services may be measured through questions at the individual, group, or population level. Given that concerns over access often relate to disadvantaged groups, the IOM directed their evaluation towards populations or sub-populations. In this regard, they recommended the adoption of population-based indicators of health care access. For example, an indicator of the 'lack of access to timely and appropriate treatment, in ambulatory-care-sensitive chronic disease states such as diabetes', might be avoiding hospital admission. Furthermore, they suggested that since telehealth is at such an early stage of diffusion, measurable effects based on population based indicators may not be discovered. In its place, the IOM proposed the following access-related indicators for telehealth, such as :

• use of telemedicine services over time;
• changes in the number of traditional consultations;
• changes in waiting time for specialist appointments;
• changes in the rates of missed appointments for consultations; and
• patient or clinician attitudes about the timeliness of consultations and the burden of different consultation options.

C. Acceptibility - Evaluating Patient and Clinician Perceptions of Care and Health Outcomes

Theme questions :

• Were patients satisfied with the telehealth service compared to the alternative(s)?
• Were attending / consulting clinicians satisfied with the telehealth application compared to the alternative(s)?

Patient and Clinical Perceptions of Care

Although often included within the category of 'quality of care', the IOM considered patient and clinical perceptions of care to be sufficiently important evaluation issues in their own right. They had observed that most telehealth applications gathered only patient satisfaction data, leaving larger issues concerning telehealth quality, accessibility, or costs untouched. Moreover, clinician perceptions were rarely examined in telehealth.

The tools used to assess patient and clinical perspectives usually consisted of written questionnaires. The IOM viewed these instruments as inexpensive, easy to administer, and if computer scored, convenient to analyse. In this regard, they drew attention to issues of validity and reliability of the test instruments, and to common methodological problems such as poor response, positive response bias and poor patient recall. The IOM also indicated that telehealth offered unique research opportunities through video records to correlate patient and clinical satisfaction data. Such information was thought to be invaluable in order to improve telehealth practices.

Based upon their assessment, the IOM proposed a number of questions to guide examination of acceptability from both a patient perspective (Figure 7, below), and a clinician perspective (Figure 8, below).

Stability of Patient Perceptions in Telehealth

The IOM was particularly apprehensive about the stability of patient perceptions in telehealth and recommended this as an area for further study. The following example best illustrates this point. Evidence presented (unpublished) indicated that although patients in a telecardiology study had not found the experience unpleasant (93%), nor an invasion of privacy (95%) nor were they overly concerned about the lack of physical contact (88%), only 67% and 51%, respectively, chose to use the service again for either an emergency or a follow-up appointment. One year later, only 30% would use the telehealth system in an emergency. Furthermore, if telehealth were the only option, a full 30% would choose to go elsewhere.

D. COST - Evaluating Health Care Costs and Cost-Effectiveness

Theme questions :

• What were the costs of the telehealth application for participating health care providers or health plans compared to the alternative(s)?
• What were the costs of the telehealth application for patients and families compared to the alternative(s)?
• What were the costs for society overall compared to the alternative(s)?
• How did the costs of the application relate to the benefits of the telehealth application compared to the alternative(s)

Telehealth Costs and Perspectives

According to the IOM, the question as to whether or not telehealth was a good investment rests with a definition of its true purpose, the range of alternatives to which it was compared and to the costs associated with each of these alternatives.

Any economic evaluation should measure the costs and benefits of alternative ways of managing a condition. Costs are intended to measure the value of resource use associated with the intervention. Cost-effective analysis compares costs and health effects of at least two alternatives in natural units (years of life gained) whereas cost-benefit analysis expresses both costs and benefits (years of life gained) in monetary terms. Costs may include both variable (operational) and capital costs and analyses should focus on differential, or marginal costs.

Most public policy decisions concerning the costs of telehealth incorporate the societal perspective, which analyses the total cost of resources used to provide a service versus the alternative. A cost analysis may also be used to determine how monetary costs and savings are distributed among the various parties of any society such as patients, providers, and insurers. The perspective of analysis is of particular importance in telehealth, as improved access to health care has, more often than not, been the reason for its inception.

If telehealth applications require large start-up costs that cannot be reasonably shared by other users, then the application will result in a higher per cost unit. Initially, low use of the telehealth application will produce similar results. If the health effects or cost implications can be shown to accrue over time, then these costs may be discounted. What is important is a clear documentation of the actual use and the per unit cost of the resources required to provide the telehealth service.

The IOM recognised a number of major challenges in carrying out a cost analysis. First, given that telehealth technology is new with sizeable fixed costs and multiple potential uses, the apportioning of joint costs of production over users requires a compromise between accounting conventions, which applies administrative rules to costs, and economics which allocates joint costs according to demand for each service. Cost analyses cannot possibly resolve such issues. Second, telehealth applications once in place, may lead to alternative applications, such that per unit costs of telehealth may decrease, while the total use and expenditures may increase. Third, rapid technological change in telehealth may render meaningless a static study of the costs, benefits, and harms. The IOM recognised that telehealth, as a dynamic process, requires ongoing evaluation and as indicated earlier, a sensitivity analysis may be required to factor in such variables as expanded applications - and proficiency related-cost reductions. As with other forms of technological evaluation, investigators of telehealth may not be able to determine the effect of the technology on costs over an episode of illness. Moreover, the longer the interval that needs to be tracked, the more difficult it will be to collect and to attribute relevant data to the clinical telehealth intervention.

Questions pertaining to health care costs and cost-effectiveness are presented in Figure 9, below.

Decision Rules for Analysing Cost-Effectiveness Results

The IOM, in referring to the literature, suggested that for certain configurations of cost-effectiveness, decisions are straight forward, for example :

• If an alternative is more costly and performs less well, then reject it.
• If an alternative is more costly and performs as well, then reject it.
• If an alternative is less costly and performs better, then accept it.
• If an alternative is less costly and performs as well, then accept it.

For other configurations, the decision is less clear, for example :

• If an alternative is more costly and performs better.
• If an alternative is less costly and performs less well.

In such cases, the IOM acknowledged that cost effectiveness at best can only be a guide to decision-makers. It has been suggested that a year of healthy life gained for less than $100,000.00 may be reasonable. This however, does not factor in the overall budgetary implications.

'Ideal' desirable attributes of evaluation criteria

The 'ideal' attributes of evaluation instruments put forth by the IOM should apply equally to all evaluation criteria and to quantitative and qualitative evaluation measures alike. For most of these attributes, the IOM recommended the introduction of a 'controlled vocabulary' to provide uniform descriptions of patient problems. Such a vocabulary currently exists; the Uniform Medical Language System (UMLS). This system is designed to remove ambiguous terms and should introduce uniformity in the medical information systems by precluding the use of unauthorized terms.

The attributes of evaluation criteria include the following : reliability, validity, responsiveness, interpretability, feasibility, flexibility, and documentation.

According to the IOM, an evaluation instrument or criterion is :

• reliable, if repeated use under identical circumstances by the same or different users produces the same results.
• valid, if it measures the properties, qualities, or characteristics it is intended to measure.
• responsive, if it can detect important differences in outcomes across evaluation groups or time periods.
• interpretable, if users find the results of its application understandable.
• feasible, if users can accomplish the required activities, collect the necessary information, and analyse the resulting data within available evaluation resources and without imposing excess burdens on those whose cooperation is required for the evaluation.
• flexible, if it is adaptable to a variety of evaluation problems or circumstances.
• documented, if the protocols for applying and interpreting it are specified and if evidence of its successful use is summarised or cited.

Alternate Frameworks and Perspectives

In this report, the IOM framework has been selected as the single most significant and complete contribution to telehealth evaluation at this time. However, other models or perspectives exist. Awareness of these alternates provides a clearer understanding of telehealth evaluation, and complement the IOM framework. Below, selected studies are identified and their content synthesised within distinct subsections that broadly describe a topic area.

Evaluation of cost, quality and access

• Bashshur RL. On the definition and evaluation of telemedicine. Evaluation of Telemedicine Systems p 25-29. Telemed J. 1995 Spring; 1(1): 19-30

Although other technical systems in medicine have been adopted without being subject to the same level of scrutiny as telehealth, one would argue today that evaluation is both necessary and useful in both the private and public arenas. Evaluation of the first generation telehealth projects demonstrated that the telehealth technology of the time often far exceeded community needs and their capacity to use them, such that cost-effectiveness could rarely be established. Although evaluation of telehealth had demonstrated clinical efficacy, few conclusions could be drawn on the effects of telehealth on the health care delivery system.

Optimal evaluations of telehealth require that optimal telehealth systems be put in place. The health care needs of the community and providers must be defined, the 'informational' requirements for remote diagnosis, follow-up, and education must be identified, and the technical capabilities of the system must be fully exploited. Rigorous research designs can then be used to determine the effects of telehealth on access, cost, and quality of health care. Telehealth systems constitute 'innovation bundles' and as such it is futile to consider them as fixed models of health care delivery.

Two types of research questions are appropriate for telehealth evaluation. The first involves biomedical research, and the second involves health services research. In biomedical research evaluation, the diagnostic and therapeutic accuracy, precision, reliability, sensitivity / specificity, and safety of the telehealth technology is assessed and compared to the clinical standard, usually the traditional 'face-to-face' model of health care delivery. This is carried out through performance studies and clinical trials.

In health services research, the focus is on the effects of telehealth on health care delivery in terms of access, quality, and cost of health care from the perspectives of clients, providers and society. Acceptance of the technology is also considered. The methods involve field studies (surveys, field observations) and experimental studies (controlled and quasi-experimental).

Evaluation of telehealth through health services research consists of three consecutive stages : evaluability assessment, formative evaluation, and summative evaluation. Evaluabilty assessment includes the determination of specific problems and issues to be evaluated (access, cost, and quality) and specification of objectives in terms of the major stake-holders (benefits and costs to clients, providers, institutions and community). Formative (or intermediate) evaluation examines the effects of the system on the delivery of health services, and then summative (or ultimate) evaluation examines the effects of telehealth systems on health outcomes.

Formative assessment of health care delivery includes examination of, for example : content of care (diagnosis, treatment, follow-up, prevention); process of care (scheduling, waiting time, service time, patient flow, case finding); and intermediate outcomes (outpatient visits, hospital admissions, length of stay). In addition the effects on clients (functional status, satisfaction, access to care, knowledge, attitude); providers (patient load, patient mix, satisfaction, knowledge, attitude); institutions (productivity, efficiency, provider mix); and the community (availability of health resources, local economy, cost-sharing, and other human services) are noted.

Bashshur warns that the short-term policy assessments derived from the results of demonstration projects is fraught with difficulties, mainly because of the scientific requirements underlying sound evaluation. These requirements encompass program objectives, program outputs / effects and validity threats.

• Bashshur RL; Grigsby J. Position paper : telemedicine effects : costs, quality, and access. J Med Syst. 1995 Apr; 19(2): 79-80

This paper contends that the cost of care issue in telehealth evaluation is the leading concern given that, historically, high levels of technology have more often than not contributed to the higher costs in health care. Evaluation of cost issues has been hampered by the absence of consensus on a precise definition of telehealth and its specific role in health care delivery, and also by the lack of 'fully optimal systems' from which to collect and analyse valid and reliable cost data. With respect to a consensus definition of telehealth, the author recognises the confusion generated over the distinction between telehealth and telemedicine. There is, however, general agreement that telemedicine is subsumed under the broader concept of telehealth, where telehealth includes patient care, education and research applications. The lack of optimal systems from which to analyse data emerges primarily from the limited feasibility and clinical effectiveness format of many of the first generation telehealth projects.

Bashshur recommends that evaluation research must now examine the impact of telehealth as a 'system of care in health care delivery' and determine the effects of telehealth on the cost of health care, the quality of care, and peoples access to it. The perspective of the analyses must include clients, providers, and society as a whole.

Economic analyses should include analyses of total costs, investment costs and return on investment. Total costs refer to the function of volume of service utilisation and price of units of service (analysis of reimbursable costs compared to traditional arrangements (payers)). Investment costs reflect the sum of capital and maintenance costs (planners and developers). Return on investment refers to the relationship between investment costs and specified effects or net benefits (policy makers). The most important return on investment is the effect of telehealth on health (preventing disease, treating illness, ameliorating pain and suffering) derived by comparisons with alternative health care systems.

• Bashshur RL. Telemedicine effects : costs, quality, and access. J Med Syst. 1995 Apr; 19(2): 81-91

This paper suggests that a dearth of systematic research exists on these issues (costs, quality, and access); that telehealth represents a technology-based response to some of the most intractable problems in health care delivery (high cost, uneven quality, maldistribution, and limited access); that optimal systems must be designed to match the specific needs of target populations; and that a clear definition of telehealth and telehealth systems is needed.

What are telehealth effects (costs and benefits) on the health care system with particular emphasis on costs and its interactions with quality of care and access to care? Clinical feasibility and relative effectiveness of telehealth has been established for specific clinical services and technologies are limitless. Potential benefits of telehealth accrue to substitutions (such as the site of care, type of service provider, and content and process of care). The merit of telehealth derives from its ability to distribute and control the use of medical services for maximal health benefits.

Bashshur offers a framework for telehealth assessment :

Essential Perspectives	Types of Effects
	Accessibility	Cost	Quality
Client
Provider
Society

Comprehensive assessment of telehealth encompasses three main effects and three essential perspectives and their interactions, as shown above.

Accessibility refers to the relative ease or difficulty of obtaining health care services. A client's accessibility is defined in terms of the extent to which they face geographic, economic, architectural, cultural, and / or social barriers to health care. Client target populations and the major beneficiaries of telehealth include the geographically remote (rural areas), the institutionally confined (correctional, nursing homes), and those traditionally underserved, such as inner city residents and the elderly. The bridging of distance and time barriers between clients and providers will direct savings primarily to the clients. Telehealth may also improve access to consultants and referral sources for remote providers while central providers may develop a larger and more diverse provider and client base. From the perspective of society as a whole, telehealth may improve the quality of life in rural communities if people are willing to accept the longer service times.

Cost effects in telehealth can be analysed with cost-effective analysis (CEA) and cost-benefit analysis (CBA) once the desired outputs, or sets of inputs (interventions), and costs have been defined. The preferred approach in telehealth evaluation is to compare telehealth to the alternative arrangement(s). Essentially, CEA determines the least costly alternative to achieving stated objectives while CBA ranks by order of cost a list of desired objectives. The CEA approach is one of sub-optimization. Both approaches concentrate on measurable costs and benefits.

Quality effects in telehealth are technical (continuity and coordination of care, timeliness, specialist consultation, and a "built in" second opinion) and interpersonal (personal treatment and satisfaction of both provider and client). The central provider's quality derives from the monitoring and control functions made possible by telecommunications and computer technology. They also incur increased experience in their speciality and greater exposure (appreciation) towards remote medicine. The remote provider's quality may relate to a reduction in isolation and educational opportunities. From a societal perspective, one would reduce high rates of usage for special conditions in small areas, thereby reducing overall costs.

Interactions include that of accessibility and cost, such that should the use of the service increase without a commensurate decrease in price, the total costs of heath care will rise. This must be measured against the level of need and the subsequent effects on health. If quality of care improves in the rural setting, such as in adherence to prevailing norms, what would be the interactions with cost?

Finally, more research on individual experiences with telehealth is required and the potential for perceived development of a two-class health care system needs to be addressed.

• Bashshur RL. Critical issues in telemedicine. Telemed J. 1997 Summer; 3(2): 113-26

Should telehealth technology address specific needs or address broader concerns within health care ? Often technology is a solution looking for a problem, i.e., the 'technological imperative' in telehealth. Unfortunately there is also a lag between technological advances and our ability to capitalise on them. The search for solutions should start with clinical and health care problems and once this has been resolved, organizational and fiscal structures should be developed to maximize benefits, control abuses and share costs. This paper also advocates the development of telehealth health networks (e.g., LAHN, WAHN, HAHN, and IAHN).

Carrying out scientific evaluation of telehealth is often at odds with the short-term goals of policy-makers, especially with respect to making decisions on mature, steady-state telehealth operations. The true merit of telehealth can only determined by comparing it with alternative arrangements, yet no where does the knowledge of telehealth and its effects on health care begin to approach the knowledge of the alternatives. Moreover, the results of short term telehealth evaluation studies are hampered by scientific requirements in that a programs objectives are not or cannot be precisely defined, described, or measured. Program outputs are often not attainable during the evaluation period and program effects may be numerous and conflicting. Difficulties with validity arise due to sample selection bias, program implementation bias, history effects, small sample size effects, and the lack of mature, steady-state systems from which to derive measures of outcome.

The hazards of limiting telehealth to rural applications is discussed in terms of the perceived equity of a two-tier medical system. Also, issues related to the professional maturation of telehealth are discussed in terms of unexpected hesitancy and reluctance on the part of health care providers. The solution to the widespread utilisation of telehealth is proving its cost-effectiveness. Cost effectiveness is related to issues of cost containment associated with the introduction of structural changes, to telehealth substitution rather than supplementation, and to curtailing inappropriate use.

Program objectives require that explicit (measurable) objectives must be specified knowing that many may not be measurable or may be unanticipated. Determination of program outputs / effects include judging when and under what conditions it is appropriate to collect the requisite data. Timing of data collection should occur under normal 'steady state' conditions and only when the system has reached maturity and full fidelity (strength and integrity).

Telehealth is a complex intervention ('innovation bundle') and therefore suffers from validity threats. Validity threats include among others, selection bias of subjects, implementation bias (maturation required), and history effects (midcourse changes often required). These may be best addressed through randomised, controlled trials (RCT's) if possible with low volumes of use, or through quasi-experimental designs where prospective random assignment, manipulation of experimental variables, and blinding of subjects is not required.

• Bashshur RL. Rethinking the evaluation and priorities in telemedicine. Telemed J. 1998 Spring; 4(1): 1-4

According to this editorial, the true merit of telehealth has yet to be demonstrated and verified on a convincing scale with respect to the problems of cost, quality and access to health care. On a positive note, any question about clinical safety should be put to rest as problems with most telehealth applications relate to difficulties with specific technological configurations. Moreover, continued research on the economic or clinical merit of telehealth based on technological considerations is futile given the pace at which IT renders obsolete the assumptions made about its current capabilities and costs.

Evaluation of telehealth is identified as being difficult. This is related to the inability to take telehealth applications to maturity (full-fidelity implementation as systems of care and steady-state operation) until reimbursement for services is provided (in the U.S., Congress has voiced concern with : clinical safety, standards, reimbursement, over-utilisation, and financial liability of beneficiaries), and to measuring the impact of the telehealth technology before the end point to the development of that technology. In light of these difficulties, more short-term objective measures must be established. Recognition is given to non-objective issues such as benefits to rural providers (reduced isolation, access to medical advances, relationship with tertiary care specialists, increased medical knowledge etc) which should also be considered in any evaluation.

The article also notes the need for development of telehealth 'triage criteria' in order to determine when to use telehealth vs patient transport.

• Haughom JL; Gibson LJ. Improving the cost, quality, and access to healthcare in community hospitals through the use of reorganized integrated delivery systems and implementation of sophisticated clinical information systems : an organizational experience. Medinfo. 1995; 8 Pt 2: 1558-61

Containing rising costs and declining access in health care while maintaining quality is dependent on two components : development of an integrated delivery system (IDS) for all stakeholders in the health care delivery process (physicians, hospitals, and payers) and on the implementation of advanced information systems. In this regard, the Community Health Record (CHR) is being implemented, the cornerstone of which is the computerized patient record (CPR). The CPR is an on-line longitudinal record of all patient problems and treatment modalities collected across multiple venues of care. The CHR, by improving access to information and standardising operations, produced a data model to support health care delivery while eliminating costly documentation with the CPR, and served as a vital link to monitor cost, quality and access to health care. The effects of the CHR and the CPR are clearly documented with flowcharts.

• Huston JL; Smith TA. Evaluating a telemedicine delivery system. Top Health Inf Manage. 1996 Feb; 16(3): 65-71

Telehealth allows health care providers to 'move information rather than people'. Evaluation of telehealth is recognised to be ongoing, and should focus on three major areas : quality of patient care, legal issues including patient confidentiality, and the cost-effectiveness of medical services.

Quality of patient care issues include reliable and replicable diagnoses, patient response to treatment, identification of types of consultations and their respective risks as compared to no consultation or a delayed consultation and other morbidity and mortality risks. Technical issues to be examined are minimal standards and user satisfaction.

Legal issues include protection of the confidentiality of patient information. This issue alone has lead to the relatively slow development of the computerized patient record. Other issues include data ownership, malpractice, and physician licensing.

Cost-effectiveness of health care delivery must be compared to the telehealth approach.

The authors indicate that much of the information needed to effect an evaluation plan is already collected but it is not standardized nor monitored for quality. They also suggest the introduction into telehealth evaluation of existing quality of care review programs (e.g., speciality areas should review respective parallel telehealth programs).

Clinical efficacy and safety evaluation in telehealth; CTCG / SWOG

• Perednia DA. Telemedicine system evaluation, transaction models, and multicentered research. J AHIMA. 1996 Jan; 67(1): 60-3; quiz 64-5

The most basic form of technology evaluation of telehealth concerns safety and efficacy. Thus, the technology must be able to transmit "medically important data so that correct diagnoses can be made and appropriate treatments instituted." The effect of telehealth must be compared to those acquired under controlled laboratory conditions referred to as the 'gold standard' of good clinical practice.

Analyses of the clinical utility of telehealth must also be carried out on the way it effects the processes of medical care delivery by comparing the way equivalent cases are handled in the absence and presence of telehealth. Medical effectiveness is defined as the accuracy (diagnostic test) or therapeutic effect (treatment) under actual field (not laboratory) conditions. It may not, therefore, be possible to measure effectiveness in a clinical trial. Surrogate measures can be used to measure the concordance of diagnosis and treatment plans under telehealth, and compare them with those acquired from the laboratory and the clinical follow-up. Cost-effectiveness of telehealth must also be compared to the alternative methods of health care delivery. Cost data must be collected for both patients and providers.

Measuring clinical utility and cost-effectiveness of clinical telehealth requires robust, simple, and effective data collection instruments. However, the rural nature of telehealth, restricted time frames, and the limited number of cases renders it virtually impossible to collect enough statistically valid data over a reasonable amount of time. The author states that "probably there is not a single telehealth project in the USA that is, by itself, currently capable of providing meaningful data on medical efficacy, clinical utility, or cost effectiveness".

A potential solution to this problem is to create a mechanism for pooling data from many telehealth projects. In this regard, Clinical Telemedicine Cooperative Group (CTCG; based on SWOG, South-West Oncology Group) has created an infrastructure of research and administration to support its members so that they can undertake co-operative evaluations of their own projects. With this approach, geographic or site specific successes or failures can be identified and hardware configurations can act as independent variables. Evaluation indicators in health care are categorized as input, process and outcome indicators, and good research design will focus on the best indicators to answer the questions asked.

• Houtchens BA; Allen A; Clemmer TP; Lindberg DA; Pedersen S. Telemedicine protocols and standards : development and implementation. J Med Sys. 1995; 19(2): 93-119

The features of telehealth that need to be assessed through evaluation include safety and efficacy, clinical utility, and cost effectiveness. Evaluation of safety and efficacy must address telehealth's impact on technical capability and reliability, diagnostic accuracy, diagnostic decisions, therapeutic decisions, therapeutic capabilities, knowledge acquired and retained by primary care providers, and patient outcomes. Evaluation of telehealth has yet to address most of these measures.

Protocols for evaluation should specify what is to be measured or observed (indicators), how the measurements or observations will be obtained and recorded reliably (methods), with what the measurements or observations are to be compared (controls or 'gold standards'), how many measurements are needed to draw statistically valid conclusions (power analysis), and how will the results be published (external peer review).

CTCG will enable institutions to devise, share, and abide by high quality protocols for multi-site evaluations.

Context-centered evaluation in telehealth

• Burghgraeve P; De Maeseneer J. Improved methods for assessing information technology in primary health care and an example from telemedicine. J Telemed Telecare. 1995; 1(3): 157-64

The authors develop and argue for a context-centered framework for telehealth evaluation of primary health care applications that emphasizes the human dimension in contrast to the technology-centered evaluations.

Text analysis of articles from Medline keying on information technology, evaluation, primary health care, patients and patient satisfaction yielded over 200 articles. Further text analysis on selected articles (76 of 200) keying on 'evaluation' in connection with health care information systems (+ / - the application on information systems) produced three main categories. Articles that dealt with methodological problems (22%), how and why health information systems are accepted (20%), and evaluation development, use of particular systems, and empirical research concentrating on specific aspects (58%).

The articles selected showed the following striking characteristics :

1. Evaluation research concentrated on what was technically wrong (reliability) with health information systems and if they transcended technical issues they referred to patients or cost-benefits.
2. Articles often reflected the 'opinion' of an authority.

Evaluation, therefore, has to overcome the restrictive view that one should take technological solutions to health care medical problems for granted, and the view that all technology represents progress. The authors proposed a contextual frame (see below) that allows an evaluation to question the introduction of specific applications of IT. Telematics in health care was the focus. Questions such as 'what environmental aspects are disturbed by the introduction of a new information system' are addressed. It is important to note that in developing a contextual frame, one is unavoidably guided by subjective elements.

In structuring the contextual information into a meaningful whole, the first assumption was that environmental differences were important. The overall medical goal in 'acting for the good of the patient' required a specific interpretation. The health care provider's aim was to ensure the patient the best possible primary health care (standard of WHO). In the contextual frame, primary care may be represented by first contact or accessibility of care, continuity of care, and comprehensiveness of care. All three aspects of care should be delivered to the patient, hence an evaluation of information systems cannot be reduced to one focus i.e., on only one aspect of care. It is then necessary to clarify what is meant by accessibility, continuity, and comprehensiveness of care. Finally, all aspects of the primary health care environment, namely structure, process, and outcome, are taken into account, to help make tangible the abstract concepts of accessibility, continuity, and comprehensiveness of care.

Table : A contextual frame for the assessment of telematics in primary health care.

	Accessibility / First Contact	Continuity of Care	Comprehensiveness of Care
Structure
Process
Outcome

• Whitten PS; Allen A. Analysis of telemedicine from an organizational perspective. Telemed J. 1995 Fall; 1(3): 215-26

The paper describes how, in a typical telehealth program, each individual site saw less than 1 consultation per week during 1994. The low utilization rates were attributed to cost factors and technological barriers. To that time, most evaluations of telehealth concentrated on medical aspects, some on cost-effectiveness, but none on organizational issues. Through a case analysis of the University of Kansas telehealth program involving direct observation, interview, and survey, organizational factors were found to be critical to the success of telehealth.

An inductive approach was used to develop research questions after unstructured interviews with key members of the medical centre and rural telehealth providers. The following research areas were selected for detailed study : structure and boundaries, organizational definition and goals, leadership and decision making, and membership and roles. Questions were formulated in each of these areas and observation, interview and survey techniques were used in the data collection.

An observation phase contained 100h of watching telehealth participants at work (at least 1 full day at each rural site (8) and six weeks at the medical centre hub. Interview goals included interviews with at least 50% of telehealth providers (75% rural / 25% medical centre) which included administrators, nurses, physical therapists, technicians, and clerical staff, and 1 randomly selected rural physician from each of the 5 rural sites.

The survey phase was based on content analysis of interviews and on observation data. It was constructed to measure levels of agreement (using Likert scales) within the 4 research areas previously defined. All personnel were also surveyed by mail.

Structures and Boundaries : Despite being the first site set-up in 1991, with 5 sites added later, no organizational structure was apparent, especially for the rural providers. Boundaries were difficult to identify so they were defined by the clients that telehealth served. Primary clients were identified as patients needing to see a specialist (95%). With respect to boundaries, 80% indicated that telehealth begins when a physician decides that a patient needs a consultation and 50% stated that telehealth care is ongoing.

Organizational Definition and Goals : Most agreed (90%) that the overall purpose of telehealth was "to provide access to specialized health care for patients living in rural areas" and 88% agreed with the statement that it also functioned as a "back-up for health-care practitioners in rural areas." Most could not articulate any organizational goals to increase the use of telehealth.

Leadership and Decision Making : The majority (85%) indicated that they had no idea who was in charge of the overall telehealth program. The leader at the rural site could always be identified. Respondents indicated that over 50% of the decisions in a telehealth consultation involved medical treatment and diagnoses and less than 50% felt it involved scheduling issues. In line with this, 60% agreed with the statement that the consulting physician made most of the decisions during a consultation and 40% all parties involved in a consultation jointly made decisions.

Membership Roles and Responsibilities : These elements were ambiguous as a 'seat of the pants' philosophy reigned. Ironically, setting-up the telehealth consultation (phone) was frustrating and burdensome (85%). Most respondents felt that one phone call is what it should take (90%). Other issues of concern were the lack of formal feedback (70%), knowledge about the technology (70%), services (75%), and training (95%).

Ancillary Findings : Most felt (95%) that the 'no hands' component of telehealth was the greatest difference from traditional consultations yet more similarities than differences were observed (80%).

This paper identified major problems with respect to organizational structure, leadership, technology (and how it is to be used) scheduling, strategic goals and communication with the medical community and the public.

Economic evaluation of telehealth

• Lobley D. The economics of telemedicine. J Telemed Telecare. 1997; 3(3): 117-25

Between 1990 and 1995 the number of funded telehealth trials in the USA grew from 5 to over 100. Most of these trials were technology-driven and engaged in remote diagnosis and consultation, CME, and medical informatics. Two major questions emerged from these trials with respect to telehealth and health care. Will telehealth add to medical costs or provide a means for reducing them? And, what will the possible effects of telehealth be on the demand for health care. Published information is, therefore, required on the economic costs and benefits of telehealth. The effects of telehealth on the activities, functions and roles of hospitals, consultants, GP's, and nurses is also required before large-scale implementation of telehealth can take place.

There is a considerable lack of data on the economics of telehealth and consequently no framework for analysis. It is likely that after the introduction of telehealth, specialists would receive a greater number of requests for services resulting in the need for new protocols for payment and liabilities. Telehealth is untenable without reimbursement. In the USA, there is concern with the 'unreasonable use of telehealth' and destabilisation of rural providers and hospitals. At the time of publlication payment policies still required 'face-to-face' interactions. Although it is clear that the cost of the requisite technology has decreased, many barriers to widespread use of telehealth still exist. These include the relative cost of telehealth to various parties vs alternatives, specialist reimbursement, specialist liabilities (local, national, international), suitability of telehealth with respect to different medical conditions, changes in working practice and relationships, confidentiality and data security, technical compatibility, and acceptability to both providers and users.

Structured economic analysis (i.e., the analysis of the cost and benefits of telehealth) should be able to answer many of these questions, but little work has been carried out on these issues thus far. Costs, savings, and benefits may be examined by speciality or disease category, and by processes of health care such as triage, pre-transfer, urgent evaluation; medical / surgical follow-up and medication check; supervision and primary care (no physician available); routine consultations and second opinion based on history, physical examination, and test data; transmission of diagnostic images and medical data; management of self-limited conditions and chronic diseases; and public health, patient education, and preventative medicine. Costs include capital and variable costs. Savings vary according to speciality but might include patient movement, staff travel, time savings, saved laboratory tests, better use of specialists, scheduling, and more effective treatment and rapid patient recovery. Pricing of medical services where relevant would be under the influence of normal market conditions such as a wider more affluent market, shop around patient strategies, and price undercutting.

Structural changes that are difficult to quantify after the introduction of telehealth include among others, greater specialization of skills and knowledge transfer, demand for second opinion, scandalisation of telehealth diagnoses and treatment, and changes in work patterns.

Key markets for telehealth are radiology, dermatology, cardiology, education, emergency / trauma, psychiatry, obstetrics, gynaecology, pathology, orthopaedics and, by the end of the decade, would include CPR, medical image files, radiological networks, dermatology, pathology, and CME.

Existing and potential opportunities for telehealth can be classified as follows : only option (space, battlefield, disasters and emergencies (airlines)); remote medical provision option compared to other alternatives (relative costs and benefits of telehealth vs travel option); movement of patient option is undesirable (prison); urban option (cost reasons and increased communication); convenience option (best expertise and second opinion) and new markets option (additional facility).

• McIntosh E; Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare. 1997; 3(3): 132-9

Economic evaluation of telehealth based on a cost-consequence framework is presented. Key evaluation questions are linked to cost-consequences. Issues dealt with include changing technology, study design & small sample size, inappropriate of standard economic evaluations, and valuation of both health and non-health outcomes.

An important aspect of economic is opportunity cost. Its evaluation is concerned with minimizing these costs so that the best use can be made of scarce resources. Economic evaluations identify, measure, value, and compare the costs and consequences / effects of the alternatives under question. The three main techniques of evaluation include : cost-effective analysis (technical efficiency and unidimensional outcome), cost-benefit analysis (all costs and benefits are valued, usually in monetary terms), and cost-utility analysis (form of cost-effective analysis where the basic outcome is measures in 'healthy years'). Since in telehealth evaluation it is difficult to attribute health benefits or changes in health outcomes to telehealth, usually a cost-consequence matrix approach is used where the possible costs and consequences of the introduction of telehealth relative to the conventional system are compared.

Cost-Consequence Matrix :

Costs	Consequences
	+ve Consequences	No Difference	-ve Consequences	Insufficient Consequences Evident
Cost Savings	+	+	+ / -	?
Small + / - Costs	+	+ / -	-	?
Costs	+ / -	-	-	?
Insuf. Cost Ev.	?	?	?	?

+ = little difference in cost and beneficial consequences (service is worthwhile)
- = costs are greater and no difference in consequences
+ / - = greater costs and benefits
? = insufficient evidence, future studies recommended or sensitivity analyses

As a starting point for telehealth evaluation, information on costs and consequences may be outlined in a balance sheet. With this approach one can identify gaps in data, areas for sensitivity analysis, and areas where you cannot express cost and consequences in the same units, such as dollars. Therefore one documents costs on variables that can be valued while at the same time describing benefits in descriptive terms on variables that cannot be valued. In doing so, all variables remain in the decision tree.

Costs and consequences relevant to telehealth will reflect the cost, quality of care, and access to it from different perspectives of patients, clinicians, purchasers and providers. Costs include hardware, software, consultant's time, travel costs, running costs, administrative costs, staff changes, number of referrals and treatment costs. Consequences related to health include effects of bringing treatment forward in time and clinical confirmation via a built in second opinion. Consequences not related to health include improved quality of service, transfer of skills, speed of service, education and reassurance. Equity issues may also be addressed (i.e., who bears the cost and who receives the benefits, and the relative magnitude of these effects). Distribution weights may be established by 'willingness to pay' techniques.

The cost-consequence approach helps in identifying the main objectives of telehealth, but they can be more formally identified by using a set of key questions, i.e. :

1. When should an economic evaluation be carried out? Key points : Incorporate economic evaluation into the clinical study at the beginning, collect the relevant costs, and at the end of the study carry out a sensitivity analysis using updated costs. Identify, measure and value changes in the process of care where changes in health status cannot be carried out.
2. Whose perspective should the evaluation adopt? Key points : Costs and benefits to all sectors of society are affected by telehealth. Broad perspective and not solely concerned with financial flows.
3. Does the introduction of telehealth lead to an increase in capital costs? If so, are these additional costs offset by lower running costs? Key points : Usually there is a large initial capital outlay, plus maintenance and running costs, compared to conventional service. Other items include 'pay back', 'worth', sensitivity analysis based on future reduced capital and running costs.
4. Will the number and level of staff change; will their skill mix change? If so, what are the cost implications? Key points : Changing role of GP, changing role of Practice Nurse, throughput.
5. Will the costs of treatment of certain patient groups be increased or decreased? If so, by how much? Key points : Changes in management should produce changes in costs; e.g., telehealth's effect on bringing a diagnosis forward in time and its effect upon the treatment administered, the costs of that treatment, the health outcome for the patient should be determined.
6. How much will patient outcomes be improved, if at all? Key points : Lack of effectiveness data, to identify what changes occur in patient management. Most important question, 'Is the provision of a complete and timely set of diagnostic images important in terms of the health outcome'. To date, evaluations have focussed on 'outcome surrogates' (process measures). Does telehealth accrue health benefits, or is it simply another way of delivering exactly the same service. Patients psychological outcome is also important (reduce patient anxiety).
7. Are their non-health outcomes which should be included in the evaluation? Key points : Non-health benefits need to be identified, measured, and valued. Valuation is most difficult and may require 'conjoint analysis' taking into consideration the non-health attributes of a patient's utility function (patient satisfaction from goods and services received).
8. Will consultation and referral patterns change after telehealth? If so, what are the cost implications? Key points : Any increase in patient visits to the GP after introduction of telehealth has to be compared to the cost savings realized because of the expected decline in the number of unnecessary referrals to specialists.
9. Will activity levels change? Will these changes in throughput affect the cost effectiveness of the program? Key points : Increased activity of telehealth should bring down the cost of a telehealth cosultation and lower patient travel costs.

• Sisk JE; Sanders JH. A proposed framework for economic evaluation of telemedicine. Telemed J. 1998 Spring; 4(1): 31-7

Telehealth differs more in 'degree' than in 'kind' from other technologies, and therefore with minor exceptions is subject to the same principles of economic evaluation. The fundamental question addressed in this paper is how services delivered with telehealth compare with those delivered through alternative means. Most importantly, if the net benefit is acceptable, economic analysis will determine whether the net benefit is worth the cost.

Cost-effective analysis, where the costs and health effects of at least two alternatives are compared, is presented with respect to their effects / consequences (health benefits or health outcomes). In this analysis, intermediate outcomes may come into play and dynamic modeling may be useful as consequences may change over time. Costs measure the value of resources used, and measuring the actual use and per unit cost of resources is preferable. Cost must be examined from the perspective of society, third parties, patients and providers. Decision rules for analysing results is then presented. Conceptual challenges specific to an economic evaluation of telehealth include multiple usage, expanded use, and technological changes. Sensitivity analyses may then be used to vary these values over reasonable ranges. The paper concludes with a brief discussion of payment options for telehealth such as fee-for-service, bundled, and capitation.

When and how to assess fast changing technologies

• Mowatt G; Cairns JA; Bower DJ; Grant AM; Brebner JA; McKee L. When and how to assess fast changing technologies : a comparative study of medical applications of four generic technologies. Health Technol Assess. 1997; 1(14): i-vi, 1-149

This paper questions if there is an optimal time to assess fast-evolving health technologies, and examines what factors influence the timing of assessments and the choice of methods. Effective evaluation is made increasingly more difficult by the rapid diffusion of new technologies, media influences, and the influences of product champions.

Literature searches concerning the timing of health technology assessments, actual assessments of both new and mature health technologies, and interviews of key individuals were carried out. The results as to when to assess were inconclusive, yet initial and continuing assessments were recommended. Many factors influenced the timing of assessments, but not in any scientific sense. Product champions and opinion leaders introduce new technologies into clinical practice leading to rapid diffusion, but this may then limit proper evaluation. This effect can often be countered with regulatory controls. Funding of telehealth initiatives, often restricted in time are too often technology based. Media coverage can influence evaluation in either way. Randomised controlled trials are often compromised by the time physicians acquire a preference of the traditional mode of health care delivery vs alternative. Evaluations should be all encompassing and comprise clinical outcomes, social, ethical, effect on patterns of health care demand and cost effectiveness, not just technology or commercially driven, but needs based.

• DeChant HK; Tohme WG, Mun SK, Hayes WS; Schulman KA. Health Systems Evaluation of Telemedicine : A Staged Approach. Telemed J. 1996; 2(4): 303-312

Previous comprehensive technology assessments described a three stage model evaluating the : 1) technical characteristics; 2) efficacy (usually under controlled conditions); and 3) clinical (patient), social, and economic outcomes of health technologies. In this paper a staged approach is proposed that has the potential to evaluate both the technical performance of specific telehealth applications, and also to assess their overall effects on the health care system. To accomplish this the type of assessment is varied with the evolution of the technology. The domains of concern are the effects of the telehealth application on the quality, accessibility, or cost of medical care. These endpoints were chosen by the parties involved in the communication pathway and the perspective of analysis.

Stage I : Technical efficacy (prove capabilities of the technology). This stage of evaluation should address the capabilities of the system to transmit accurate information reliably. This may be determined through case reports, case series, ROC analyses, and ultimately by comparisons of the efficacy of telehealth with existing practice. Once the telehealth application has been deemed reliable and accurate, it is possible to move on to Stage II analyses.

Stage II : Specific systems objectives. At such an early stage of telehealth development and diffusion, it is critical to assess single endpoint domains (quality, access, and cost) quickly and with small sample sizes. Methods used may include either randomized or quasiexperimental designs. Further product refinement should precede any global assessment in stage III.

Stage III : Systems assessment (global evaluation using multiple endpoints). The telehealth application should be mature, and the sample sizes large enough (power calculation) to enable the simultaneous analysis of multiple endpoints across the domains of quality, access, and cost. The over-riding goal is to determine the effects of the telehealth application on the health care delivery system. Since the unit of analysis in this case is the health care delivery system, randomized designs will be difficult to implement.

Stage IV : Establishing external validity (extending evaluation to different systems). Although most health technologies have not been evaluated even at the level of Stage III, Stage IV extends this analyses to other systems of care (academic setting vs managed care). In both stages, large administrative units can be assigned to telehealth or conventional practice or alternatively, quasi-experimental designs can be employed.

It is proposed that technology assessments of telehealth occur along three axes. First, the stage appropriate to the telehealth application's level of development (Stage I, II, III and IV). Second, the communication pathway in the application (any health professional to patient, physician to allied health professional, physician to physician). Third, the domain of interest (quality, access and cost). The domain(s) selected depends on the perspective of analysis and on the parties involved in the communication pathway, with the most relevant domain serving as the primary study endpoint. If weighting of the domains is necessary, this should be included in the analysis.

Clinical problem solving in telehealth

• Farland L; Lafrance JP; Arocha JF. Collaborative problem-solving in telemedicine and evidence interpretation in a complex clinical case. Int J Med Inf. 1998 Aug-Sep; 51(2-3): 153-67

This paper examines clinical problem-solving processes in the context of a telehealth consultation. Real-time clinical consultations between three regional hospitals and the academic center served as the data source, and one complex dermatology case was selected for detailed analysis. This case was identified on the basis of disagreement of diagnoses between two clinicians, evidence of pathophysiological reasoning and nodes belonging to most levels of ontological hierarchy. It was concluded that the technological context of the consultation did not overly contrive the interaction, allowing real-time problem solving processes characterizing medical reasoning in naturalistic settings.

Barriers to telehealth

• Kovner R; Havens DMH. Legislative news. Telemedicine : potential applications and barriers to continued expansion. J Pediatr Health Care. 1996 Jul-Aug; 10(4): 182-7

This paper provides a general discussion of telehealth, with a focus on barriers and legislative developments in the USA. Barriers include licensure, role of non-physician providers, legal liability, and confidentiality. Of specific interest for the USA were payment policies, identifying which specialities will be covered (radiology approved), privacy with respect to recording and storing of consultations, medical liability and acceptance of telehealth as a viable alternative to patient transfer. Currently a number of pilot projects designed to develop, test, and evaluate various payment methods, are being funded. The US is also concerned with cost-effectiveness as the Congressional Budget Office in 1994 indicated that health care costs would increase, because of the increased access to care that telehealth would provide. Other interesting thoughts include the potential to expand the role of the nurse practitioner, particularly in rural areas.

Conclusions

The specific literature reviewed here shows that no single evaluation framework is available that would satisfactorily address each conceivable telehealth application. The telehealth literature also demonstrates that, as a result, most telehealth applications are either not well evaluated, or are evaluated in an ad hoc manner. This has serious implications for the immediate future; the consequences of inaction are considerable. As with other aspects of telehealth there is an urgent need to ensure all Canadian telehealth programs are evaluated in a consistent manner.

At this time Canada lacks a single approach to evaluation of telehealth applications. Adoption of a consistent approach could be instituted through a policy requirement that the evaluation framework of the IOM be required as the base evaluation framework, with application specific evaluations superimposed upon this model.

Recommendations

Health care resources are already limited. Telehealth proponents must ensure that their combined efforts lead quickly and effectively towards identifying or refuting the perceived benefit of any single telehealth application. To do this will require acceptance of, and adoption of, a single, telehealth evaluation model.

The education of policy-makers and decision-makers in the benefits and limitations of this model would ensure 'user acceptance'. This in turn would then lead to credible and evidence-based decision making.

To achieve the above, two recommendations are proffered :

• That the IOM evaluation framework be adopted as a minimum standard for all telehealth evaluations performed in Canada, with additional application-specific approaches and questions superimposed upon it by investigators and / or sponsoring agencies.
• That consistency be sought for application-specific approaches and questions through support of appropriate research initiatives within Canada.

It is crucial that a national standard be adopted that will permit consistent and comparable evaluation of all Canadian telehealth applications. Such a consistent approach could be instituted through a policy requirement. Significant benefits would accrue to both individual telehealth applications and to the telehealth industry within Canada as a result of such a policy approach.

Appendix I - Literature search - primary citations

From the electronic literature search with over 400 citations, a total of 90 were identified relating to 'evaluation' of telehealth or telemedicine. These are listed below. (Alphabetical order by author). Citations in italics were those identified as secondary references for further assessment.

Alexander M . Telemedicine in Australia. 1: The health-care system and the development of telemedicine. J Telemed Telecare. 1995; 1(4): 187-95

Balas EA; Jaffery F; Kuperman GJ; Boren SA; Brown GD; Pinciroli F; Mitchel JA. Electronic communication with patients. Evaluation of distance medicine technology. JAMA. 1997 Jul 9; 278(2): 152-9

Balas EA; Jaffery F; Pinciroli F . Patient care from a distance : analysis of evidence. Annu Meet Int Soc Technol Assess Health Care. 1996; 12: 17

Bashshur RL. Rethinking the evaluation and priorities in telemedicine. Telemed J. 1998 Spring; 4(1): 1-4

Bashshur RL. Critical issues in telemedicine. Telemed J. 1997 Summer; 3(2): 113-26

Bashshur RL. On the definition and evaluation of telemedicine. Telemed J. 1995 Spring; 1(1): 19-30

Bashshur RL. Telemedicine effects : costs, quality, and access. J Med Syst. 1995 Apr; 19(2): 81-91

Bashshur RL; Neuberger N. Telemedicine and health policy : is there life after recissions? Telemed J. 1995 Summer; 1(2): 151-4

Bashshur RL; Grigsby J. Position paper : telemedicine effects : costs, quality, and access. J Med Syst. 1995 Apr; 19(2): 79-80

Brebner E; Brebner JA; Ruddick-Bracken H; Norman JN. The development of a telemedicine laboratory as a medical faculty resource. J Telemed Telecare. 1998; 4 Suppl 1: 29-30

Brennan PF. Characterizing the use of health care services delivered via computer networks. J Am Med Inform Assoc. 1995 May-Jun; 2(3): 160-8

Burghgraeve P; De Maeseneer J. Improved methods for assessing information technology in primary health care and an example from telemedicine. J Telemed Telecare. 1995; 1(3): 157-64

Cameron AE; Bashshur RL; Halbritter K; Johnson EM; Cameron JW. Simulation methodology for estimating financial effects of telemedicine in West Virginia. Telemed J. 1998 Summer; 4(2): 125-44

Canadian Society of Telehealth. The impact of telemedicine on health care management. G7 SP4 Scientific meeting; Regensburg, Germany; November 1998. Report to Health Canada, December 1998

Canadian Society of Telehealth. Evaluation and cost effectiveness of telemedicine. G7 / G8 SP4 Forum; Melbourne, Australia; February 1999. Report to Health Canada, March 1999

Canna M. Western governors' policy group lists barriers to telemedicine. Hosp Technol Ser. 1995 Jul; 14(7): 10-2

Caryl CJ. Malpractice and other legal issues preventing the development of telemedicine. J Law Health. 1997-98; 12(1): 173-204

Crowe BL. Cost effectiveness analysis of telemedicine. J Telemed Telecare. 1998; 4 Suppl 1: 14-7

Crowe BL; McDonald IG. Telemedicine in Australia. Recent developments. J Telemed Telecare. 1997; 3(4): 188-93

Cochrane Library. The impact of telemedicine as an alternative to face to face patient care, on professional practice and patient care (Protocol for a Cochrane Review). Cochrane Library, Issue 1, 1999. Oxford: Update Software.

Cykert S; Flannery MT; Huber EC; Keyserling T; Moses GA Elnicki DM; Hannis M. Telephone medical care administered by internal medicine residents : concerns of program directors and implications for residency training. Am J Med Sci. 1997 Sep; 314(3): 198-202

Dakins DR. Evaluation studies show telemedicine's impact on outcomes. Telemed Telehealth Netw. 1997 Apr; 3(2): 8, 11

Dakins DR; Jones E. Cream of the crop : 10 outstanding telemedicine programs. Telemed Telehealth Netw. 1996 Dec; 2(11): 24-34, 37-4

De Chant HK; Tohme WG; Mun SK; Hayes WS Schulman KA. Health Systems evaluation of telemedicine : a staged approach. Telemed J. 1996 Winter; 2(4): 303-12

Ehler-Lejcher K. Tips on structuring a telemedicine program. QRC Advis. 1997 Aug; 13(10): 1-6

Emery S. Evolving technology thwarts aim of cost analysis. Telemed Telehealth Netw. 1997 Feb; 3(1): 20-3, 25

Falsone JJ; Moidu K; Sheenan CA; Gottlieb K; Galusha L; Nair S. Is telemedicine justifiable? Conn Med. 1998 Jul; 62(7): 423-7

Farland L; Lafrance JP; Arocha JF. Collaborative problem-solving in telemedicine and evidence interpretation in a complex clinical case. Int J Med Inf. 1998 Aug-Sep; 51(2-3): 153-67

Field MJ. Telemedicine : a guide to assessing telecommunications in health care. Washington, D.C. : National Academy Press, 1996 - IB: 0309055318

Fisk MJ. A comparison of personal response services in Canada and the UK. J Telemed Telecare. 1995; 1(3): 145-56

Gomez EJ et al.. A telemedicine system for remote cooperative medical image diagnosis. Comput Methods Programs Biomed. 1996 Jan; 49(1): 37-48

Grigsby J; Kaehny MM; Sandberg EJ; Schlenker RE; Shaugnessy PW. Effects and effectiveness of telemedicine. Health Care Financ Rev. 1995 Fall; 17(1): 115-31

Grigsby J; Schlenker RE; Kaehny MM; Shaughnessy PW; Sandberg EJ. Analytic framework for evaluation of telemedicine. Telemed J. 1995 Spring; 1(1): 31-9

Harrison RM; Clayton W; Wallace P. Is there a role for telemedicine in an urban environment? J Telemed Telecare. 1997; 3 Suppl 1: 15-7

Harrison R; Clayton W; Wallace P. The future role of telemedicine at the interface between primary and secondary care. J Telemed Telecare. 1996; 2 Suppl 1: 87-8

Hassol A; Irvin C; Gaumer G; Puskin D; Mintzer C; Grigsby J. Rural applications of telemedicine. Telemed J. 1997 Fall; 3(3): 215-25

Haughom JL; Gibson LJ. Improving the cost, quality, and access to health care in community hospitals through the use of reorganized integrated delivery systems and implementation of sophisticated clinical information systems : an organizational experience. Medinfo. 1995; 8 Pt 2: 1558-61

House AM. Who you gonna call? Telemedicine in Canada takes off. Leadersh In Health Services. 1996 Jul-Aug; 5(4): 4-7

Houtchens BA; Allen A; Clemmer TP; Lindberg DA; Pedersen S. Telemedicine protocols and standards : development and implementation. J Med Sys. 1995; 19(2): 93-119

Huston JL; Burton DC. Patient satifaction with multispeciality interactive teleconsultations. J Telemed Telecare. 1997; 3(4): 205-8

Huston JL; Smith TA. Evaluating a telemedicine delivery system. Top Health Inf Manage. 1996 Feb; 16(3): 65-71

Jennett PA; Hall WG; Morin JE; Watanabe M. Evaluation of a distance consulting service based on interactive video and integrated computerized technology. J Telemed Telecare. 1995; 1(2): 69-78

Jennett PA; Watanabe M; Hall WG. The use of advanced computer technology to enhance access to health care and to respond to community needs : the results of the evaluation of a technology-based clinical consultation service. Medinfo. 1995; 8 Pt 2: 1479-81

Kearney B. Health technology assessment. J Qual Clin Prac. 1996 Sep; 16(3): 131-43

Kovner R; Havens DMH. Legislative news. Telemedicine : potential applications and barriers to continued expansion. J Pediatr Health Care. 1996 Jul-Aug; 10(4): 182-7

Lemke HU. Economic impact of modern telematic technologies. Acad Radiol. 1998 Sep; 5 Suppl 2: S436-40

Lewis M; Moir AT. Medical telematics and telemedicine : an agenda for research evaluation in Scotland. Health Bull Edinb. 1995 Mar; 53(2): 129-37

Lobley D. The economics of telemedicine. J Telemed Telecare. 1997; 3(3): 117-25

Lott CM. Legal issues in telemedicine technology or Legal interfaces in telemedicine technology. Mil Med. 1996 May; 161 (5): 280-3

Masys DR. Telehealth : the need for evaluation (editorial; comment). J Am Med Inform Assoc. 1997 Jan-Feb; 4(1): 69-70

Maclean JR; Brebner JA; Norman JN. A review of Scottish telemedicine. J Telemed Telecare. 1995; 1(1): 1-6

McDaniel JG; Moehr JR; Muller HA. Impediments to developing wide area networks in health care. Medinfo Proceedings. 1995; 1491-1495

McIntosh E; Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare. 1997; 3(3): 132-9

Mendelson DN; Salinsky EM. Health information systems and the role of state government. Health Aff Millwood. 1997 May-Jun; 16(3): 106-19

Miles SM. Case study : telemedicine trial and applications development. Healthc Inf Manage. 1995 Summer; 9(3): 41-7

Mitchell BR; Mitchell JG; Disney AP. Uses adoption issues in renal telemedicine. J Telemed Telecare. 1996; 2(2): 81-6

Mowatt G; Cairns JA; Bower DJ; Grant AM; Brebner JA; McKee L. When and how to assess fast changing technologies : a comparative study of medical applications of four generic technologies. Health Technol Assess. 1997; 1(14): i-vi, 1-149

Ohinmaa A et al.. Economic evaluation of telemedicine in northern Finland. Annu Meet Int Soc Technol Assess Health Care. 1997; 13: 73

O'Rourke M; Gallivan S. Telemedicine : evaluation or stagnation. Stud In Health Technol Inform. 1997; 43 Pt A: 48-52

Pelletier- Fleury N; Fargeon V; Lanoe JL; Fardeau M. Transaction costs economics as a conceptual framework for the analysis of barriers to the diffusion of telemedicine. Health Policy. 1997 Oct; 42(1) 1-14

Perednia DA. Telemedicine system evaluation, transaction models, and multicentered research. J AHIMA. 1996 Jan; 67(1): 60-3; quiz 64-5

Perednia DA. Telemedicine system evaluation and a collaborative model for multi-centered research. J Med Syst. 1995 Jun; 19(3): 287-94

Perednia DA; Grigsby J. Telephones, telemedicine, and a technologically neutral coverage policy. Telemed J. 1998 Summer; 4(2): 145-52

Preston J. Texas telemedicine project : a viability study. Telemed J. 1995 Summer; 1(2): 125-32

Puskin DS; Brink LH; Mintzer CL; Wasem CJ. Joint federal initiative for creating a telemedicine evaluation framework. Telemed J. 1995 Winter; 1(4): 395-9

Puskin DS; Sanders JH. Telemedicine infrastructure development. J Med Syst. 1995 Apr; 19(2): 125-9

Reddy S et al., . Experiences with ARTEMIS, an internet-based telemedicine system. Proc AMIA Annu Fall Symp. 1997: 759-63

Roca OF et al.. Factors influencing distant tele-evaluation in cytology, pathology, conventional radiology and mammography. Anal Cell Pathol. 1996 Jan; 10(1): 13-23

Rovetta A; Falcone F; Sala R; Garavaldi ME. Telehealth in Africa. Stud Health Technol Inform. 1997; 39: 277-85

Schlachta LM; Pursley-Crotteau S. Leveraging technology : telemedicine in disease management and implications for infusion services. Infusion 1997 Nov; 4(2): 36-40

Sheng OR; Hu PJ; Au G, Higa K; Wei CP. Urban teleradiology in Hong Kong. J Telemed Telecare. 1997; 3(2): 71-7

Speedie SM; Stensland J. A model for evaluating the cost impact of telemedicine as compared to face-to-face care. http://www.peds.umn.edu/telemed/ataposter.html

Sisk JE; Sanders JH. A proposed framework for economic evaluation of telemedicine. Telemed J. 1998 Spring; 4(1): 31-7

Siwicki B. Telemedicine providers ponder the profitability issue. Health Data Manag. 1998 Apr; 6(4): 85-6, 88, 90-1

Siwicki B. Measuring the benefits of telemedicine. Health Data Manag. 1997 Nov; 5(11): 86

Siwicki B. Telemedicine. Lower costs could lead to growth spurt. Health Data Manag. 1997 Dec; 5(13): 15

Sommer TJ. Telemedicine : a useful and necessary tool to improving quality of healthcare in the European Union. Comput Methods Programs Biomed. 1995 Sep-Oct; 48 (1-2): 73-7

Tangalos EG. Clinical trials to validate telemedicine. J Med Syst. 1995 Jun; 19(3): 281-5

Taylor P. A survey of research in telemedicine. 2 : telemedicine services. J Telemed Telecare. 1998; 4(2): 63-71

Van-Gennip; Enning CW; Osseyran A. VATAM : validation of telematics applications in medicine. Annu Meet Int Soc Technol Assess Health Care. 1996; 12:11

Vyborny KM. Legal and political issues facing telemedicine. Ann Health Law. 1996 (5): 61-119

Whitten PS; Allen A. Analysis of telemedicine from an organizational perspective. Telemed J. 1995 Fall; 1(3): 215-26

Wootton R. Telemedicine : a cautious welcome. BMJ. 1996 Nov 30; 313(7069): 1375-7

Wright D; Androuchko L. Telemedicine and developing countries. J Telemed Telecare. 1996; 2(2): 63-70

Yellowlees P. How not to develop telemedicine systems. Telemed Today. 1997 May-Jun; 5(3): 6-7, 17

Yellowlees P. Practical evaluation of telemedicine systems in the real world. J Telemed Telecare. 1998; Suppl 1: 56-7

Non-peer reviewed :

HAL Corporation. The Economics of Investing in Health Information Infrastructure. Office of Health and the Information Highway, Health Canada. March 1999.

Hailey D. Assessment of telehealth applications : version 1. Edmonton, Alta. : Alberta Heritage Foundation for Medical Research, c1997

Picot, J. Sector Competitive Frameworks Series. Telehealth Industry, Industry Canada, Health Industries Branch, 1998,09,18.

Shariatmadar A; Angus DE.. A review of economic evaluation literature on health information infrastructure. Office of Health and the Information Highway, Health Canada. April 1999.

U.S. Dept. of Health and Human Services . Exploratory evaluation of rural applications of telemedicine : final report, Feb 1, 1997. Rockville, Md : Office of Rural Health Policy, Health Resources and Services Administration, U.S. Dept. of Health and Human Services, 1997

Appendix II - An overall conceptual model

In addition to review and evaluation of the literature, several other issues of relevance were discussed by the panel. These elements are first, an overall conceptual model for telehealth; second, which telehealth solutions require evaluation; and third, at what stage of development evaluation is required? Each is briefly described below.

An overall conceptual framework

The Panel built on the CIPP model for program evaluation, a concept originally proposed by Stufflebeam. The CIPP model provides a process flow model for evaluation and consists of quadrants of activity, beginning with Context and proceeding through Input to Process and finally to Product. It was recognised from the outset that a simple model that lacked duplication or overlap between concepts or elements of an evaluation activity was not a likely goal. For this reason, it is entirely likely that others would place a proposed evaluation element in a different location, or perhaps even duplicate it in several locations. To minimise concern over this issue, the Panel modified the CIPP model to identify some external influential factors that envelop the evaluation process, and other internal factors that were felt to be potentially common to each quadrant of the CIPP model.

When considering evaluation of telehealth initiatives, additional external factors were felt to have an indirect, sometimes insidious, impact on each of the CIPP elements. For this reason, Policy, Presumptions, and Communication were placed in a ring that encapsulates the entire CIPP process. Policy, or the lack of policy, was felt to be a major factor that could impact a telehealth activity even before concept development, but certainly through evaluation, and into implementation. Similarly, Presumptions held by each of the diverse groups of individuals required to design, implement, evaluate, and manage a telehealth activity can have far reaching adverse effects. Finally, communication can be a strong element in determining the success of any telehealth activity, while lack of communication will, if not scuttle the process, may at least derail it. Developing the correct policy setting, revealing and clarifying presumptions, and ensuring constant free flow of communication amongst and between supporting groups of individuals will enhance the chance of success.

In addition to external elements, many factors were identified that could be viewed as facilitators or inhibitors of telehealth activities. Some were considered to be both a facilitator and an inhibitor either within or between CIPP quadrants.

As a result of these deliberations the panel developed a conceptual evaluation model illustrated in Figure 10 below.

Why consider issues that may seem irrelevant to evaluation of a telehealth activity? Two reasons exist. First, in order for a sound evaluation process to occur, it is essential that factors influencing success or failure of the telehealth initiative are clearly identified. This provides perspective. Consider reading a report that indicates a simple or elegant telehealth solution was evaluated and shown to be technologically simple and clinically superior, yet it was not implemented and therefore classified as a failure. What if the policy environment did not support, or at least permit, telehealth activities (Policy issues), or the system had been designed for physicians when the primary users were nurses (Presumption issues), or the system had been designed for physicians but no discussion with or involvement of the anticipated primary user group took place (Communication issues). Under such circumstances, it can almost be guaranteed the application would be viewed as a 'failure'. Yet transplant this telehealth activity to another more supportive policy environment or correct the presumptions or communication oversights, and the product might well be of great value.

The second reason relates to the evaluation process. Just what is it that may need evaluating? It need not be the technical or clinical aspects of the application. If policy is an issue, perhaps policy implications should become a focus of the overall evaluation process. If a crucial inhibitor is found, then means of correcting, disabling, or removing the issue should be evaluated. Unless the correct features are identified for evaluation, the final result will be limited in its value.

Conversely, there is great value in having a core group of factors evaluated for all telehealth activities. This will begin to build a sound base for comparison and judgement between telehealth options.

Appendix III - Which telehealth solutions require evaluation?

There is consensus that many, perhaps most, recent technological innovations in medicine have been adopted without the requirement to undergo the extreme scrutiny and evaluation that appears to be destined for telehealth. Opinion at a recent G7 / G8 meeting vacillated from 'why evaluate at all' to 'telehealth will be adopted regardless of evaluation'. The crux lies in the growing reality that technology can probably deliver whatever we desire -- a vast array of choices are available. Therefore, we must make intelligent choices to avoid costly errors in judgement. This requires sound evaluation data.

The expert panel supported a concept illustrated in Figure 11 below. As the complexity and predictability of either the input or outcome increases, then so too does the likelihood that evaluation is worthwhile, perhaps essential. Following from this model, three areas within the graph can be identified, each representing the impact of the technology on the current process. ADOPTION - Some technologies are so simple that they are capable of simply being adopted and absorbed into any current process (e.g., adoption of a facsimile machine for transmitting laboratory results to an ICU).

INTEGRATION - Other more complex technologies require integration into existing systems (e.g., integration of a PACS for a radiology department). TRANSFORMATION - Finally, some technologies represent or require such radical change in terms of the role of stakeholders or fundamental approach to delivery of health care that they will transform an existing process (e.g., transformation of an extramural nursing program through introduction of a novel telehomecare application).

Appendix IV - At what stage of telehealth development is evaluation required?

At what stage of telehealth development is evaluation required ?

In general terms, the literature is in agreement that telehealth applications will undergo a natural evolution after they are introduced. Even institutions recognised as early adopters of telehealth will still experience a learning curve following introduction of each new telehealth application into the 'real world' environment. Further, once the principle of the process is established, it is likely that technology enhancements will then impact the process. If the technology impact is sufficiently great, it may be necessary to re-evaluate. In essence this is no different from the 'Continuous Quality Improvement' paradigm adopted by the health care sector in the last decade. Evaluation therefore is not a static, one-off process, but a dynamic continuum, as illustrated in Figure 12 below. Telehealth evaluation would benefit from adoption of telehealth programs to allow a sufficient level of maturity to be reached that meaningful, 'steady-state' / 'real-world' evaluations can be performed.