Bringing Evidence to the Point of Care

Final Report. May 2003

1. Overview

Health care systems around the world are faced with the challenge of improving patient safety and decreasing the risk of adverse events. In health care, we define an adverse event as an unplanned, undesired, harmful occurrence related to the health care process, rather than to an underlying disease [1]. Adverse events can arise from a number of factors, including poor care design, ineffective communication between team members, and the failure to apply relevant evidence of effective practice in a timely and appropriate fashion. In 1999, the Institute of Medicine's report "To Err is Human: Building a Safer Health System" highlighted the risk of adverse events in clinical settings [2]. This report estimated that 44,000 to 98,000 people die in hospitals in the United States each year as a result of an adverse medical event, and that adverse events were among the top ten causes of death in that country, causing more deaths each year than do AIDS, breast cancer or motor vehicle accidents [3]. The morbidity of adverse events is not known, but it is thought to be even greater. In the Canadian health care system, the frequency of adverse events is not known, but extrapolating from the American experience, as many as 10,000 people may die each year from such events. A national study to assess the frequency of adverse events is currently underway.

Adverse events also cause an enormous financial burden to health care systems. The Institute of Medicine reports that adverse events cost the United States approximately $38 billion annually, and approximately $17 billion of this amount is associated with preventable errors. In one study, preventable adverse drug events were responsible for $2.8 million annually in a single 600-bed teaching hospital [4].

One significant source of medical errors is inconsistent access to and application of relevant evidence. Research evidence, generated at an exponential rate, is not readily available to clinicians and, when it is available, it is infrequently applied in clinical practice. Moreover, clinicians are limited by their inability to afford more than a few seconds per patient to find and assimilate relevant evidence. The consequence of these challenges is that high-quality evidence is inconsistently translated into practice, leading to gaps between evidence and practice, significant practice variation and, in some cases, medical error. The evidence from research findings needs to be translated into information that is appropriately formatted for clinicians and patients, and information systems can make this translated information available for real time clinical decision making.

To help meet these challenges, we propose to fill the gap between the knowledge needs of the clinicians and the best evidence available in the clinical literature. To achieve this, we aim to provide wireless mobile computers packaged with appropriate evidence resources to family physicians and general internists, in order to determine if these devices can improve patient care and clinical practice, both in the clinics and at the bedside. In an attempt to achieve this goal, we assessed the information needs at the point of care of hospital and community-based clinicians, and developed and evaluated formats for delivering this information on mobile computers. The initial work on this project led us to understand that clinicians would like additional practice tools available on mobile computers, including an online prescribing tool. Before we complete a randomized control trial of these devices packaged with the various practice tools, it is essential that we complete usability testing of the practice tools, in order to ensure that they are appropriate for busy clinicians.

This research was designed to provide insight into how to deliver mobile information services in a professional environment, and to present solutions for providing decision support information on small, portable or wearable platforms. We have clearly identified a need in the health care community for efficient and effective practice tools on mobile computers. We anticipate the ability to provide mobile devices packaged with relevant high-quality practice tools for clinicians in the near future.

In this report, we describe the work that has been completed in the past twelve months. This work includes the evaluation of relevant literature, the development of a user interface for mobile devices, the development and implementation of a testing protocol, as well as the development of a search engine and preliminary work on the development of an online prescribing tool. In the next phase of our research program, we propose to develop and evaluate an online prescribing tool for mobile computers linked with relevant evidence. We will determine if providing this tool to family physicians and general internists will meet their practice needs and improve patient care.

This project has resulted in the development of a unique trans-disciplinary collaboration that includes University of Toronto colleagues from: the Department of Medicine (Dr. Sharon Straus, Principal Investigator); The Bell University Laboratory Collaborative Effectiveness Lab (Professor Mark Chignell); the Knowledge Management Lab (Dr. John Mylopolous); the Department of Computer Science (Dr. Graeme Hirst); the Department of Health Policy Management and Evaluation (Dr. Andre Kushniruk); and the Faculty of Law (Richard Owens).

2. Analysis of Progress against Objectives

This progress report describes the results from the third phase of a five-phase project, as well as the proposed methods for completion of phase 4. Phase 1 was funded by a grant from the Health Evidence Application and Linkage Network (HEALNet) (see Progress Report, April 2002). Briefly, 275 general internists and 275 family physicians residing in Ontario, and 120 medical house staff from the Internal Medicine program at the University of Toronto were surveyed in order to determine their familiarity with computers, what evidence resources they would like to have available to them, and how they would like this information presented. This information was used to develop sample materials for evaluation on mobile computers.

Phase 2 of the study involved formatting the evidence resources for use on mobile computers, and developing an effective user interface for the devices. We proposed to apply the knowledge learned from cognitive studies of human-computer interactions in designing this interface, and to use an extensive iterative process to format evidence resources for mobile computers.

During Phase 3 of the project, a small pilot study was performed in order to determine the impact on patient care. It involved the evaluation of the use of high-quality evidence resources on mobile computers for family physicians and general internists. This phase included additional usability testing of these devices by and on university affiliated clinicians in both rural and urban settings. Feedback from these evaluations led us to consider developing additional clinical practice tools for mobile computers, including an online prescribing tool. This tool will be integrated with the high-quality evidence resources and a drug reference database, as well as with clinical data from the electronic health record. In the next phase, we will use an explicit iterative process to develop and evaluate this tool. A working group has also been created to work on a search engine that can be used in the upcoming pilot study, scheduled for August 2003. All of this information will be used to develop a full-scale randomized trial of the use of mobile computers and of the impact of bringing evidence to the point of care on relevant clinical outcomes. Phases 2 and 3 of this study were funded by Health Canada and by the Bell University Laboratories.

In addition, we proposed to establish linkages with the Royal College of Physicians and Surgeons of Canada and with the Canadian College of Family Physicians; both professional societies have requirements that clinicians must meet in order to ensure ongoing clinical competence, and we are discussing the possibility of providing continuing medical education (CME) credits for clinicians who access the various high-quality evidence resources that we will make available. We want to provide CME credits for each search that clinicians conduct with the resources.

Our research team includes experts from the Department of Computer Science who are working on developing an effective and efficient search engine, and our budget for this study includes salary support for two graduate students from this Department. We have requested support for graduate students from the Department of Mechanical and Industrial Engineering who provide expertise on human-computer interaction and assist in the development of the user interface, as well as with the evaluation of the prototypes. The research team also includes investigators with expertise in educational computing (Dr. L. Spero) and knowledge translation (Dr. D. Davis, Dr. S. Straus). Dr. Straus has expertise in evidence-based medicine (EBM), research methodology, and medical informatics; she also coordinates the study.

Manuscripts describing the study and its results have been prepared with input from the entire research team. The team has presented the results of this project at annual meetings of the American Medical Informatics Association, the Society of General Internal Medicine, the Cochrane Collaboration, and the Human Factors and Ergonomics Society, where it won 'Best Paper' in 2002. And, recently, one of the residents (Dr. Richa Mittal) working on an aspect of this project was awarded First Prize in the Department of Family and Community Medicine Research Competition at the University of Toronto.

Professor Chignell and Dr. Straus have recently applied for funding from the Natural Sciences and Engineering Research Council (NSERC) CRD competition. Because of the unique collaborations that have developed, the EPOCare group has been able to explore other research opportunities. For example, some members of this research collaborative participated in the development of a research program led by Dr. Straus, entitled "Innovations in Patient Safety and Knowledge Translation", which was recently funded for 5 years by the Canadian Institutes of Health Research (CIHR).

3. Background and Rationale

As was described earlier, one significant source of medical errors is inconsistent access to and application of relevant evidence. Research evidence, generated at an exponential rate, is not readily available to clinicians; when it is available, it is infrequently applied in clinical practice. Moreover, clinicians are limited by their inability to afford more than a few seconds per patient to find and assimilate relevant evidence. The consequence of these challenges is that high-quality evidence is inconsistently translated into practice, leading to gaps between evidence and practice, significant practice variation and, in some cases, medical error.

In previous studies we noted that house staff needed access to information within 30 seconds; if the search required more time, it was abandoned. While making evidence available to the house staff increased the extent to which it was sought and incorporated into patient care decisions, the system was too slow for regular use [5, 6]. With the developments that have occurred in information technology since this study was completed, we are attempting to narrow the gap between the knowledge needs of clinicians and the best evidence available in the clinical literature, by providing these evidence-based medicine resources at the point of care, using mobile devices. There are a number of challenges in providing these resources on mobile devices. Firstly, the screen size is small, which limits the amount of text that can be seen, and makes table layout difficult. Secondly, the devices have limited input methods and memory. Thirdly, clinicians must be able to access the information on these devices quickly. Consequently, the provision of clinical evidence at the point of care poses a challenging problem, particularly with regard to user interface design.

In an attempt to meet this challenge and bring evidence to the point of care using mobile computers, in the earlier phases of this project we conducted a series of usability testing sessions of mobile computers packaged with high quality evidence. Family physicians and general internists were asked to complete a series of tasks using these devices. Based on the analysis of these sessions, it was determined that clinicians wanted access to a drug reference database at the point of care and to an online prescribing tool that integrated the relevant evidence with the order entry system.

Four studies have evaluated computerized physician order entry systems [7-10]. One of these studies was a randomized trial [7], another was a cross sectional study [8], and the final two were interrupted time series studies [9, 10]. These studies showed a reduction in medication errors, but no impact on the clinically important outcome, i.e., adverse drug events. The lack of observed effect is likely due to the small sample size of these studies. The studies exclusively involved internally developed systems at two university-affiliated institutions. Dissemination of internally developed systems has been impaired by the features of these systems, and by the challenge of linking them with information technology systems at other institutions. This highlights the fact that different clinical settings have different facilitators and barriers to patient safety, which must be considered before widespread implementation is achieved.

There is a lack of rigorous research on the impact of a mobile computer-based online prescribing tool. Moreover, little is known about the impact of order entry systems on patient care in the primary care setting versus the secondary care setting, or in university-affiliated versus non-university-affiliated practice settings. In the next phase of this project, we propose to develop and evaluate an online prescribing tool that is linked with relevant evidence for use on mobile computers. Provision of this tool to family physicians and general internists could meet their information needs and improve patient care.

4. Objectives

The objectives of the Bringing Evidence to the Point of Care (EPOCare) project were originally stated as:

1. To format chapters of the resources entitled Evidence-Based On Call (EBOC) and Clinical Evidence as well as clinical practice guidelines selected by the Guideline Action Committee (GAC) / Ontario Program for Optimal Therapeutics (OPOT) for use on the mobile devices, using the information collected during Phase 1 of the project.
2. To analyze data from the focus groups which comprised medicine residents, general internists, and family physicians, and to use the results to determine what changes need to be made to the format of the evidence materials.
3. To analyze data from the individual usability testing sessions evaluating the prototypes, and to make necessary modifications to them.
4. To prepare a pilot test of the mobile devices packaged with the high-quality evidence resources for a sample of general internal medicine residents, for a one-month period.
5. To further refine the format of the materials, depending on feedback from the pilot study and from an additional round of usability testing.

In response to what we have learned from the early phases of this project, we have added additional objectives, including:

1. To develop and evaluate an online prescribing tool that can be integrated with high-quality evidence resources, a drug reference database, and the electronic health record.
2. To complete a pilot test of these devices on an inpatient medicine unit, in order to determine the impact on the process of clinical care.

5. Methodology

In Phase I we conducted surveys of over 500 clinicians, including family physicians, general internists, and medical house staff, in order to determine their information needs. A sample of clinicians who completed the survey was invited to participate in a focus group, where their needs were explored in more detail. Research colleagues from the Departments of Computer Science and Mechanical and Industrial Engineering joined clinicians from family medicine and general internal medicine on clinical rounds, in order to evaluate their knowledge needs. The results of this phase were used to develop user interfaces for the mobile devices.

Several chapters of Evidence-Based On Call (EBOC) and Clinical Evidence, as well as guidelines selected by the Ministry of Health and Long-Term Care's Guidelines Advisory Committee were reformatted for use on the mobile devices, using the information collected during Phase 1 of the project (see Progress Report, April 2002 for more details). We were given access to the XML files for these resources, for development and evaluative purposes. With input from experts on human-computer interaction, a user interface for the mobile devices was designed. A cognitive evaluation of health care systems and user interfaces was done using Kushniruk et al.'s (1997) methodological approach.

During Phase 2 we developed and evaluated user interfaces for these evidence resources, and clinicians (medicine residents, general internists, and family medicine residents) were asked to complete a series of tasks relevant to their clinical practice. This phase of testing was completed using a HyperText Markup Language (HTML) - based prototype with the look and feel of a finished interface, showing visuals as well as placement of controls. It also contained realistic navigational elements, allowing the designers to assess the layout of information and navigation. However, the prototype provided limited interaction and users could not enter a live search, since the interface was not connected to a back-end search engine. The responses of the clinicians as they completed the tasks were audio recorded, and the computer screen was video recorded. The transcripts from both the audio and video recordings were transcribed, coded, and analyzed independently by two investigators. Feedback from these sessions was used to revise the prototypes and to complete additional rounds of usability testing. This latest round of testing included clinicians from rural (Sault Ste Marie) and urban settings. Actual devices were used, including iPAQs and Xybernauts, instead of the emulator which was used in the initial phase of testing. Again, clinicians were provided with a series of tasks relevant to their clinical practice, and their reactions and comments were recorded. These recordings are currently being transcribed. The final round of urban testing has been delayed because of the SARS outbreak, but it is anticipated that it will be completed by the end of the summer of 2003.

In March 2003, mobile computers packaged with the evidence resources and a drug reference database were provided to medical house staff on an inpatient general medicine service. Their feedback on the devices was elicited in order to further refine the resources and to prepare for another pilot trial in August 2003.

In order to have software suitable for a field trial, the prototype will have to be scaled up to a fully interactive version that can address a wide range of queries about clinical evidence, not just the small number of scenarios that were the focus of the initial user testing. The prototype that has been used in the usability testing was a static mockup. Our intention was, and is, to follow user testing with the development of a more fully functioned interactive prototype that can be used in a field trial. It is intended that the first version of the interactive prototype being developed by the EPOCare team will eventually be followed by a more sophisticated version that will use innovative knowledge management tools being developed by Professor John Mylopoulos and his research team in the Department of Computer Science at the University of Toronto.

In early 2003, a subgroup of EPOCare researchers was formed to work on building the first version of the interactive prototype. This subgroup included Natalia Modjeska, Peter Wong, and Eric Tursman, with assistance from Natasha Martin, Rick Bodner, Anna Malandrino, and Mark Chignell. The work began with the consideration of a number of architectural alternatives. The group focused on the 'Clinical Evidence' database first, with the expectation that once a system would be working with that source, other sources, such as 'Evidence-Based On Call' (EBOC) would be tackled. The 'Clinical Evidence' and EBOC sources were both available in the form of XML files. After detailed analysis, a solution path was chosen, involving an off-the-shelf search engine (MG, or Managing Gigabytes, developed by the New Zealand Digital Library Project) and an extraction process that can retrieve specific XML entities of relevance to particular queries, to particular types of queries. An initial modeling process applied a number of scenarios to the XML version of 'Clinical Evidence'. XML tags / entities, where relevant types of answers were to be found, were identified. The XML document was then broken up into a number of files representing different types of questions that might be asked. A simple wireless interactive prototype was constructed, using MG as the search engine and GSDL as a wrapper. This first interactive version of the prototype is now working. Users can input a keyword query to this prototype and then review the answer returned by MG. The returned documents are hyperlinked and browsing is allowed. The first interactive prototype was created in April 2003.

We are currently - as of early May 2003 - enhancing this initial interactive prototype by building a browsing interface for the search results, where the results are presented as choices and hyperlinks. The logic of this approach is to bring the user to the neighborhood where the answer to her question is likely to be found, and then let the user choose what information is actually relevant or contains the desired answer.

At this time, the browsing functionality is available, but we need to determine what sections of the database need to be hyperlinked to each other. We are working on a query expansion module that uses medical vocabularies from the Unified Medical Language System (UMLS) Thesaurus developed by the US National Library of Medicine (NLM). This query expansion is an important part of the required functionality, because the terminology used in the database might be different from the search terms in the user's query (e.g., CE contains 'domicile oxygen', while the user might be looking for 'home oxygen').

6. Results

Analyzing transcripts from the usability sessions completed to date has produced very interesting results. Clinicians liked having instant access to the high-quality evidence on the mobile devices, and felt it would be useful in their clinical practice. All participants expressed interest in having drug information (including drug dosage, adverse events, and costs) available on a mobile device, and wanted to know the validity of the evidence. The need to be able to access relevant information easily and immediately was a common concern amongst all three user groups.

General internists had twice as many positive comments as negative ones regarding statistical concepts, whereas family physicians had equal numbers of positive and negative comments (p=0.002). General internists reported satisfaction with the amount of detail provided in the evidence resources, whereas family physicians felt they contained too much detail about evidence and statistics (p=0.04). Family physicians were satisfied with the material if they felt the source of data was trustworthy (e.g., guidelines issued by the Canadian Medical Association). Neither group was interested in accessing full-text articles on the mobile devices, although they wanted a way to save relevant citations for later use.

Half of the participants (57%) owned a Personal Digital Assistant (PDA), and in all cases but one it was a Palm Pilot. None of the family physicians owned a PDA. PDA owners made more polar comments (both positive and negative) about user interface design, including suggestions for improvement, than those who did not own such a device (p=0.007).

Older clinicians (aged > 40 years) had mixed reactions to EBM content, whereas younger clinicians made almost twice as many positive comments as negative ones (p=0.03). Older clinicians were less familiar with EBM terminology, and were not as interested as the younger group in viewing that information. Older clinicians also lacked the same familiarity with evidence resources.

A tabular format was the preferred representation for displaying numerical data, even when a custom format graph was available as an alternative. For text, participants preferred bullet points, highlights of key words, and colour-coded sections that increased the participants' ability to scan the content more easily. White space was found to be an important visual aid in locating chunks of information.

The preference for searching or browsing through a list of topics seemed to depend on the nature of the question (e.g., How easy is it to map my question to the list of given categories?) and the task at hand (Do I have a few minutes to read up on the latest evidence about meningitis before I see my patient?)

All participants expressed interest in having drug related information (e.g., generic and brand names; dosage; coverage by the Ontario Drug Benefits program; adverse effects; indications / contraindications; and interaction with other medications) available on a mobile device. They indicated a preference for having this information integrated with the evidence provided in Clinical Evidence and Evidence-Based On Call. The request may stem, in part, from residents and internists who already have pharmacopoeias loaded on their PDAs. Family physicians who did not own a PDA explained that they consult drug textbooks frequently in their daily practice, but would prefer to use a wireless mobile device in order to get the latest drug information, which changes frequently. Clinicians were also interested in receiving CME credits for any searches that they would complete using these devices.

The feedback from the participants was also used to develop an educational session to introduce EBM, the use of evidence resources, and the use of the mobile devices. This program will be further evaluated during the next pilot study, to be completed in August 2003.

7. Discussion

We have not identified any other rigorous research project that aims to provide clinicians with high-quality evidence resources and other practice tools on mobile computers. Although the user-centered design process for commercial software development has been practiced in industry for many years, clinicians have not typically been involved in the up-front design process for medical applications used in clinical settings. Our project has been a collaborative effort in which end-user clinicians have been involved in the design process, from the user requirements to the prototype generation stage.

Although comments and observations from 40 clinicians cannot be generalized to a larger population, the consistent findings from this study can be used to direct the next phase of the project. For example, in response to participant feedback, we are attempting to develop and evaluate an online prescribing tool that is integrated with the evidence resources. Ultimately, we hope to be able to integrate the evidence resources and online prescribing tool with clinical data from the electronic health record. This goal raises issues about security of clinical information, which our colleagues from Computer Science and Law are helping us tackle. We also anticipate developing the high-quality patient information resources that both patients and clinicians have requested. We will also explore establishing links between hospitals, individual clinicians, and pharmacies to allow clinicians to have access to patients' current medication lists. Similarly, links could be made between pharmacies, clinicians, and the best current evidence, so as to promote use of this evidence. Links could also be made to validated decision aids, in order to promote patient-centered decision making.

Data from this program of research will be used to prepare for a full-scale randomized trial of the impact of mobile computers packaged with practice tools on relevant clinical outcomes. This study will be designed as an efficacy study, and it is important that we optimize the resources and devices before it is completed. The ultimate goal of this program is to improve patient outcomes within our health care system by providing fast, easily accessible evidence-based information to front line clinicians at the bedside or in clinics. It is hoped that by providing the latest research evidence that may aid in diagnosis and therapy choice when needed, clinicians' prescribing practices and patient safety will improve.

This project is unique in that it has resulted in a trans-disciplinary collaboration that includes colleagues from the Departments of Computer Science, Mechanical and Industrial Engineering, Cognitive Psychology, and Medicine. This project has produced extensive cross-fertilization leading to the exploration of additional research projects and funding opportunities, such as the recently funded research program entitled "Innovations in Patient Safety and Knowledge Translation". We also hope to establish a knowledge management centre with personnel experienced in the iterative process necessary for the design, evaluation, and implementation of a variety of electronic knowledge resources, so that this expertise can be extended to other health care professions. Evidence resources would be customized according to the individual health care professional's unique needs. And it is hoped that a similar evaluative process developed by our research team could be used to provide high-quality resources for patients.

8. Progress / Activities from May 2002 - April 2003

• We enhanced a low-fidelity, HTML-based prototype with sample content for display on wired mobile devices (specifically Xybernaut and iPAQs).
• A second round of testing using these mobile devices was completed with groups of family physicians and general internists in Sault Ste Marie.
• Additional field-testing was started in March 2003 with rural and urban doctors. The urban doctors' field-testing is expected to be completed by end of summer of 2003.
• We continue to update a basic web site for the public to view. The web site (Http://www.cebm.utoronto.ca/projects/) is a summary of the project and has generated much interest, and we are maintaining a list of contacts and potential research collaborators. We have also created an internal web site where we post our reports and progress.
• We have presented the results of this project at a variety of research meetings around the world, and these presentations have also generated interest in the project and the possibility of international, multi-centre research partnerships.
• We have applied for additional funding for the proposed program of research from a variety of funding agencies, including the Natural Sciences and Engineering Research Council (NSERC).

9. Technology / Knowledge Transfer Activities

There has been significant progress made on this project, which has been underway since July 2001. The project is constantly attracting more students and researchers wanting to join the team, and our web site (Http://www.cebm.utoronto.ca/projects/) has regular activity and inquiries. The project team (biographies are located in Appendix A) has been very active not only in presenting the work that we have done to date, but also in submitting articles for publication. Listed below are the presentations and publications produced to date.

Publications

Peer-reviewed Journal Articles

• Takeshita H, Davis D, Straus SE. Clinical Evidence at the Point of Care in Acute Medicine: A Handheld Usability Case Study. Human Factors and Ergonomics Society (HFES), April 2002. Awarded 'Best Paper'.
• Danicic R, Straus SE, Chignell M, Takeshita H, Modjeska N, Kushniruk A. Quantitative Content Analysis of Evidence-Based Medical Information Retrieval System. Submitted to the American Medical Informatics Association (AMIA), 2003.

Refereed Conference Proceedings

• Straus SE, Evans M, Goel V, Davis D. Bringing Evidence to the Point of Care: What do Clinicians Want. (Abstract) Biomed Central Meeting Abstracts: 4th International Cochrane Colloquium, 2001; 1: op 047.
• Straus SE et al. (2002). Bringing Evidence to the Point of Care: Survey of Clinicians' Needs. (Abstract). JGIM, 2002.
• Lenga I, Straus SE. Information Needs of Medical housestaff. (Abstract). JGIM, 2002.
• Danicic R, Chignell M, Takeshita H, Modjeska N, Kushniruk A, Straus SE. Effective Delivery of Evidence at the Point of Care. JGIM, 2003.
• Nui Yun, McArthur G, Hirst G. Answering Clinical Questions. (Accepted). Annual Meeting of the Association for Computational Linguistics, 2003.

Other (includes articles, reports, monographs, special papers, review articles, government publications, etc.)

• Yun Niu's Ph.D. thesis for the Department of Computer Science is underway, and is based on the development of a query-answer system for the EPOCare project.
• Harumi Takeshita's M.A.Sc. thesis was completed in 2003, and was based on this project. It is entitled "Usability Evaluation for Handheld Devices Presenting Clinical Evidence at the Point of Care".
• Dianne Davis' Master's report was submitted to the Mechanical and Industrial Engineering (MIE) department at the University of Toronto in 2003, and was based on this project.
• Michelle Graham's undergraduate thesis was submitted to the MIE department at the University of Toronto in 2002, and was presented at the Special Interest Group for Design of Communication (SIGDOC) Conference in 2002.
• Danielle Lottridge's undergraduate thesis for the MIE department was based on this project, and is a study of the beliefs of the public about their relationships with their physicians and the potential impact of mobile computing on the physician-patient interaction.
• Natasha Martin is working on her M.Sc. thesis for the MIE department; her thesis is based on this project.
• Eric Tursman is working on his M.Sc. thesis, which will focus on the development of a voice recognition system to allow searches to be triggered by voice commands.
• Feature articles on this project were also written for magazines, including Physicians' Chronicle, Wireless Computing, and the University of Toronto Alumni Magazine.

Presentations (includes lectures, workshops, and other presentations)

• Straus SE. Evidence Dissemination and Implementation. Medical Grand Rounds, McMaster University, June 2001.
• Straus SE. Evidence Implementation. (Poster presentation). Annual Meeting of Teachers of Evidence-Based Medicine (EBM), Palermo, September 2001.
• Straus SE, Takeshita H, Davis D, Rodrigues-Gianolli P. Bringing Evidence to the Point of Care. Bell University Labs Mobile Computing Seminars, University of Toronto, October 2001.
• Straus SE. Bringing Evidence to the Point of Care: What Do Clinicians Want? (Podium presentation). 4th International Cochrane Colloquium, France, October 2001.
• Takeshita H. EPOCare. Presentation for Human Factors students at University of Waterloo, November 2001.
• Straus SE, Takeshita H, Davis D, Rodrigues-Gianolli P. Bell University Labs Research Labs and Projects, University of Toronto, November 2001.
• Rodrigues-Gianolli P, McArthur G. ToX: Toronto XML Engine, CASCON 2001, November 5 - 7, 2001.
• Straus SE. Knowledge to Practice. Department of Psychiatry Continuing Education Rounds, University of Toronto, December 2001.
• Takeshita H, Chignell M. Bringing Evidence to the Point of Care. Iwate University (Japan) delegation, March 2002.
• Takeshita, H. Bringing Evidence to the Point of Care. Open House for High School Students, University of Toronto, March 2002.
• Straus SE. Knowledge to Practice. National Health Service (NHS) Clinical Librarians' Conference, UK, March 2002.
• Straus SE. EBM in Practice. EBM Workshop, School of Population Health, University of Queensland, April 2002.
• Straus SE. Current Issues in EBM. Queensland Health Authority, April 2002.
• Straus SE. Teaching EBM. Clinical Quality Improvement Forum, Queensland Health Authority, Royal Brisbane Hospital, April 2002.
• Lenga I, Straus SE. Information Needs of Medical Housestaff. (Poster presentation). Annual Meeting of the Society of General Internal Medicine (SGIM), Atlanta, May 2002.
• Straus SE, Evans M, Davis D. Bringing Evidence to the Point of Care: Survey of Clinicians' Needs. (Poster presentation). Annual Meeting of the SGIM, Atlanta, May 2002.
• Takeshita H. Evidence at the Point of Care: Usability Testing. SIGDOC, 2002.
• Takeshita H, Kushniruk A, Straus SE. Usability Study of Evidence at the Point of Care Using Mobile Devices. AMIA, 2002.
• Takeshita H, Davis D, Straus SE. Clinical Evidence at the Point of Care in Acute Medicine: A Handheld Usability Case Study. HFES, 2002.
• Danicic R, Chignell M, Takeshita H, Modjeska N, Kushniruk A, Straus SE. Effective Delivery of Evidence at the Point of Care. Annual Meeting of the SGlM.
• Straus SE, Takeshita H. Update on the EPOCare Project. Bell University Lab Seminar Series, October 2002.
• Straus SE. Bringing Evidence to the Point of Care. UK Consortium of Librarians, 2002.
• Straus SE. Knowledge Translation and the Future of EBM. University of Queensland, May 2003.

10. Project Status (completed / outstanding work, etc.)

We are currently completing another round of usability testing, and we anticipate that this will be completed by the end of summer of 2003. We are also beginning to work on developing and evaluating an online prescribing tool; the proposed methodology and timeline are outlined below.

0 to 6 months

We will systematically search the Internet and MEDLINE to identify drug reference databases that could be used on a mobile computer. We will also contact two hospital pharmacists and two clinicians, in order to identify additional relevant databases. Access to each of these databases will be obtained from their publishers, and each will be evaluated independently by two clinicians using criteria to assess their validity and clinical usefulness [see Appendix C]. Based on this evaluation, the database that is rated most highly by both clinicians will be selected for use in this study.

Material from the selected drug reference database will be formatted for the mobile computer, using the results from the usability testing completed during the earlier phases of the EPOCare project. For example, during the usability testing sessions, clinicians suggested that material about adverse events, drug interactions, and dosage be placed in a prominent position and that it be easily navigable, with bottom line summaries provided. Linkages will be provided to more detailed, high-quality evidence from resources such as Clinical Evidence and Evidence-Based On Call. These latter resources have been formatted for use on mobile computers as part of the EPOCare project.

Along with experts in human-computer interaction we will develop a user interface for an online prescribing tool that allows clinicians to complete a prescription, and that will provide links to the relevant evidence, as outlined above. It will also provide templates for the forms that must be completed when prescribing a drug that is classified as a 'Limited Use' agent by the Ontario Drug Benefits Formulary.

The prototypes for the online prescribing tool and drug reference database will be presented on mobile computers to ten family physicians, ten general internists, and ten medicine residents during usability testing sessions lasting approximately one hour. The clinicians will be randomly selected from lists of practicing clinicians in Ontario, and will include university affiliated and non-university affiliated clinicians. Physicians practicing in rural or urban settings will be included. Participants will be asked to complete a series of tasks chosen as representative of real uses of the technology under study, and each task will begin with a relevant clinical scenario. Clinicians will be asked to respond to a series of questions, and to 'think aloud' as they use the devices. The testing sessions will be audio recorded, and the computer screen will be video recorded. Participants' ability to complete the tasks and the time taken to complete them will be recorded.

In parallel to this, experts in computational linguistics will develop a search engine in order to allow efficient searching of the drug reference database and the evidence resources. There are several challenges to this task, including variable medical terminology and clinicians' need for quick access to relevant information.

6 to 9 months

The tapes from the second series of usability sessions will be transcribed, and the contents analyzed. A coding scheme will be developed in order to complete a quantitative content analysis of the transcripts. Two team members will independently code the transcripts, and inter-coder reliability will be assessed. Any disagreements on the coding will be resolved by consensus. Once coding is completed, a PERL computer program will be used to extract all codes embedded in the transcript, and a statistical analysis will be completed.

The results from this analysis will be used to modify the user interface for the online prescribing tool and the drug reference evidence. Revised versions of the prototypes will be tested by thirty family physicians, general internists, and medical residents who have not participated in the previous testing session. Participants will be asked to complete a series of tasks relevant to their clinical practice, and to respond to a series of questions. Their comments will be audio recorded, and the computer screen will be video recorded.

9 to 12 months

Tapes from the usability sessions will be transcribed and the contents analyzed, as described above. The results from this analysis will be used to modify the user interface. Based on clinicians' different needs, it is anticipated that two versions of the prescribing tool will be developed, including a prescribing tool linked with material from a drug reference database, and a prescribing tool linked with material from a drug reference database and from high-quality evidence resources.

12 to 24 months

Preparations for a pilot study to evaluate the online prescribing tool will occur. This phase will include development of several features that clinicians have expressed interest in having available to them, including patient information sheets, pharmacy links, patient decision aids, and continuing medical education credits for searching for drug information and drug alerts, as well as integration with clinical data from the electronic health record. Integrating evidence and an online prescribing tool with relevant clinical data from an electronic health record will require an extensive period of development, and will include colleagues from a number of disciplines such as data security, human-computer interaction, law, cognitive psychology, and medicine, amongst others.

Appendix A: Introduction to Students and Researchers & Co-Investigators

Co-Investigators

All of the principal investigators involved with this project are faculty members in undergraduate, graduate and postgraduate programs.

Vivek Goel is Vice Provost, University of Toronto

Dave Davis is Associate Dean, Continuing Education, University of Toronto

Researchers & Staff

Sharon Straus is a general internist / geriatrician in the Department of Medicine at the University of Toronto and is supported by the Knowledge Translation Program at the University of Toronto and by a Career Scientist Award from the Ontario Ministry of Health and Long-term Care. She is the Principal Investigator of the EPOCare project. Her research interests include knowledge translation, the precision and accuracy of diagnostic tests and evidence-based medicine. She recently received funding from the CIHR to develop a research program in patient safety and knowledge translation and was recently given a Premier's Research Award.

Andre Kushniruk is an Associate Professor in Information Technology (York University) and in the Department of Health Policy, Management and Evaluation at the University of Toronto. He has expertise in areas of health informatics, evaluation and human-computer interaction in health care. He has written numerous papers on topics related to improving the usability of health care information systems and has been a key researcher on a variety of collaborative projects in the areas of medical informatics and usability engineering. His work includes the development of novel methods for conducting the analysis of computer users and he is currently extending this research to the remote study of system usability, particularly in the study of evidence-based Web resources. He has experience consulting on a variety of large projects with hospitals throughout Canada and the United States in developing health care information systems. Dr. Kushniruk has taught at a number of universities in Canada. He holds undergraduate degrees in Psychology and Biology, as well as a M.Sc. in Computer Science and a Ph.D. in Cognitive Psychology. For the past ten years he has worked on health informatics related projects with a number of universities including the University of Toronto, McGill University, Columbia University, the University of Victoria and Mt. Sinai Medical School in New York.

He is currently working in collaboration with the University of Toronto on a BUL project assessing the use of evidence-based resources deployed on hand-held devices. His role in the project is in the area of providing assistance in aspects of usability testing, and in particular cognitive analysis of data (both video and audio recordings) from users' interaction with the evidence-based resources provided on various types of hand held devices. His interest is in determining barriers to use and understanding of information provided and providing iterative input into development of intelligent retrieval systems (that are being developed as part of the project) in order to improve uptake of evidence at point of care.

John Mylopoulos (B.Eng in Electrical Engineering Brown University in 1966, PhD Princeton University in 1970). John Mylopoulos joined the University of Toronto in 1970. He is the recipient of the first ever Outstanding Services Award given by the Canadian AI Society (CSCSI), a co-recipient of the most-influential paper award of the 1994 International Conference on Software Engineering, a fellow of the American Association for AI (AAAI) and the president of the VLDB Endowment Board. His research interests include conceptual modeling and its applications to databases, software engineering and knowledge management systems. He is currently leading research projects on repositories, data integration and software reengineering in collaboration with IBM's Centre for Advanced Studies and Mitel.

Mark Chignell is a Professor of Mechanical and Industrial Engineering at the University of Toronto. He is an associate director of the Knowledge Media Design Institute at the University of Toronto, director of the Collaborative Effectiveness Laboratory (one of the Bell University Laboratories at the University of Toronto), and director of the Interactive Media Laboratory. He has a number of research interests that are ultimately aimed at augmenting human intellect through user-centered design of innovation collaboration and communication applications. He is particularly interested in individual differences and their impact on technology acceptance and use.

Dr. Lawrence Spero is a Professor of Pharmacology, Director of the Faculty of Medicine Division of Educational Computing and Manager of the Bell University Health Communications Laboratory. He was a founder member of the Health Care Interactive Videodisk Consortium (HCIVC) sponsored by IBM. HCIVC was a group of 17 medical schools, which collaborated to produce and evaluate approximately 40 interactive videodisks. He has been involved in the production of interactive video programs since 1987 and was an IBM Consulting Scholar from 1990-1995. He is also director of project Daedalus which is creating user-friendly tools for the creation of accredited, professional continuing education Interactive Digital Video programs. He has created over 20 interactive programmes and has over 200 papers in pharmacology and approximately 20 in the area of medical education.

He is responsible for the evaluation and implementation of new educational technologies in the faculty of medicine. He has also been heavily involved in faculty development, initiating new policies for the faculty with regard to IT and coordinating electronic resources with the hospitals and the Gerstein Medical Library. He is currently a member of the University Taskforce on Intellectual Property and New Media.

His research interests include how medical students learn spatial anatomy and whether this can be added by 3D computer models, streaming video delivery and management and webcasting in support of education in a healthcare setting and the delivery of healthcare information to the point of care using mobile computing devices.

Contribution to the project: Provide resources for web development and database management. Consult on existing medical content for mobile computers. Experience in IT design and evaluation.

Graeme Hirst is a Professor in the Department of Computer Science at the University of Toronto. His provides expertise in computational linguistics and he is working on this project to develop an efficient and effective query and answer system. Graeme Hirst received a PhD in Computer Science from Brown University in 1983, and has worked at the University of Toronto ever since.

Professor Hirst's research has covered a broad but integrated range of topics in computational linguistics, natural language understanding, and related areas of cognitive science. These include the resolution of ambiguity in language understanding; psychological reality in natural language systems; the preservation of author's style in machine translation; recovering from misunderstanding and non-understanding in human-computer communication; and linguistic constraints on knowledge-representation systems. His present research includes the problem of near-synonymy in lexical choice in language generation; computer assistance for collaborative writing; and applications of lexical chaining as an indicator of semantic distance in texts. A recent spinoff of this research is an intelligent spelling checker. From 1994 to 1997, Professor Hirst was a member of the Waterloo-Toronto HealthDoc project, which aimed to build intelligent systems for the creation and customization of health-care documents.

Professor Hirst was the founding editor of Canadian Artificial Intelligence, and is on the editorial boards of Machine Translation and Computational Linguistics, having been book review editor of the latter for more than a decade. He has written or co-authored over 60 research papers, and is the author of two monographs: Anaphora in Natural Language Understanding (Springer-Verlag, 1981) and Semantic Interpretation and the Resolution of Ambiguity (Cambridge University Press, 1987). He is the recipient of two awards for excellence in teaching, and a best-paper award at the AAAI-84 conference. He has supervised more than 35 theses and dissertations, four of which have been published as books.

Anna Malandrino works with Professor Chignell in the Interactive Media Lab in conjunction with her work at the Collaborative Effectiveness Lab of Bell University Laboratories (which is funded by Bell Canada) in the capacity of Research Project Manager. She coordinates the Bell University research projects, which collaborates with a number of other departments. At the moment there are 5 departments involved: Mechanical and Industrial Engineering, Department of Medicine, Department of Psychology, Computer and Information Studies and the Department of Sociology. She also helps to oversee the CE lab activities. Anna is working on the "Bringing Evidence to the Point of Care" project in the capacity of project manager and is overseeing the budget and assisting Dr. Straus with the overall scheduling and milestones of the project.

Peter Wong is currently involved in the Bell University Lab (BUL) "Bringing Evidence to the Point of Care" project. Peter is responsible for researching on the different types of browsers that can be used for the mobile devices. The main investigation is to determine the capabilities and restrictions of the browsers so that we can display the information effectively on the mobile computers. Peter is assisting the User Interface (UI) team to implement some mock-ups for the focus groups that are being carried out and is doing the programming for the development of the search engine.

Students

Patricia Rodrigues Gianolli has an M.Eng from the University of Toronto and is the Manager for the Knowledge Management Lab in the Department of Computer Science at the University of Toronto. Along with John Mylopoulos and Greg McArthur, she is working on the development of a prototype knowledge management system that supports clinical decision-making. Her research interests are: data integration, management of semi structured data represented by XML documents, and the representation and retrieval of knowledge in knowledge-based systems.

Gregory McArthur has a B.Sc. and an M.Sc from the University of Toronto and is a research associate in the Knowledge Management Lab in the Department of Computer Science at the University of Toronto. Along with Patricia Rodrigues Gianolli, he is working on the development of an effective and efficient search engine. His research interests are: Representing, managing and reasoning about knowledge, the use of ontology's, conceptual models and semantic models in knowledge management systems.

Harumi Takeshita completed her M.A.Sc. in the Interactive Media Lab, one of three Human Factors research labs in the Mechanical and Industrial Engineering Department at the University of Toronto. She returned to academic life after four years of industry experience at designing multimedia software user interfaces for Nortel Networks. Her current research interest is in User Interface (UI) design of mobile devices (also the working title of her thesis), and is contributing to BUL's "Bringing Evidence to the Point Of Care" project.

Natalia N. Modjeska holds B.A. and M.A. degrees in General Linguistics from Umeå University, Sweden. She is currently pursuing a Ph.D. degree in Computational Linguistics at the University of Edinburgh, Scotland. Since September 2001, she has been visiting the Computational Linguistics group and the Interactive Media Lab at the University of Toronto and has joined the 'Bringing Evidence to the Point of Care' project. She will be working on the usability testing and the User Interface (UI) design work.

Natasha Martin is a M.A.Sc. student under the supervision of Prof. Mark Chignell. Natasha completed her BA Hons. at the University of York in Psychology and Mass Communications. She is currently working on the EPOCare project which is supervised by Mark Chignell in the Department of Mechanical and Industrial Engineering and she is investigating the adoption of technology and it's acceptance by doctors in the medical environment.

Romana Danicic is a M.A. Fin. Eng. student under the supervision of Prof. Tom Salisbury. Romana completed her B.Sc. Hons. at York University in Mathematics and Computer Science. She is investigating the relevance of evidence based content for clinical decision making and different ways of presenting and navigating through that information on handheld devices. Romana is also working on the integration of drug information with the evidence based resources.

Eric Tursman a M.A.Sc. student under the supervision of Prof. Mark Chignell. Eric completed his B.A.Sc. at the University of Toronto in Industrial Engineering, receiving the Centennial Thesis Award for his fourth year thesis. He is investigating the implementation and customization of a search engine and interface for EPOCare applications. He is also helping facilitate the ongoing usability testing sessions of the current project phase. Eric is planning an investigation into alternative input methods for small-devices in a medical setting.

Danielle Lottridge is an undergraduate student under the supervision of Prof. Mark Chignell for her undergraduate thesis. Danielle will complete a B.Sc. Hons. at the University of Toronto with a specialist in Human-Computer Interaction, given by the Computer Science Department. She is currently working on the EPOCare project which is supervised by Mark Chignell in the Department of Mechanical and Industrial Engineering and she is investigating the socio-psychological impacts of mobiles devices in the medical setting. Danielle will be continuing her work on this project as she will be starting her M.A.Sc. under the supervision of Prof. Mark Chignell in the Fall of 2003.

Yun Niu is a PhD. student supervised by Professor Graeme Hirst in the Computer Science Department at the University of Toronto. She completed her M.A.Sc at the Nanyang Technological University of Singapore. Yun Niu currently is working on the EPOCare Project with the knowledge management group. She is applying computational linguistics technologies to refine the information retrieval result. The goal is to provide accurate and concise answers to questions posed by clinicians.

Appendix B: References

1. Leape L, Brennan TA, Laird N et al. The nature of adverse events in hospitalised patients. Results of the Harvard Medical Practice study II. N Engl J Med 1991;324:377-84.
2. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. Washington: National Academy Press, 1999.
3. Ortiz E, Meyer G, Burstin H. The role of clinical informatics in the Agency for Healthcare Research and Quality's efforts to improve patient safety. Proc AMIA Symp 2001;508-12.
4. Bates DW, Spell N, Cullen, DJ. et al. The costs of adverse drug events in hospitalized patients. JAMA 1997; 277:307-311.
5. Sackett DL, Straus SE. Finding and applying evidence during clinical rounds. The evidence cart. JAMA 1998;280:1336-8.
6. Straus SE, Sackett DL. Bringing evidence to the point of care. JAMA 1999;281:1171-2.
7. Bates DW, Leape LL, Cullen DJ et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA1998;280:1311-6.
8. Bates DW, Teich JM, Lee J, Seger, D et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6:313-21.
9. Teich JM, Merchia PR, Schmiz JL, et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med. 2000 Oct 9;160(18):2741-7.
10. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of 'corollary orders' to prevent errors of omission. J Am Med Inform Assoc 1997;4:364-75.

Appendix C: Criteria to Assess the Validity and Clinical Usefulness of Drug Reference Databases

Quality of Information

1. Explicit methodology is used for searching the evidence:
   Yes No Can't tell (i.e. not reported)
2. Explicit methodology is used for appraising the evidence:
   Yes No Can't tell (i.e. not reported)
3. Content is peer reviewed:
   Yes No Can't tell (i.e. not reported)
4. Content is referenced:
   Yes No Can't tell (i.e. not reported)
   
   If yes, access to abstracts provided?
   Yes No
5. Content quality is graded (i.e. levels of evidence reported):
   Yes No Can't tell (i.e. not reported)
6. Content is updated every:
   months Can't tell (i.e. not reported)
7. Does the site list the developers/creators?
   Yes No
   If yes, who are the developers/creators?
   Pharmaceutical company
   Professional Society
   Other
8. Does the site list competing interests of the developers/creators?
   Yes No

Usefulness of Resource

1. Search engine is present for drug name, or any text word?
   Yes No
2. PDA version of resource is available
   Yes No
3. Number of clicks required to get to information about starting dose of drug:
4. Number of clicks required to get to information about starting dose of drug in an elderly person:
5. Number of clicks required to get to information about adverse effects of drug:

Content

1. Are approved indications for drugs commonly included?
   Yes No
2. Is cost of drugs commonly included?
   Yes No
3. Is dosage of drugs commonly reported?
   Yes No
4. Are drug dose adjustments for renal patients commonly included?
   Yes No
5. Are drug dose requirements for elderly patients commonly included?
   Yes No
6. Are drug dose requirements for hepatic patients commonly included?
   Yes No
7. Is drug safety in pregnancy commonly mentioned?
   Yes No
8. Is drug safety in lactating women commonly mentioned?
   Yes No
9. Are drug interactions commonly mentioned?
   Yes No
10. Is frequency of adverse effects commonly reported (is a percent or range or percent provided)?
    Yes No
11. Is the severity of the drug interaction commonly mentioned?
    Yes No
12. Cost of Resource (per individual subscription) $US_