MOXXI III Final Project Report

Health and the Information Highway Division, Health Canada
February 2004

Table of Contents

• Main Investigators, Collaborators and Partners
• 1. Project Description
• 2. Achievements
  • 2.1 Conceptual Framework
  • 2.2 Preliminary Work
  • 2.3 Evaluation Design
  • 2.4 Detailed System Specification
  • 2.5 Development
    • 2.5.1 MOXXI Electronic Prescription Software Development
    • 2.5.2 Development of Asthma and Diabetes Advanced Decision Support Systems
    • 2.5.3 Pharmacy Interface Development
    • 2.5.4 Shared Care Development
    • 2.5.5 Automated Compliance Support System Integration
    • 2.5.6 Alert System Integration
    • 2.5.7 Asthma Advanced Decision Support System Integration
  • 2.6 Study Set-Up
    • 2.6.1 Support Infrastructure
    • 2.6.2 Training Program and Documentation
    • 2.6.3 Communication Tools - Support Service, FAQs, and Other Medias
    • 2.6.4 Setup and Configuration of the Servers and the Data Repository
    • 2.6.5 Physician Recruitment
    • 2.6.6 Pharmacist Recruitment
    • 2.6.7 Patient Recruitment
    • 2.6.8 Baseline Characteristics
    • 2.6.9 Project Management Committee and Production Meetings
  • 2.7 Rollout
    • 2.7.1 Site Implementation
    • 2.7.2 Site Testing
    • 2.7.3 Training
  • 2.8 Intervention
    • 2.8.1 Observation and Process Evaluation
    • 2.8.2 Support
  • 2.9 Follow-Up Assessment
    • 2.9.1 Data Collection
  • 2.10 Dissemination and Future Directions
    • 2.10.1 Scientific Evaluation
    • 2.10.2 Scalability and Sustainability
    • 2.10.3 International and Provincial Project Review
    • 2.10.4 Project Reports
    • 2.10.5 Communication Plan
  • 2.11 Lessons Learned
    • 2.11.1 Technological Performance
    • 2.11.2 Partnerships
    • 2.11.3 Technology Obsolescence
    • 2.11.4 System Analysis and Technical Requirements
    • 2.11.5 Performance of Technologies
    • 2.11.6 Prototype and Feedback on Interventions Prior to Implementation
    • 2.11.8 The Difficulties of Selecting Vendors
    • 2.11.9 Managing Contracts
    • 2.11.10 Qualified Technical Personnel Recruitment
    • 2.11.11 Personnel Turnover
    • 2.11.12 Research Versus IT - Bridging the Communication Gap
    • 2.11.13 Project Management, Administration and Resources
    • 2.11.14 Multiple Partners, Multiple Environments
    • 2.11.15 Sustainability
  • 2.12 Conclusions
    • 2.12.1 Responsiveness to Needs
    • 2.12.2 Further Development
• 3. Products and Documents Generated
• 4. Main Impact
  • 4.1 Human Resources Impact
    • 4.1.1 New Skills for Physicians
    • 4.1.2 Changes in Access to Health Service Information
    • 4.1.3 Cost Effectiveness
  • 4.2 Privacy and Protection of Information
    • 4.2.1 Barriers Created by Privacy Requirements
    • 4.2.2 Unexpected Findings
  • 4.3 Policy and Research Implications
• 5. The Future
• 6. Communications

Main Investigators

McGill University, Faculty of Medicine

Robyn Tamblyn, PhD, Principal Investigator
Allen Huang, MD, Co-Investigator
Gillian Bartlett, PhD, Co-Investigator
Pierre Ernst, MD
Jean-François Yale, MD
Roland Grad, MD
Robert Perreault, PhD
Michal Abrahamowicz, PhD
Robert Platt, PhD

McGill University, Faculty of Management

Alain Pinsonneault, PhD Laurel Taylor, PhD
Université de Montréal, Faculté de médecine, Département de Pharmacologie
Pierre Larochelle, MD
Université de Montréal, Faculté de Pharmacie
Claudine Laurier, PhD

Collaborators And Partners

Canadian Health Infostructure Partnership Program (CHIPP)
Collège des Médecins du Québec
Ordre des pharmaciens du Québec
Association des Bannières et Chaînes de Pharmacie du Québec
Association Quebecious des Pharmaciens Proprietaires (AQPP)
La société informatique des pharmaciens Inc. (SIP)
Informatique Demers, Lambert, et Desrochers Inc. (DLD)
Bell Mobility
Hewlett Packard
Pharmaprix
Le Groupe
Jean Coutu
TAGGE Medical Solutions
Syscor
Astra Zeneca

1. Project Description

McGill University in Montreal and several partners are using secure information and communication technologies to enhance the safety and effectiveness of drug prescribing and dispensing practices and to promote proper use of medications among patients.

The MOXXI project's electronic prescribing, integrated drug and disease management system links doctors, pharmacists, patients and external sources of information on drug monographs and alerts. The overall objective is to reduce the risk of drug-related illness and to improve the management of chronic diseases, such as asthma, that depend heavily on regular and appropriate medication use.

MOXXI has several elements, most of them at an experimental or pilot stage. For instance, it uses electronic prescriptions to reduce the chances of misinterpretation by pharmacists. It also gives authorized health professionals access to information on all dispensed prescriptions, to reduce the risk of incomplete drug information about current drugs at the time of prescribing.

MOXXI developed an electronic guidance and alert system, based on clinical and epidemiological research. The information can help ensure that patients receive the most appropriate drugs and dosages for the proper duration, and to avoid drug allergies and other known hazards.

The MOXXI project also has a component to verify prescription compliance. It combines counseling by pharmacists with an automated call system that reminds patients to take their daily medications and to refill prescriptions that are close to running out. The technology can also be used to stop medication orders and to supply patients with information about managing their diseases.

McGill's project team, funded in part under CHIPP, is working with the Canadian Institute for Health Information to develop Canadian data standards that will allow the MOXXI system to be integrated with laboratory diagnostics and other elements of the general health care system.

Other principal partners include the Integrated Health Care and Research Network of Quebec; la Régie régionale de la santé et des services sociaux - Montréal Centre; the Institute of Clinical Research of Montreal; the College of Physicians of Quebec; l'Association des bannières et chaînes des pharmacies du Québec; l'Association des pharmaciens propriétaires du Québec; and Tagge Medical Solutions Inc.

With rising costs for new drugs and utilization rates, prescription drug use represents the fastestgrowing sector of health care spending. In 2000, 15.4% of health care spending in Canada- $15.1 billion-was on drug treatment, whereas less than $1.8 billion was spent on drugs two decades ago. There is evidence that the potential benefits of drug treatment are compromised by the under- and over-use of prescribed medication for certain conditions, errors in the drug, dose and duration of therapy prescribed, and sub-optimal compliance. Prescribing and transcription errors are estimated to account for 5% to 23% of drug-related hospital admissions. Almost half of hospital admissions for drug-related illness are preventable as illustrated in Figure 1.

Figure 1 - Hospital Admissions for Drug-related Illnesses

The challenge for the Canadian health care system is to determine how to maximize on the benefits of existing and new drug treatments while minimizing the costs. In this regard, priority needs to be placed on identifying and evaluating the effectiveness of interventions to address patient safety, particularly as it relates to errors in drug and dose selection, as these errors alone are estimated to account for the majority of avoidable adverse events leading to hospital admission, serious disability and death.

As 70% to 80% of all prescriptions in Canada are written by primary care physicians, the implementation and evaluation of interventions that can reduce avoidable errors in prescription drug management and improve adherence to evidence-based treatment recommendations are essential. There is consensus among professionals and policymakers that computerization of drug management, including electronic prescriptions, expert alerts for dosing and prescribing problems, enhancement of drug management and compliance support follow-up is the most promising method by which optimal use of prescription medication can be enhanced to improve health outcome.

The primary goal of the Medical Office of the Twenty First Century (MOXXI-III) project is to determine whether technology-based integrated delivery systems among primary care physicians, community-based pharmacists and their patients will improve drug treatment and follow-up. The participating physicians utilize a personal digital assistant (PDA) with an integrated drug management system which provides a prescription drug profile of their consenting patients, a compliance indicator, alerts for excess dose, drug-drug, drug-disease, drug-allergy, and therapeutic duplications and an electronic prescription pad. Phase II of the project will implement advanced decision support for asthma, a shared care program between community pharmacists and primary care physicians for targeted patients with poor compliance, and an automated telephone compliance support for refills and daily reminders for medication-taking.

This state-of-the-art integrated prescription drug management system links information from primary care physicians, retail pharmacies, and patients in a geographically circumscribed area in Montreal. The innovative nature and value-added benefits of this system has been recognized, as MOXXI was the only Canadian application to win the North American 3G CDMA A-List Awards (www.3gcdmalist.com/winners.html).

2. Achievements

Implementation Plan: Goals Reached, Challenges and Success Factors

To test the potential benefits of technology-based integrated delivery systems, the Quebec-based MOXXI team developed a prototype of an integrated clinical prescribing and disease management system. The objective of the MOXXI system is to reduce potentially preventable drug-related illness and optimize chronic disease management by :

1. minimizing prescribing and prescription transcription errors;
2. improving medication compliance; and
3. pilot testing evidence-based alerts and reminders for chronic disease management in asthma and diabetes.

To reduce prescribing and transcription errors, we had to develop a technology-based delivery system that would provide: (a) access to information on all dispensed prescriptions; (b) a guidance and alert system for drug and dose selection, as well as drug-allergy, drug-disease, and drug-drug interactions, therapeutic duplication and excess therapy duration problem detection; (c) electronic prescriptions that can be directly integrated into pharmacy management systems without transcription; and (d) communication of medication stop and change orders to dispensing pharmacists.

To improve medication compliance, an advanced telecommunication messaging system for medication refill reminders and disease self-management education was developed, along with an automated referral program for drug review and counseling by trained clinical pharmacists for target essential medications in the elderly and asthma and diabetes patients with sub-optimal compliance.

The MOXXI-II initiative [electronic transmission, with consent, of current drug profiles for persons in the public drug program] was extended so drug profile information for persons in private insurance plans (approximately half of the province) could be transmitted to primary care physicians and targeted speciality clinics. To extend the model to support the MOXXI-III objective, we needed to:

1. develop interfaces with the main software providers for community-based pharmacies;
2. develop a model for secure transmission between private and public sector institutions; and
3. deploy the electronic authorization and consent signaling mechanism developed by the Quebec team.

In addition, we were to pilot test an electronic prescription transmission and stop order function that will allow prescriptions and stop orders, written in electronic medical records, to be electronically sent to the dispensing pharmacy.

The final work plan to reach these goals involved 10 major tasks that were to be completed over 2 years. The following details what was accomplished, what were the success factors in reaching our goals as well as the obstacles that arose during the course of the project.

2.1 Conceptual Framework

Table 2.1 - Task 1: Conceptual Framework

Task #	Task Name	Sub-tasks	Description
1	Conceptual Framework and Design	Redefining the Project Scope	Redefine project scope in response to the major cut in the awarded funding, synchronization with COMPETE Project.
		Redefining Intervention	Redefine areas of intervention, define new goals, define desired outcomes, synchronization with COMPETE Project.
		Define Global System Architecture	Define system architecture according to the intervention choices, internal and external consultation review. Integration with IRIS infrastructure

The conceptual framework and design of the project was completed as scheduled. It included defining the work plan for the MOXXI III project as it now stands. The project scope was redefined to adjust to the level of funding accorded by CHIPP: the COMPETE project and MOXXI became two separate projects. Documents relating to this matter were already given to CHIPP when this occurred at the very beginning of the project. We are keeping an informal communication link with COMPETE.

2.2 Preliminary Work

Table 2.2 - Task 2: Preliminary Work

Task #	Task Name	Sub-tasks	Description
2	Preliminary Work	CHIPP Appendices Completion	Executive summary, project workflow, organizational chart, budget and cashflow projections, technology description, description of partners, project summary sheet, sources of funding, project evaluation plan, risk management plan, proprietary rights and ownership, communication/ dissemination plan, partner contribution forms.
		Preliminary Technology Studies	Test different wireless PDA solutions. Determine data back end environment (hardware, database management system, application servers, gateways, firewalls, etc.)
		Agreements and Contracts	CHIPP, partnerships, non-competition, nondisclosure, conflict of interest.
		Define Mandates	Members, partners, consultants.
		IRB Approval	Research protocol, application form, consent form.
		RAMQ Data	Definition of live RAMQ data requirements
		CAIQ Approval	Assess legal and ethical issues, application form.

Task 2 was completed as planned. CHIPP appendices were completed as required, the work plan was established and we began selecting the technologies that would allow us to put an integrated e-prescribing and drug management system in place. Technologies were assessed for both the back end and the front end (IPAQs) of our solution. First, to ensure the connectivity between ourselves and the clinics, we hired an external firm to test wireless networks of Bell, Microcell and AT&T's products in the participating physicians' clinics. Based on the report produced, Bell's solution was selected and a package was negociated. Previous experience in MOXXI II had indicated that mobility was essential, as physicians were very mobile (multiple office settings); thus we looked for a light, portable solution. The Compaq iPAQ was selected because at the time, it was the only PDA that could be used with Bell's 1X card; it was also one of the most powerful PDAs on the market and sported a color display that was easy to read.

We successfully obtained the Institutional Review Board (McGill) approval for the patient brochure, pharmacist brochure, physician brochure and all three types of consent forms. Definition of RAMQ data requirements were completed but the RAMQ was late in delivering them. The RAMQ had initially thought that what had been developed during MOXXI II was not re-usable and time was spent evaluating alternative architectures, to conclude that modifications were required, but were not as major as was initially thought. The estimate was used to establish the base of the current contract.

CAIQ approval was obtained though we were delayed because the role of one of our partners (TAGGE) and the flow of patient information had to be redefined (see below for explanations). There were two factors involved in obtaining the CAIQ approval without major problems: first, the current research and management team had developed an excellent relationship with them through experience and previous projects. Thus the project team had an excellent understanding of what was expected. Also, the team chose to involve the CAIQ from the first: it is much easier to modify a project in its early stage when most of it is still on paper to satisfy the CAI's requirements then to do so when resources have been committed.

The signature of various agreements and contracts were more challenging: first, there were some delays in obtaining the signature of the contribution agreement which had an impact the project's timeline. Second, the role of TAGGE changed radically, and this had an impact on the project's timeline. Initially, TAGGE was responsible for establishing the communication link between MOXXI and the pharmacies: patient data would thus go through the TAGGE server. Because of confidentiality issues, the McGill's IRB as well as our lawyers strongly recommended for us to rethink this strategy. In the end, we chose to ensure the development of these interfaces ourselves: this required modification to our IT architecture and some additional resources that we had not originally planned for. Third, negotiations with participating pharmacy service providers, professional orders and banners were difficult at times. We believed that getting the buy-in of the professional orders was of utmost importance for the success of the project. We had previously worked with the Collège des médecins in previous MOXXI projects and had a good relationship with them, which helped in obtaining their support once again. This was not the case on the pharmacist side: it was the first time we approached their various professional orders and we did not grasp quickly enough the complexity of their relationships to one another, nor were we aware of some existing tensions between some of the participants. Thus, in 2002, the project's research and management team invited the major players to an information session presenting the project and asking them to participate. The would-be participants were wary because the plan was already laid out: they felt excluded from the decision process and this created resistance that was difficult to overcome. We then asked experienced partners within the private sector to help us deal with the problem. Instead of solving it, it aggravated it, because the pharmacists felt the project represented private interests. The communication and relational skills of the MOXXI team members were key in reassuring pharmacists, owners, professional associations and service providers. There remains a single service provider who has not developed an interface between the MOXXI project and its databases, as the costs for this interface were inflated 8-fold over their competitor's.

The contract signatures with Imagina (development of asthma and diabetes algorithms) and Vigilance Clinique (medication and interaction database) were much simpler. Imagina approached us; they had previous experience in healthcare related projects and understood the context in which we evolved. Also, they had a known expertise developing web-based applications on PDAs, which is not easy to find. We gave them an analysis contract at first; as it was successful, we went on to have them develop the diabetes algorithm, finally followed by the asthma algorithm.

Vigilance Santé was selected as our medication database as it had been very favorably compared to American products such as First Databank and had the additional advantage of being bilingual.

2.3 Evaluation Design

Table 2.3 - Task 3: Evaluation Design

Task #	Task Name	Sub-tasks	Description
3	Evaluation Design	Evaluation Framework	Research design
		Content Specification	Define content for targeted clinical areas, general decision support, including research outcomes
		Develop Data Collection Instruments	RAMQ data functional requirements, formats, access, transmissions

Task 3 was comprised of the elaboration of the research design, the content specification and the development of data collection instruments - in fact, putting together the research protocol, included in the revised appendix given to CHIPP. These activities had to be finished for us to submit the CHIPP indexes, but could still be updated as the project evolved, which it did. There was one major variation from the original plan: the MED-ECHO data has not been included in the intervention because at the time of the intervention, the contract between the RAMQ and the MSSS was not yet finalized.

Furthermore, although IRIS-Q contributed funding to the establishment of an on-line access to the hospitalization prolonged negotiations have not made access possible during the lifetime of the project.

2.4 Detailed System Specification

Table 2.4 - Task 4: Detailed System Specification

Task #	Task Name	Sub-tasks	Description
4	Detailed System Specifications	Specialized Content	Advanced decision support for asthma, diabetes. Patient referral program for medication compliance. Message content, trigger rules, user information presentation.
		Detailed Informatics Architecture	Database server - first tier, application server - second tier, telecommunications inter-operability infrastructure, security, wireless PDA integration methods.
		Process Specifications	Define data sources and responsibilities, including security, and transmission protocols.
		Integration with IRIS Infrastructure	Integration of MOXXI database into IRIS database server.
		User Interface	Develop intuitive user interface & initial training program. Preliminary screen shots.
		Prototyping	Develop prototype, test alpha system, incorporate feedback.
		Technical Documentation	Implement a comprehensive documentation management system. (eg meeting minutes, specification documents, source code, design specs, including secure document areas).

In task 4, we aimed to finish the detailed system specification including: the content of the asthma and diabetes decision support systems and the patient referral program for medication compliance, the detailed IT architecture, the information process (definition of data sources, transmission protocols, etc.), the user interface, the prototype. We also determined we would need a template to organize the technical documentation.

The IT architecture and the information process were defined, taking into consideration the fact that TAGGE would no longer act as an intermediary. Security standards were dictated by the MUHC policies and infrastructure. The data transmission protocols were defined with the RAMQ. Wireless communication architecture with the McGill University Health Center was finalized. Having to go through the MUHC for communication with the external world increased the complexity of the architecture and is a challenge that continues to be dealt with daily. Some modifications were done later during the project due to performance issues with the MUHC network.

The basic infrastructure needed to support the MOXXI integrated drug management system was provided by hardware and software acquired by IRIS-Quebec including the telecommunications networks, the Sun servers and Oracle database. In addition, IRIS-Quebec funded the consultative services of CGI to provide a documentation system and full integration of MOXXI into the clinical notes system that would be used by out-patient hospital clinics because it would increase the utility and breadth of an integrated drug management system for both primary care and specialty physicians. The IRIS-Quebec infrastructure architecture includes a system that is scalable, extensible and maintainable. The relational database management system is Oracle 9i Enterprise Edition - this manages all access to the data in the database (data manager and data warehouse). The middle-tier server is Oracle 9i Application Server - this product is used to run the Research Toolkit application (including web services and business logic). The development tool is Oracle Internet Development Suite. This tool will be used to manage and accelerate the development of the Research Toolkit using standards such as Java, XML, CORBA, & HTML. The data warehouse development tool is Oracle Warehouse Builder, which will be used to accelerate the development of data loading and processing jobs for the data warehouse. The data mining tool is Oracle's OLAP Tool, which will be used to analyze data. The data backups will be performed by a SUN Storedge Library that is capable of storing approximately 2.2TB of data. The Servers are SUNs for Development and Testing (& Q/A). These servers are all dual processor with 4 GB RAM and 72 GB HD. The production server is a SUN with dual CPUs with 6 GB RAM and 300 GB RAID 5 HDs. The management console is a SUN Management Center and Oracle Enterprise Manager. This will be used to monitor the operation of the system and tune it. Security is handled with Oracle Advanced Security option which exceeds all industry standards including all leading encryption/decryption standards as well as automated audit trails. The firewall is handled by a combination of hardware and software - these are Cisco and CheckPoint. IRISQ is also compliant to MSSS-RTSS specifications.

We also attempted to establish a comprehensive document management system to allow the team access to relevant information easily. CONCERT, software used by CGI to manage information, was implemented. After several weeks of usage, it became clear that CONCERT was too bureaucratic for our context: entering all the required information soon became too onerous a task. We returned to organizing information in a less formal yet still structured way on a central server, directories pertaining to the relevant subjects were created and ordered.

The design of the screenshots for the e-Rx system was completed, as was a prototype with some of the simpler screens. The screenshots were reviewed by the research team and deemed satisfactory.

The content of both decision support systems was developed in collaboration with medical experts in asthma and diabetes, overseen by a MOXXI team member with a clinical background and some IT experience. These decision support systems were to provide physicians with detailed prescribing and management support according to the current Canadian evidence-based asthma and diabetes guidelines. Based on the patient's symptoms, test results and current medications, physicians were to be provided with the appropriate choices for drug therapy and with preventive measures when required (including reminders for intervention for hypertension, for dyslipidemia, for microalbuminuria, for decreased foot sensitivities, for smoking, for use of ASA, depending on the data obtained in the previous sections) in the individual patient. Therapy is guided not only by providing appropriate drug choices but also by providing key messages to justify the suggested therapy and alert messages when less appropriate therapy is selected. All the messages provided to physicians were translated from the current guidelines. Other key elements of the guidelines such as diagnostic criteria, the need for and information on how to refer a patient for asthma education or to a dietician or to a Diabetes Education Centre, the why and wherefore of prescribing an individualized self-management plan, are all included as an integral part of the two advanced decision support systems. Content documents were created and reviewed by the researchers and by clinicians participating in the project, before being approved for development.

The patient referral program for medication compliance, more commonly referred to as the Shared Care program, was designed by pharmacists from the University of Montreal. The shared care program is an assessment and counseling consultative service that is provided through a partnership between physicians and trained community pharmacists for patients experiencing medication compliance difficulties. It targets a subset of patients who are using medication that is essential and effective in the management of chronic illness, and who are experiencing compliance difficulties. The content of the intervention was designed to document the care delivered as well as to generate an automated e-mail consult note for both the dispensing pharmacists (if different) as well as back to the referring physician. The work done by the consulting pharmacist includes: a) the creation of a complete medication list, including the OTCs and herbals b) a drug review c) an assessment of life habits in order to give appropriate counseling on how and when to take the medication and d) a compliance evaluation. The flow of information between participants was established: patients who are eligible and recommended for shared care are identified to the primary care physician - the physician gets a warning on his PDA when his patient's target drug compliance is below 70%. He can then proceed to electronically refer the patients to a participating pharmacist for the intervention. The referral signal is electronically sent to the MOXXI server before being transferred to the pharmacist, along with the patient's coordinates and pharmacological profile. The pharmacist contacts the patients to make an appointment, keeping a log file of all contacts. Following the intervention, a pharmaceutical consult report is created and electronically sent to the physician enrolled in the study. For other physicians, the pharmaceutical opinion is mailed as usual. Two follow-up telephone contacts are conducted with the patient after the counseling session. Follow-up status, problems and interventions were to be documented in the web-based application system. Those specifications were documented. We planned to begin the development of the shared care program after the e-Rx module was finished.

2.5 Development

Table 2.5 - Task 5: Development

Task #	Task Name	Sub-tasks	Description
5	Development	Software Development	For PDAs, PCs and call-up system: translate modules in French-English, develop version upgrade process.
		Application Integration	Integrate internal PDA applications and databases, integrate applications with Purkinje software, integrate phone system and web site.
		Drug Database Integration	Vigilance Clinique information validation & integration
		Communication Link	MOXXI server, TAGGE server, pharmacies, RAMQ, physicians.
		Internal Quality Assessment	Integrated system testing (performance, failures, data integrity, security audit)
		External Standardized Performance Testing	Alpha & beta software & system testing

Task 5 was the development task. This task included developing the electronic prescription software for the physician's iPAQs in French and in English, integrating the TAGGE home monitoring system, developing the asthma and diabetes decision support systems, developing the shared care system, integrating the alert systems, integrating the RAMQ and pharmacy data, testing communication links and testing the alpha and beta version of all applications.

All development sub-tasks were completed but none of them were on schedule. Significant additional costs were incurred in this category. We encountered many problems during this phase, most of them seemingly created by a poor choice of personnel or outsourcing contracts. In retrospect, we realize that there are very few competent personnel in larger or small consulting companies who can provide required services.

2.5.1 MOXXI Electronic Prescription Software Development

We chose a developer who had worked on MOXXI II because of our successful previous relationship and his knowledge of the clinical environment. This person had the mandate to develop the electronic prescribing system, implement the asthma and diabetes decision support systems, integrate the alert system, implement the shared care system, and create the interfaces between these applications, and integrate MOXXI with TAGGE and their home monitoring system. The software development began in February 2002 because of delays in the contract signature with CHIPP. The initial efforts were concentrated on the development of the electronic prescribing and integrated drug management system, as this was to be the cornerstone of the project, into which the other applications would link. The programmer chose Visual Basic as the programming language for the system because of his knowledge of it. As it turned out, the programmer did not have the level of proficiency sufficient to program on an iPAQ. The development environment on hand-held computers is very different from the environment on a PC platform. Resources (on-line help, debugging tools, etc...) are few, the processing power of the machine is limited, the memory available is restricted, the screen is small and requires clever interface design to avoid increasing layers. Thus it takes an extremely competent person to be able to develop something of this complexity in such an environment. We did not immediately realize that this person would be unable to accomplish the tasks established, and we incurred further delays trying to let him solve the problems he had created. The programmer was relinquished from his service contract. It then took three months to find another candidate with the appropriate skill set. This new person had extensive experience in IT and had worked on a similar project in a hospital environment. At first it seemed that although major parts of the application had to be re-written, it was possible to salvage nearly 50% of what the first programmer had done. Thus three months later, in December 2002, a first version of the application was deployed. It did not have all the functionalities: more specifically, the integration of the alert system and the capacity to electronically send prescription and order stop and/or change in a prescription were not completed. Performance was not exactly what was expected but was tolerable.

Our goal was to give this first version to physicians to get their feedback before going further. At this point in time it was essential to provide something to physicians to maintain their involvement. Not delivering on time to users has a negative impact on their perspective of the project and undermines the trust they have in the team so it's important to give them something even if it's not a complete version. Physician's feedback was then used to improve the application as we added additional functions. Before the application was given to physicians, our own team of participating clinicians tested the system and commented on the interface and available functions. This helped the debugging process tremendously.

As the required functionalities were developed and as we integrated change requests in the application, we realized that the initial's programmer's application design placed some constraints on what we were able to do. During the summer, we worked at stabilizing the application and its environment. In the fall, work on the final major feature, the stop and change order, began anew. We re-designed this function based on our newly gained experience. Tests of the prototype showed that it would reduce the overall speed of the application by 20% or more, which was unacceptable, as the iPAQs performance was already less than expected. In clinics, speed is of the essence, as an average appointment lasts approximately 7 minutes. Thus we began to work on a version of the application which used a different programming language, would be object-oriented, and would allow us the shed the weight of the intrinsic design flaws of our first programmer's application. This version's completion and release are planned for March 2004.

2.5.2 Development of Asthma and Diabetes Advanced Decision Support Systems

When we changed our internal developer in May 2002, we also chose to outsource the development of the asthma and diabetes expert systems to an external firm, Imagina. Initial development estimates had been done by the first developer, however we expected, based on performance, that he had grossly underestimated the amount of work required to deliver these applications. Imagina's first task was to do a technology assessment report to validate what would be the best way to proceed. Developing on iPAQs was more difficult than anticipated; scalability was an issue, as was maintenance. Three alternative models were proposed. The one that seemed to offer the best possibilities was a web-based design: the application would reside on a central server and would be accessible from anywhere via the web. It would pull information from the central database and the calculations required for decision support would run on the server. The initial specification document produced by our content experts implied fairly complex calculation that would be executed quickly on the server, thus improving performance. The evaluation was deemed satisfactory and we contracted Imagina to carry out the development. Imagina developed the decisions support systems for asthma and diabetes based on the specifications written by the content experts. While Imagina delivered the diabetes and asthma algorithms as per our required time delivery date of December 2002, they could not be immediately tested as they were developed within an environment that was similar but not identical to the environment within which the application was to be deployed. Our second challenge was that the testing of the diabetes and asthma algorithms were delayed by the necessity of working out the infrastructure details for testing within the hospital based environment for the physicians who designed the applications.. We then tested the asthma algorithm with 4 general practitioners at the Jewish General Hospital. It was deemed completely unusable. First, data transmission was extremely slow; it took approximately 15 seconds for a page to be downloaded. Initially, this was diagnosed as a network problem. We were too far behind the MUHC's firewall and it affected our performance. Correcting this situation helped improve the data transmission speed only fractionally. Additional tests showed that the iPAQ has a very low level of performance to encrypt and de-encrypt data. Encryption is essential if we are to keep patient data confidential and secure. Thus we had to minimize the number of times the iPAQ had to access the server to display information. Furthermore, the design was not considered user friendly as it was necessary to navigate through many screens before getting to information that users deemed relevant.

We then had users evaluate the diabetes algorithm. The performance problem remained; in this case, one major problem was the high level of data entry required before the system suggested any treatment recommendations. It became apparent that physicians would not do all the data entry required to get the results. Minimizing this data entry meant creating a link with and integrating test results from laboratories. This was not within the project's scope and so we chose to concentrate our energies on reworking the asthma decision support system.

As asthma is a simpler disease to manage than diabetes, we concluded an agreement with Imagina that they would redevelop the asthma decision support system and would assume part of the costs. Based on the physician's feedback and expert input from the MOXXI team, we developed a 'two screen' design. The initial screen displays the patient's current medications, asthma control check list, peak flow measures, ER visits and home monitoring results when they are available, and allows physicians to update this information. The second page offers recommended treatment or warning based on information entered in the first page. This simplified design was implemented and an alpha version of the application was delivered December 5th, 2003. We are currently testing the application. More testing is required before we deploy it to participating physicians. We do not foresee any problem in terms of training, as the interface is very simple and user friendly.

2.5.3 Pharmacy Interface Development

Since TAGGE would not be an intermediary between the pharmacies and ourselves anymore, we hired an external firm based in Toronto to develop the interface with the various pharmacy service providers. This turned out to be a mistake: what should have been a relatively simple application took months to develop, and was finally abandoned. First, there was an unforeseen culture clash that was deeper than a simple language barrier: the pharmacy's IT service providers and our consultant met to clarify requirements and ensure that ThinWeb would develop a product compatible with all the pharmacists' software. During the meetings, it became apparent that the participants simply did not seem to understand each other. Meetings were held and everyone left understanding something different. In the end, this generated feelings of frustration that from our participating pharmacy software providers that we had to manage. Second, the consultant's internal structure made it difficult to ensure proper follow up. ThinWeb was based in Toronto but the programmer working on MOXXI was based in Nova Scotia. Communications and coordination between them, ourselves and the pharmacy's service providers was not always efficient, and this created additional tensions. The delays and the problems compounded. We finally asked for their IT resource to come to Montreal to work directly with the people involved. ThinWeb was unable to accommodate our requirements and for this reason we terminated our contract with them. We had been delayed for more than 6 months in installing the first pharmacies. We chose to do the development internally, which overall took approximately 4 weeks spread over 4 months : delays were incurred because at that point in time, we had lost our priority with the vendors and they tested and modified their software when they were able to. Our application corresponded to the IT service providers' requirements and adapted to their request, smoothing ruffled feathers.

An unforeseen difficulty lay with the fact that it was not always possible for pharmacy's IT service provider's software to be modified to accommodate a completely integrated electronic prescription system. Two of the pharmacy software vendors were willing and able to modify their current software so that it would be possible for their pharmacist to both pull electronic prescriptions from the MOXXI server and send their patients' pharmacological profiles. The two other vendors were not able to develop the interface both ways. In one case, the system in place was old and had reached its limits. The vendor was developing a new version of the software and was understandably unwilling to invest major resources to develop something that would not be re-used with their updated system. In the other case, business priorities took precedence over research. At this time, the pharmacists using these vendor could not retrieve electronic prescriptions, but could send their patient's pharmacological profile electronically to the MOXXI server, where the information was integrated with the RAMQ data before being transferred to physicians. One of these vendors has indicated that the new version of his software has the necessary fields so that we shall be able to have a fully integrated electronic prescription with the new product.

TAGGE's telephone compliance support system and the shared care program were put on hold. Their development date was postponed to January 2003 as we needed to concentrate our efforts on the electronic prescribing system. These two systems had to be implemented in the physician's offices after the electronic prescription system had been successfully implemented.

2.5.4 Shared Care Development

After testing the asthma and diabetes systems, we chose not to develop the shared care as an electronic application. At this point, we realized that even with highly competent resources, developing complex applications on an iPAQ was not easy. Also, we wished to test the intervention before going further as this service program had never been implemented and thus the ultimate requirements for IT-supported intervention could not be effectively defined. Creating a paper-based tool forced us to refine the content. Over the course of the summer, our colleagues at the University of Montreal prepared the tools, organized training sessions and made ready to train the pharmacists in the fall, as most of them were not available during the summer months. In the meantime, we finalized the interface that would advise the physician that his patient was eligible for shared care (compliance less than 70%), allow him to refer the patient on his iPAQ and send the referral information to the MOXXI server. Since the shared-care referral and consultation reports are not yet electronically automated between the pharmacist and physician, a monitoring process was created to ensure that the various components of the sharedcare program were executed as planned. A weekly report is generated to identify the patients referred to the shared care program. This report includes fields that the MOXXI physician coordinator uses to manually enter information. These fields include: the date the referral is faxed to the physician, date of the initial consulting session, date the initial report is received, date of the first & second follow-up session, date the final report is received and date the initial and final reports are faxed to physicians. Included in this report are dates which highlight any delays in the process. For example, if an initial report is not received within 3 weeks of the date the referral was faxed to the pharmacist, a field will identify a date when to follow-up with the pharmacist to understand why this step did not occur. A similar procedure occurs for the followup sessions. The report was also created to identify any drop-outs from the program and to document the reason for abandoning the shared care program. Currently there are 946 patients eligible for the shared care program. The pharmacists were trained in August, September and October of 2003. The referral notices to the shared care program have only recently begun in October: the randomization of patients was implemented only in September because our resources were otherwise occupied and that the delivery of the decision support systems and the compliance support systems were late.

2.5.5 Automated Compliance Support System Integration

As for the integration of the automated compliance support system developed by TAGGE, we began preparing to implement it in September 2003. TAGGE's system still needed fine-tuning. Patients using TAGGE would receive automated telephone calls for refill reminders and an option for reminders to take their medications. We had to feed into their system the required information to contact patients and send the necessary drug information and develop the log files we wished to receive from them. At this point in time, the data transmission and communication protocol with TAGGE are established. The automated telephone compliance support system has been developed and is currently undergoing the last phase of testing. It is scheduled to go live in the last week of January 2004. Because of the unanticipated delay in the implementation of this compliance support system, those randomized to this research arm are currently being called and notified that they will begin receiving calls from the automated system in the near future. The notification call provides the participants with a reminder about the project (as it may be one year since they signed the consent form) and the reason behind the automated compliance system.

2.5.6 Alert System Integration

Integrating the alert system with the electronic prescribing system was more complex than initially planned: the software had not been developed for an iPAQ, and we were thus using the application to connect wirelessly to the server which would then run queries and send the answer back to the physician. Displaying the information in a useful and user-friendly manner to physician was a challenge. Connectivity was an issue, and the alert system was not always available. Also, in one of their updates, Vigilance Clinique modified part of their products output, which forced us to rewrite part of the application do deal with the new data format. The integration of this component was completed in June 2003.

2.5.7 Asthma Advanced Decision Support System Integration

Figure 2.1 - Asthma Control Dashboard

Integration of asthma had been developed prior to the redevelopment of the algorithm. When physician enters a patient's file, a screen pops up indicating that his patient is asthmatic, would he like to access the decision support system? (see Figure 2a). The physician enters the algorithm, where he will eventually receive recommendations or warning on management of his patient's treatment. If he chooses to prescribe the treatment, the prescription information is sent to the MOXXI system on the iPAQ. Physicians can then print the prescription and the information is integrated in the physician's database. It will later be synchronized with the MOXXI server. 90% of what has been done is re-usable as-is, 10% remains to be added to the original program, as we added functionalities making the application more user-friendly. Integration with the diabetes algorithm was not attempted as we chose not to implement it because of the re-design and development costs.

Integration of RAMQ data was completed as planned. No major difficulties were encountered. Previous experience with RAMQ data from MOXXI II greatly facilitated this step. The asthma web-based self-care portal was developed by a master's student in training, but his work was not satisfactory. We chose to abandon the implementation of this web portal as it wasn't part of our original plan. Given the situation, we preferred to concentrate on our core applications.

2.6 Study Set-Up

Table 2.6a - Task 6: Study Set-Up

Task #	Task Name	Sub-tasks	Description
6	Study Set-Up	Set Up Central Support Infrastructure	Develop training program and documentation (MD & pharmacists), set up central support office, communication tools (including website).
		Practice Workflow Re-Engineering Startup Package	Including all required documents, process, personnel, infrastructure installation.
		Set Up MOXXI Application Services & Data Repository	Install and configure server environment. Set up support & maintenance. Implement MOXXI application, migrate MOXXI-II physician databases to server. Implement data validation rules.
		Set Up Research Toolkit	Database and application server, demilitarized zone, RAMQ link.
		Set Up TAGGE Customized Services	Implementation of MOXXI functional requirements for automated patient call-up system (complementary decision support).
		Set Up MOXXI Patient Call Centre	Recruit and train a nurse.
		Physician Recruitment	Initial selection & contact, information session & brochure, enrollment & study agreement.
		Pharmacist Recruitment	Information session, press release, establish partnerships with major pharmacy chains, consolidate pharmacist entente for referral program (+/- 10 pharmacists).
		Patient Recruitment	Adapt and implement existing patient recruitment process.
		Patient Data Set- Up	For enrolled, consenting patients: 1) extract data from the RAMQ medical services and pre-populate production database, 2) download historical baseline data to MOXXI central office.
		Baseline Characteristics	Data collection & analysis of baseline population, non-enrolled patients, anonymous controls.

Task 6 was the study set-up. Sub-tasks included developing the training material for all phases of the project for both pharmacist and physicians, preparing the needed communication and support material, configuring the back-end (servers, MOXXI II data migration, etc.).

2.6.1 Support Infrastructure

To ensure our capacity to support participants meant having a way to assign and keeping track of problems that would crop up during the project. We implemented a system that we called the ProblemTracker from the NetResults Corporation. This system allowed the team to assign a problem to a resource and to keep track of the problem and of the answers of the people involved in solving it until it was taken care of. We used it mostly for follow-up of technical problems.

We determined three levels of support: the first line of support would be the physician and the pharmacist coordinator. Patients and physicians would contact the physician coordinator if they had questions, wished to comment on the project or, in the case of physicians, if they had technical difficulties and required assistance. Pharmacists could do the same with the pharmacist coordinator. Both these people had cell phones in order to be available at all times. If the calls were placed at their office and they were absent, a central voicemail system allowed the project manager to take the messages and quickly re-direct the calls depending on their level of urgency. This allowed extremely efficient and timely response.

If the problems were of a technical nature that exceeded the coordinator's capacities, they would be re-directed to a specific resource. The problems were divided into the following categories: hardware problems, bugs in the system, and communications problems that were attributed and directed to the proper team member.

2.6.2 Training Program and Documentation

Ensuring support also meant ensuring that we would have the proper tools to train participants on the various components of the MOXXI system. We established a training strategy based on two principles we felt would minimize resistance to technological change and to ensure that users would quickly develop and retain the required abilities:

Incremental change

We chose to implement change incrementally, to facilitate the learning curve of participants. Training goals were broken down into small, manageable pieces: users would be trained on one component at a time, so that they could quickly master the new skills required. These small successes are key in letting the user feel in control and able to integrate the tools in his or her workflow and encouraging them to move to the next level.

Permanent change

Many steps were taken to ensure that participants would adapt quickly to using the system and that they would remain proficient in its usage (rather than going back to their usual ways of working).

For physicians, training sessions were organized for small groups of 2-7 physicians, to facilitate support and favor exchanges between participants. The training team was a specially trained general practitioner, supported by a MOXXI team member. The physician trainer could relate to what participants experienced in their day-to-day life, address their concerns and demonstrate clear and relevant advantages of the system. After each training session, personalized support and follow-up were dispensed within the physician's clinics. MOXXI team members trained on MOXXI spent 2 half days per week for the first two weeks in the clinics. The training sessions were planned at the beginning of the week to ensure that MOXXI team members could visit the clinic the next day and encourage the physicians to start using the system immediately. The MOXXI team members remained available in the physician's waiting room and would help physicians use the system each time if they so required. This was an opportunity for the team members to inform patients about the project and to answer patients' questions.

As for pharmacists, as the interface with the pharmacies was developed by their regular service providers, it was determined that the service providers would take care of the implementation and would inform pharmacists of any new functions added to the system. MOXXI team members would have the same information as pharmacists, to be able to help them should they so require. We evaluated that the implementation would require less adaptation effort from pharmacist than from physicians, as they were already using computers and as service providers developed an interface that minimized the impact on the pharmacist's workflow.

For the shared care program, specific training would be given to select shared care pharmacists. We used peer-to-peer teaching in a small group format. Support would be given on demand, as pharmacists would use the system only when a patient was referred to the program by a physician.

Combining the two principles above resulted in the following training processes:

For physicians:

1. Learning to use the iPAQ
   1. Optional 3 hour training session (weeknight)
   2. In-clinic follow-up
   3. Personalized support and training on demand.
2. Learning to use the electronic prescribing system
   1. Mandatory 3 hour training session (weeknight)
   2. On-site personalized follow-up and support during the weeks following training
   3. Personalized support and follow-up on demand.
3. Learning to use the advanced decision support systems
   1. Mandatory 3 hour training session (weeknight)
   2. On-site personalized follow-up and support during the weeks following training
   3. Personalized support and follow-up on demand.

For pharmacists:

1. Learning to use the electronic prescribing system (new functions of their system)
   1. Service providers providing information and support
   2. Personalized support and follow-up on demand by MOXXI team members
2. Learning the shared care intervention and using the shared care system:
   1. Mandatory 3 hour training session (weeknight)
   2. Personalized support and follow-up on demand.

A. MOXXI training session accreditation

Following the suggestion of participants, we asked for the MOXXI training sessions to be accredited, which they were. Physician training sessions were accredited by McGill's Continued Medical Education Centre (CME). They are accredited as AMA (Physician Recognition Awards) Category 1 credits. Physicians following the MOXXI training sessions became eligible for 6 to 8 Mainpro-M1 credits. Pharmacist training sessions were also accredited.

B. Material

Training material was developed during the months preceding the projects, in both French and English. The user guide for the electronic prescribing system was developed using the beta version of the software; it was later updated to represent the modifications made to the software. Training programs were developed, which included the following: course plans, evaluation documentation to provide feedback, letters and invitation to training sessions, powerpoint presentations, on-line questionnaires and standardized tasks.

Training material for the advanced decision support systems was not developed because the systems were not completed on time.

Material for the shared care program was developed in collaboration with the faculty of Pharmacy at the University of Montreal and our team. Training sessions were organized by the pharmacist coordinator.

For pharmacists, training on e-Rx retrieval was taken care of by service providers, who informed pharmacists of the procedures to follow on their systems. They also informed the MOXXI team so that we could support pharmacists and answer their questions if they had any. As support and questions could be answered by service providers, we did not initially develop documentation about their system's interface. This is something that could be done for us to have additional tools to help pharmacists.

2.6.3 Communication Tools - Support Service, FAQs, and Other Medias

Communication is key in maintaining participants' involvement and in facilitating the change process. To ensure adequate communication, many communication tools were developed to inform participants about the project's activities.

Table 2.6b - Communication Tools

Tool	Audience	Description
On-line support	Physicians, pharmacists, patients	Participants can reach a MOXXI team member over the phone if they require assistance or simply have questions or comments. A central number was printed on all brochures. Business cards were distributed to physicians and pharmacists including the cell phone number of their respective coordinator.
FAQ document	Physicians	A document containing the physicians most frequently asked questions was created and distributed to participants. This tool also ensured uniform response to questions.
MOXXInfo	Physicians, pharmacists, partners	Project's monthly journal, informing participants and partners of the project's status and of events relating to the project. Also included tips on optimal MOXXI system use, article published on the project, and other information of interest for participants and partners.
Website	Physicians, pharmacists, patients and the general public	Anyone can consult the website if they have general questions about the nature of the project; specific sections have been developed for physicians, patients and pharmacists to address specific concerns each group could have regarding the project. www.moxxi.mcgill.ca
Brochures and recruitment material	Physicians, pharmacists, patients	Material explaining the project and targeting specific audience. Posters and brochures were given to participants. Physicians would distribute brochures to patient, minimizing the time it takes to explain the project.

Additionally, we planned to organize at least one event to gather together the MOXXI research team, the participants and the partners to increase participants' sense of ownership and present project's results.

Articles published on the projects were always transmitted to participants, and they were informed of any media event through MOXXInfo.

2.6.4 Setup and Configuration of the Servers and the Data Repository

The back end environment was configured successfully: we setup and configured the Application Server for Web Services and Business Logic, a Database Server, a Backup Server, a Mobile Device Server and a Firewall. The data communication links between the RAMQ and the Hospital Centers and the Research Toolkit Servers were operationalized. The MOXXI II database was implemented and MOXXI II data was migrated to the servers that we would use for the project.

2.6.5 Physician Recruitment

Our goal was to recruit 40 general practitioners whose professional address was in one of the following municipalities: Baie d'Urfé, Beaconsfield, Dorval, Dollard-des-Ormeaux, Kirkland, Pointe-Claire or Pierrefonds. To be eligible, physicians had to be 30 years of age or older, work 4 to 5 days a week in a private practice situated in of the cities previously mentioned, and bill by the act (not billing an hourly rate).

The first recruitment effort occurred in February 2002. A list of all potentially eligible physicians was obtained from the Collège des médecins du Québec. The lists we received from the Collège were not complete and we had to infer quite a bit of information. Thus, we identified general practitioners by the absence of a specialized diploma in that data field. Age was estimated using the physician's license number: a physician usually finishes his studies when he's approximately 25 years old and gets his license 5 years later. The first digits of the license number indicate the year during which physician obtained his diploma. We decided that all physicians whose license number began by 97 or more would not be eligible.

Overall, 122 physicians were identified. We sent these physicians a letter describing the project and informing them that they would soon be contacted by a MOXXI team member who would answer their questions and invite them to an information session. The letter also contained the eligibility criteria.

Following the mailing, a MOXXI team member called all targeted physicians to follow up on the letter. Physician's eligibility was also confirmed at that time. 60 physicians had to be removed from the list because they did not meet our criteria contained in the table below:

Table 2.6c - Non-eligibility of physicians (first mailing)

Reason for non-eligibility	Number of physicians
Paid per hour (not per medical act)	13
Specialist	18
Works to few hours in private clinic	17
Professional address out of project's territory	5
Retired	7

Out of the remaining physicians, 28 were not interested and 26 were never reached. Due to the low response rate, we chose to extend the project's territory to Île Perrot, Pincourt, Sainte-Anne- de-Bellevue and Vaudreuil-Dorion. We also chose to include physicians aged 28 and 29, and to lower the number of days a week worked to a minimum of 3 rather than 4.

Brochures and letters were sent to the newly targeted physicians in March and April 2002, followed up by phone calls. Amongst the 43 new potential participants, 5 were not interested, 13 never answered or were never reached and 11 were inadmissible for the following reasons:

Table 2.6d - Non-eligibility of physicians (second mailing)

Reason for non-eligibility	Number of physicians
Paid per hour (not per medical act)	3
Specialist	4
Works too few hours in private clinic	3
Professional address out of project's territory	1

Physicians who had shown an interest were invited to the information sessions. These sessions occurred in February and in March. The first session was held at the Collège des médecins, but following physicians' comments we chose to hold subsequent sessions in the West Island, near their workplace. Invitations and maps were sent to physicians prior to the information session.

Physicians who were interested but who had not been to an information session were visited by a team member who gave them the same information as their colleagues. Out of all approached physicians, 25 agreed to participate. 3 had not been in the college lists and had been informed by their colleagues. During the summer months, 3 additional physicians approached the project and chose to participate.

A last recruitment effort began in November 2002, prior to the first implementation of the MOXXI system. The list included physicians that had not been identified by the initial Collège list as well as physicians who had not met the criteria before they were changed. Brochures and letters were sent out and were followed by a phone call. 89 physicians were on the list.

Out of these, 22 were not interested, 50 were unreachable (including 8 missing phone numbers) or did not answer, 15 were not eligible. Out of that pool, 3 physicians agreed to participate, bringing our number of participating physicians to 31.

Table 2.6e - Non eligibility of physicians (third mailing)

Reason for non-eligibility	Number of physicians
Works too few hours in private clinic	3
Professional address out of project's territory	1
Does not work in a private clinic	3
Retired	7

During the course of the project, 3 physicians chose to retract their consent and stopped participating. In two of the cases, the physicians had never used the system and had never integrated it into their practice. In the last case, the physician returned the material when personal matters made assimilation of additional changes in his practice too much to deal with. Thus we were left with 28 participants. After implementing the MOXXI system with the 28 physicians, recruitment of additional physician was postponed until we ironed out the bugs in the system and ensured a high enough level of technological performance.

2.6.6 Pharmacist Recruitment

A list of all pharmacies in Quebec was obtained from our partner Vigilance Santé. Fourty-five potential pharmacies were identified based on their professional address. Brochures and letters explaining the project were sent to pharmacists working in the targeted area, accompanied by a letter from their banner demonstrating support for the project. These packages were followed by a phone call 7 to 10 days later made by the pharmacist coordinator. The pharmacist coordinator individually met with each potential participant to explain the benefits and the impacts of the project on the pharmacist's practice. Thirty-one pharmacists agreed to participate. The majority of them agreed on the spot, while a few asked to think about it and returned the signed consent form by mail a little later. The others indicated they were not interested in the project. Major selling points were the interest that pharmacist held for a new technology that had the potential to increase their efficiency and the quality of their professional service (lowering number of errors). They were well aware of the patient safety issue and MOXXI promised to be a solution to a long-standing problem. The shared care project could have been an obstacle, as they felt it required substantial time investment on their part, but the fact that any pharmacist in their pharmacy could be designated ensured this would not be a problem. The fact that there were no fees for pharmacists was likewise extremely well-viewed. One of the success factors of this high recruitment rate was the fact that the pharmacy coordinator was a pharmacist himself, giving him a deep understanding of his colleagues' needs and concerns and giving him additional credibility in their eyes.

2.6.7 Patient Recruitment

In June 2002, physicians were given the material to recruit patients, which included patient brochures, posters, consent forms, a list of participating pharmacies, a guide on patient recruitment and a Pendaflex to file the consent forms once they were signed by patients. Consent forms were printed on NCR paper: the first copy belonged to the patient; the second, to the physician; and the third copy would be brought by the patient to his pharmacy.

The following recruitment process was suggested to physicians:

• Display the posters in the waiting room.
• Ask the staff who welcomes patients to give each patient a brochure about the project and to insert a blank consent form in the patient's file.
• Talk to your patient about the project and obtain his or her consent.
• If the patient consents :
  
  
  • Make sure he or she signs the consent form.
  • If patient is 18 or less, obtain a signature from a parent or a tutor.
  • Give the patient his or her copy as well as the pharmacist's copy.
  • Apply a MOXXI sticker on the patient's file to indicate that the patient is participating.
• If patient wishes to think about it:
  
  
  • Give him the consent form, make sure he has a brochure and ask him to bring the signed consent form at his next visit.

Patient recruitment began in June 2002. The physicians' goal was to recruit 700 patients or 70% of their regular practice within 6 months. Contrary to what was initially planned, all patients became eligible: it quickly became obvious that it would be too onerous a task for physicians to evaluate whether the patient lived within the West Island or to estimate the date of last visit. A single consent was necessary per clinic. Consent documents were delivered to 28 physicians in the West Island of Montreal starting on June 10, 2002 and the enrollment period ended on November 18, 2003.

A recruitment guide detailing the recruitment process and an FAQ on recruitment was given to physicians as well as a list of participating pharmacies to help patients identify their most commonly used pharmacy.

Two clinics asked for help in the recruitment process. In a clinic, MOXXI team members spend two days informing patients of the project. In another clinic, a team member contacted the physician's patient to invite them to sign the consent forms. These patients had been informed of the project before the consent forms were available. Help for the recruitment was offered to all other physicians but was refused.

A. Consent Revocation

Patients could choose to revoke their consent if they so wished. They could do so by signing a revocation form at their physician's office or by calling the MOXXI team if they preferred (phone number was printed on their consent forms as well as on the patient brochure). The team would send the patient a revocation form and cancel his consent in the database once that form had been received. On December 31st 2003, 7 patients had revoked their consent.

Physician and pharmacist could also revoke their consent. As previously mentioned, 3 physicians chose to revoke their consent. No pharmacist revoked their consent.

B. Patient data setup

When physicians consented to the project, they agreed that the RAMQ would extract all patients they had seen in the past 12 months prior to their consent date. This information was based on physicians' billing information. Thus for each physician we obtained a complete list of his patients, including their first name, last name and RAMQ number. This would be used to pre- populate the physicians' iPAQ database before they began using the MOXXI system, minimizing required data entry.

2.6.8 Baseline Characteristics

To reach our evaluation goals, much information was collected throughout the project. Baseline information included: information on study population (physicians, pharmacists, patients), information on their expectations about MOXXI and skill assessments. The following is the result of the baseline information gathered.

A. Predictors of patient recruitment

The anticipated benefits of the MOXXI system can only be experienced by patients who consented to participate. If a patient truly does not want to release their records or if they are not offered the opportunity to participate by the physician, then these "non-consenting" patients will be unable to benefit from the system. To date, a detailed investigation of the characteristics of patients who consent versus those who do not for projects that involve the use of information technology has not been conducted. Our objective was to evaluate what patient and physician characteristics predict giving consent.

The practice population for the participating community-based family physicians was defined using medical services claims and thus potentially eligible patients who could grant access to prescription information were identified. For patients within a physician's practice that did not provide consent for their physician to access their data, prescription and medical services information from administrative databases were available in a de-identified format.

Characteristics of physicians and eligible patients were retrieved from provincial administrative databases held by the Quebec Health Insurance Program (RAMQ). All databases are linked through an encrypted, unique health care number for each subject with a look-up table provided for patients who gave written consent to their physician. The validity of these databases has been previously established for healthcare research. Information retrieved from the patient demographic database included age (by 5 year group), sex, postal code, language preference and prescription insurance status. The medical services database provided the medical visits including the type, location (e.g. inpatient, emergency department, private), diagnosis, treating and referring physician, and date of all services provided on a fee-for-service basis (95% of all services provided in Quebec). The physicians' demographics were provided by RAMQ and included sex, language preference, location of graduating medical school, year of graduation, and specialty. Ease of use with the MOXXI system was measured using a timed standardized task that was performed 3 months after the physicians started using the system. For the task the physicians were required to write three new prescriptions using the electronic pad.

Frequency distributions of physician and patient characteristics were determined and the means, standard deviations (sd) and ranges were reported for continuous variables. Multivariate logistic generalized estimating equations (GEE) were used to investigate whether patient or physician characteristics increased the probability of consenting to participate. Patients were clustered within physicians with an exchangeable correlation structure. The unit of analysis was the patient with consent status (yes versus no) as the outcome of interest. Statistical analyses were conducted using SAS 8.02.

Consent documents were delivered to 28 physicians in the West Island of Montreal starting on June 10, 2002 and the enrollment period ended on November 18, 2003. Characteristics of the participating physicians are presented in Table 2.6f. Other locations where the physicians might practice included drop-in clinics, emergency rooms, hospital out-patient clinics and nursing homes. The physicians were mostly French-speaking and the majority graduated from a French medical school. On average, physicians took 2.0 minutes (sd = 1.0) to write 3 electronic prescriptions when timed for performance on the MOXXI system.

During the enrollment period, the study physicians saw a total of 82,329 unique patients. Patients that were seen only in locations other than the private office were excluded, and 73,276 patients were eligible to be enrolled in the MOXXI study. Table 2.6g presents the mean number of patients seen during the enrollment period, number days worked in eligible clinics and percent of patients that were consented. Enrollment rates varied greatly among the physicians, from 5% to almost 65%.

Table 2.6f - Characteristics of 28 study physicians

Characteristics	Frequency
Practice Settings Private clinic only Plus 1 other location Plus 2 other locations	14 (50.0%) 6 (21.4%) 8 (28.6%)
Female	13 (46.4%)
French	22 (78.5%)
Medical School University of Laval University of Montreal McGill University University of Sherbrooke North American Foreign	1 (3.5%) 18 (64.2%) 6 (21.4%) 1 (3.5%) 1 (3.5%) 1 (3.5%)
Year of Graduation 1970-79 1980-88 After 1989	12 (42.8%) 10 (35.7%) 6 (21.4%)

Table 2.6g -Practice characteristics for 28 physicians for the enrollment period

Characteristics	Mean (sd)	Range
No. of patients / day	35.0 (12.0)	12.4-61.2
Total days at clinic	486.5 (69.2)	288-523
Total unique patients	2617.0 (1063.1)	808-4923
% consented patients	20.5 (17.9)	4.7-65.2

Of the 73,276 eligible patients who were seen by the study physicians during the enrollment period, 21% consented to participate in the MOXXI project. Table 2.6h presents the patient characteristics of consenting and non-consenting patients. The top of Table 2.6h presents characteristics measured during the enrollment period. On average, patients who were consented saw their study physician almost three times as often compared to non-consenting patients (mean of 8.6 visit days versus 2.9). The bottom half of Table 2.6h summarizes patient health care utilization, prescription drug insurance status and continuity of care for the 6 months preceding the enrollment period. Co-morbidities were measured using the Charlson Comorbidity Index (CCI). Consenting patients tended to be older and more likely to have prescription drug insurance.

Physician variables included in the GEE regression included the time required to perform the standardized task; patient flow (number of patients seen per day); whether the physicians worked only in private practice or also in other settings; the language preference (this variable was highly correlated to school of graduation so only language was included in the model); year of graduation and sex. Age was not available for the physicians and year of graduation was thought to be a better marker of physician experience. Patient variables included sex; age, language preference, prescription insurance status, and number of visits to the study physicians during the enrollment period (opportunity for consent). In the six months proceeding the enrollment period, a count was made of the number of emergency room visits, the distinct specialists visited and the non-study family practitioners visited; as well as the CCI. Patients were assigned to the first study physicians they visited during the enrollment period. Table 2.6i presents the results for significant predictors from the GEE analyses.

Table 2.6h - Characteristics of 73 276 eligible patients

Characteristics	Consent n=12 203	No Consent n=61 073
Measured during enrollment
Female	7331 (60.0%)	33208 (54.3%)
Age: ≤ 18 years 19-43 years 44-63 years ≥ 64 years	515 (4.2%) 2767 (22.6%) 5214 (42.7%) 3707 (30.3%)	16965 (27.7%) 23273 (38.1%) 15142 (24.7%) 5693 (9.3%)
English	3697 (30.2%)	16381 (26.8%)
Income Quartiles ≤ $41,801 $41,801-$52,300 $52,301-$62,300 ≥ $62,300	2793 (22.8%) 2544 (20.8%) 3242 (26.5%) 3624 (29.6%)	17471(28.6%) 15517(25.4%) 14131(23.1%) 13954(22.8%)
Visits to study physicians 1 2 - 3 ≥ 4	458 (3.8%) 1686 (13.8%) 10059 (82.4%)	27118 (44.4%) 18629 (30.5%) 15326 (25.0%)
Measured in the 6 months preceding enrollment
Drug Insurance None Continuous Sporadic	6324 (51.8%) 4839 (39.7%) 1040 (8.5%)	37507 (61.4%) 18869 (30.9%) 4697 (7.8%)
Charlson Comorbidity Index > 0.5	4385 (35.9%)	14477 (23.7%)
No. of specialists visited: 0 1-2 ≥ 3	4898 (40.1%) 5163 (42.4%) 2142 (17.6%)	33228 (54.4%) 22345 (36.6%) 5500 (9.0%)
No. of non-study general practitioners visited 0 1-2 ≥ 3	8079 (66.2%) 3484 (28.6%) 640 (5.2%)	33374 (54.6%) 22628 (37.1%) 5071 (8.4%)
ER visits 0 1-2 ≥ 3	10986 (90.0%) 574 (4.7%) 643 (5.3%)	54662 (89.5%) 3713 (6.1%) 2698 (4.4%)

Table 2.6i - Multivariate GEE of predictors of consent adjusted for physician level

Predictors of Consent	GEE Estimate	Odds Ratio	95% CI
Patient-level characteristics
Female	0.13	1.14	1.09-1.20
Age: ≤18 years 19-43 44-63 ≥64 years	Reference 1.23 1.98 2.45	3.41 7.26 11.55	3.08-3.77 6.59-8.00 10.3-12.9
Income ≤ $41,801 $41,801-52,300 $52,301-62,300 ≥ $62,300	Reference 0.04 0.11 0.20	1.04 1.11 1.22	0.97-1.11 1.04-1.19 1.14-1.30
Visits to study GP: 1 2-3 ≥ 4	Reference 1.47 3.13	4.35 22.8	3.91-4.84 20.6-25.1
Drug Insurance	0.16	1.17	1.10-1.24
CCI >0.5	0.06	1.07	1.01-1.12
No. of specialists visited: 0 1-2 ≥ 3	Reference 0.05 0.16	1.05 1.18	1.00-1.11 1.09-1.27
No. of non-study GPs visited:
0 1-2 ≥ 3	Reference -0.37 -0.81	0.69 0.45	0.65-0.73 0.40-0.50
ER visits: 0 1-2 ≥ 3	Reference 0.10 0.19	1.11 1.21	0.99-1.24 1.07-1.37
Physician-level characteristics
English	0.44	1.55	1.44-1.67
Female	-0.56	0.57	0.54-0.61
Graduation 1970-79 1980-88 After 1989	Reference -0.17 -0.65	0.84 0.52	0.80-0.89 0.48-0.57
Patients/day	-0.04	0.96	0.95-0.96
> 1 Practice Setting	0.03	1.03	0.97-1.09
Standardized task time	-0.05	0.95	0.93-0.98

All physician variables significantly affected the probability of consenting to participate. For patients, only language preferences were not significant. For physicians, higher patient flow, speaking French, being female and standardized task performance decreased the likelihood that a patient would participate. Female patients were more likely to consent as well as older patients with continuous insurance and higher income levels. Patients that have seen more specialists with more ER visits are more likely to consent whereas patients that see more than one other non-study primary care physician are less likely to consent. The strongest predictor of consent was the number of times the patient saw the study physician.

We had a unique opportunity to compare physician characteristics with information from both participating and non-participating patients. We found that higher consent rates were obtained by more experienced physicians who saw fewer patients per day (patient flow) and who, paradoxically, were initially slower using the system. The slower times may have more to do with computer fluency and comfort for the older physicians. Patients that consented were more likely to be older women with higher income levels as well as prescription insurance and visits to more than one specialist. These findings appear to support the hypothesis that physicians enroll patients when there are value-added benefits, i.e. older or more complex patients that have more complete prescription drug information due to continuous provincial public insurance. The strongest predictor of consent was the number of visits the patient made to the study physicians. This emphasizes the role of opportunity for obtaining consent as well as the physicians feeling that they are the primary physician responsible for the care of the patient. For the patients who did not consent, we were not able to say whether the patient was approached by the physicians to obtain consent and declined, or whether the patient was never approached - so teasing out the individual mechanism behind a positive consent is difficult. However, the importance of the role of the physician in obtaining patient consent is obvious when the range of consent rates is examined by individual physicians (5 to 65%).

More research that includes qualitative interviews of physicians and patients is needed in order to formulate methods to maximize patient participation. For this study we only examined rates of consent as our outcome. The next step will be to assess the impact of consent rates on physician usage of the system. The interaction of consent rates also needs to be assessed for the anticipated benefits of the system - the reduction of adverse drug events.

Both physician and patient characteristics play an important role in determining the probability of consent to participate in an electronic prescribing project. The individual physician's experience seems to play a particularly important role as well as opportunity for consent. Physicians seem to preferentially enroll patients who would be the most likely to benefit (older, female patients) although the role of sociodemographics is also important.

B. Physician's expectation and usage of information technology, and their satisfaction with their current practice.

The Technology Acceptance (TAM) questionnaire was adapted and administered to the participating physicians following their MOXXI training session. The questionnaire asked physicians to rate their clinical satisfaction in addition to their expectations about the MOXXI system and their current usage of computers.

A majority of study physicians indicated a relatively high level of clinical satisfaction with their current practice including: quality of care, psychological and material rewards of practice, patient interactions, and social and intellectual work environment. The highest level of satisfaction rated by the study physicians (87.6%) was the intellectual aspect of their work (i.e. work is educationally stimulating). Furthermore, 75 % of study physicians indicated a satisfaction with the social environment of their work (i.e. relationship with pharmacists). 86.6% of study physicians were satisfied with a quality of care they were providing to patients, and 73.3% of study physicians indicated their satisfaction with patient interactions. Finally, study physicians indicated a slightly higher satisfaction with the material rewards of their current practice than with the psychological rewards (80% vs. 66.7%, respectively).

With regards to the physicians' expectation of the MOXXI system, the results of the TAM questionnaire showed that a majority of study physicians indicated a commitment to use the MOXXI system (90.4%) and had high expectations that MOXXI system would be useful (95.2%) and easy to use (68.4%). In addition, study physicians had high expectations that the introduction of the MOXXI system would bring in a beneficial impact to their current practice.

Although the majority of study physicians reported a relatively high level of computer selfefficacy (66.6%) and comfort using computers (66.6%)), their actual use of computers in their clinical practice was low. Specifically, study physicians reported a low likelihood of using a computer to communicate with colleagues (33.3%) or to obtain diagnostic or therapeutic advice (40%).

In summary, although study physicians reported a high level of satisfaction with their current practice, they also have high expectations for potential improvement, which can be addressed through the use of information technology. Physicians reported a commitment to use information technology, with the expectation that the technology should be easy to use and be useful. However, their relatively low level of current computer usage in clinical practice was seen as a potential barrier to manage the use of information technology during patients' visits.

This information highlighted the importance of providing consistent support for physicians' skill acquisition that is necessary to manage an information technology system, such as the MOXXIi system, in their practice.

C. Physician's acceptance of technology

To identify predictors of utilization of the MOXXI electronic drug management system, the participating physicians completed an adapted version of the Technology Acceptance Model (TAM) via their PDAs prior to using the system in their community practice. The TAM questionnaire includes 24 items, rated on a 5-point LiIkert scale, which examined user acceptance of computers, defined perceived usefulness and perceived ease of use as critical predictors of one's intention to use a computer and self-reported usage behavior.

Among the 30 physicians, 9 study physicians (30%) did not complete the questionnaire. A total of 21 physicians were included in the analysis. Of 21 physicians (63.3%), most (62%) were male. Approximately 43% had graduated from medical school between 1970 and 1979, 33% and 24% had graduated between 1980 and 1988 and after 1989, respectively (Table 2.6j).

Table 2.6j - Study Physicians' Characteristics

Characteristics		n	(%)
Sex	Male Female	13 8	61.9 38.1
Graduated Year	1970-1979 1980-1988 after 1989	9 7 5	42.9 33.3 23.8

Response scores for each question answered by study physicians were summed by TAM concepts (Table 2.6k). Most of physicians agreed to the three main TAM concepts that they would use the MOXXI system with most of their patients (90.5%), would be easy to use (76%), and would be useful (90.5%) (Table 2.6l). Pearson correlation coefficients were calculated between the degree of behavioral intention and other proposed TAM concepts (Table 2.6l). Perceived usefulness was moderately correlated with behavioral intention (r=0.43). All other TAM concepts, except clinical satisfaction (r=0.16), were also moderately correlated with behavioral intention. Clinical relative advantage showed the strongest correlation with behavioral intention among all the concepts (r=0.62).

Table 2.6k - Physicians' Rating of the MOXXI System:TAM Questionnaire Results

TAM Concepts	Operationalization of Concept	Mean (SD) [Range]
Behavior Intention (BI)	The degree to which a person intends to perform a certain behavior	4.5 (0.8) [2 to 5]
Perceived Ease of Use (PEU)	The degree to which a person believes that using a particular system would be free of effort	3.9 (0.9) [2 to 5]
Perceived Usefulness	The degree to which a person believes that using a particular system would enhance his/her job performance	4.5 (0.7) [3 to 5]
Clinical Relative Advantage	The degree to which an innovation is perceived as being better than its precursor	33.8 (4.8) [27 to 40]
Social Influence	Perceived support by management and peers to use the system, desire to please management and peers by using the system	4.2 (0.7) [3 to 5]
Computer Self-Efficacy	A person's beliefs about her/his ability to perform a specific tasks/ jobs using a computer	7.8 (1.7) [5 to 10]
Clinical Computer Utilization	The degree to which a person is comfortable using computer	16.5 (4.7) [5 to 25]
Clinical Satisfaction	The presence of four satisfactions facets: quality of care, psychological and material rewards of practice, patient interactions, and social and intellectual work environment	25 (3.4) [17 to 30]

Table 2.6l - Pearson Correlation Coefficient between Behavioral Intention and Other TAM

Behavioral Intention (n=21)	r (p-value)
vs Perceived Ease of Use	0.28 (0.22)
vs Perceived Usefulness	0.43 (0.05)
vs Clinical Relative Advantage	0.62 (0.01)
vs Social Influence	0.38 (0.09)
vs Computer Self-Efficacy	0.47 (0.03)
vs Computer Utilization	-0.03 (0.90)
vs Clinical Satisfaction	0.16 (0.49)

D. Pharmacist satisfaction with the quality of service offered for patients

Pharmacists from the 33 participating pharmacies were asked to complete a baseline questionnaire about their work activity before the start of the intervention. Sixty four of the 85 pharmacists at the participating pharmacies completed the questionnaires (75% response rate). Of the 64 pharmacist respondents, 45% agreed or strongly agreed with the statement "I spend too much time addressing problems related to prescriptions (i.e. incomplete prescription, illegible prescription) and 73% stated they would like to devote more time to counselling patients about prescriptions.

Pharmacists indicated that on average they spend 1.63 hours (sd=1.8) of their week communicating with physicians to address technical prescription problems such as incomplete prescriptions, renewals, or non-insured medication and 1.16 hours per week (sd=1.41) communicating with physicians to address therapeutic prescription problems such as interactions and duplications.

Answers for 64 out 85 questionnaires sent to participating pharmacists (75% response rate). The distribution of their work activity is shown in Figure 2.2. Ten percent of pharmacists' time is spent calling physicians regarding prescription legibility or prescription problems.

Figure 2.2 - Distribution of Work Activity of Montreal Pharmacists Baseline Questionnaire Reports of Work Activity by Category - 64 Pharmacists (75% Response Rate)

Pharmacists were also asked to log their calls on 4 weekdays for 2 consecutive weeks. The percentage of calls made by pharmacists and doctors and the average number of minutes spent on calls received by pharmacists from physicians are illustrated in Figure 2.3 & 2.4 respectively. There were 20 calls per 1,000 prescriptions served that were made or received by pharmacists (Figure 2.3). The main reasons for the calls were to renew existing prescriptions and to obtain new prescription (67%). Overall, the calls averaged 2.86 minutes per call (Figure 2.3), while calls placed by physicians to pharmacist for interactions averaged 3.0 minutes (Figure 2.4).

Figure 2.3 - Call Log for Pharmacists for Tuesday to Friday for Two Consecutive Weeks

Percentage of calls by reason and origin

Click on the image to enlarge or here

Summary statistics:

• an average of 256.17 prescriptions served per day per pharmacy
• 20.07 calls made or received per 1,000 prescriptions served
• an average of 2.86 minutes per call
• 57.60 minutes spent on calls per 1,000 prescriptions served

© G. Bartlett, MOXXI-III, 2003-01-14

Figure 2.4 - Call Log for Pharmacists for Tuesday to Friday for Two Consecutive Weeks

Average number of minutes spent on calls made by pharmacists by reason
(Reason, Minutes)

Click on the image to enlarge or here

2.6.9 Project Management Committee and Production Meetings

To ensure proper follow-up of project's activities, a committee composed of the project's principal investigator, the project's manager, the physician coordinator, the pharmacist coordinator and the evaluation coordinator was formed. Its role was to ensure the coordination of the efforts so that project's goals would be reached. This committee met regularly throughout the project. When necessary, additional team members would be invited so that matters could be clarified and dealt with.

Starting in January 2003, we formally extended the project management committee to include the technical staff (database administrators, developers) as well as all researchers and PhD students working on the MOXXI project, as their work benefited from a better understanding of the day-to-day occurrences happening in the project. This also ensured a better linkage between the researchers and the technical development team. It is one thing to develop a system that answers user requirements, but another to develop a system that will also collect data that is easy to analyze for researchers. We also created a 'production meeting' between the technical team and the project manager to ensure sufficient time to discuss technical matters that would arise and would have impacts on the timeline and the results but which would not be of general interest to the rest of the team. Matters that required immediate attention were thus addressed more quickly.

2.7 Rollout

Table 2.7 - Task 7: Rollout

Task #	Task Name	Sub-tasks	Description
7	Rollout	Site Implementation	Install telecommunications where needed. Support physicians in the change process. Two phase implementation.
		Site Testing	Ensure quality of service at point of care
		Training	Train end-user and support staff

2.7.1 Site Implementation

There was very little to do in terms of site implementation in the physicians' clinics. Printers were installed in all clinics. We identified each printer with a sticker and its serial number on it, as it would later facilitate the linkage of the iPAQ to the printer via Bluetooth. Pre-cut prescription paper was brought to the clinics, as well as a renewed supply of consent forms and patient's brochures.

One task was more time consuming than any others, and this was the retrieval of the consent forms from the physicians' offices. As we were late in implementing the electronic prescription system, physicians had consented hundreds of patients and that information had to be entered in the system so as to populate their database. We feared that it would seem like an enormous amount of data entry to begin with and it may discourage new users as it would represent additional work. Prior to the physician's training session, members of the MOXXI team went to the clinics and retrieved the patient's consent forms from their files (identified with MOXXI stickers). They were brought back to the office and entered in the database. This also allowed the team to make a copy of the consent forms, as the CAI had requested that we kept a copy of all consent forms after the forms were printed in 3 copies.

For the pharmacies, implementation of the electronic prescribing system took none of their time: their service providers updated the interface and gave them instructions on how to retrieve electronic prescriptions from the MOXXI server and how to send their patients' pharmacological profile to the server at the end of the day. Shared care pharmacists would receive all materials necessary to their task during the training session.

2.7.2 Site Testing

A. Physicians' Clinics

As stated previously, we had each site tested by an independent firm to determine whether connectivity was adequate or not. We re-tested the connectivity in each clinic when we implemented the system with physicians. It was important to see how the system would perform under real circumstances.

Connectivity was an issue at certain clinics. More specifically, some clinics were at the edge of the wireless 'corridor' and connectivity was poor. The fact that certain clinics were in basements caused some problems. Newer buildings with more recent materials were prone to problems, as were clinics that had no windows and that used metallic blinds. Since Bell has the most extensive network in Quebec and is an extremely solid and reliable service provider we do not believe this to be a problem that would have been solved by using an alternative service provider. Collaborating closely with Bell Mobility ensured that this problem was minimized: the antenna's orientation was adjusted to optimize connectivity within participating clinics. These problems will disappear as technology evolves, but they were a major barrier for the project.

B. Pharmacies

Because of the project's lateness, we were gathering the pharmacists' consent from physicians' offices and were planning on bringing them to each participant to they could be inputted in their system once their new interface was in place. We wished to avoid an accumulation of consent forms at pharmacies until they could do something with them. Gathering the consents and redistributing them was difficult. Patients are, we now know, notoriously poor at identifying their usual pharmacy, and ordering the consents per pharmacy became a difficult task. Also, many patients did not bother to fill out information about their preferred pharmacy, or were unintelligible. Thus many consent forms sported only their preferred banner's name, no name at all, an erroneous name or an unreadable name, which complicated distribution of consent forms. We offered help to all pharmacists with consent entry, but few required help: pharmacists know their patients and the data entry was not too cumbersome for most of them. At this time physicians are encouraged to give the yellow sheet to their patient so he / she can bring the consent form directly to their pharmacist, thus avoiding this whole process.

When the electronic prescription software interface was implemented in pharmacies, our physician/pharmacist coordinator went to pharmacies to test the software's interface and test the communication link and integration using mock-up prescriptions. This allowed us to find out problems that had not been thought of back when the transmission protocol was designed.

First, some software providers had slightly customized software versions for some of their pharmacist, which we were not aware of until we tested the interfaces. This required the provider to adjust his software to allow his pharmacist to retrieve prescriptions. Second, we found out that pharmacies' software does not contain a complete list of DIN numbers. In fact, pharmacy software provider's list of medication is comprised of: the RAMQ list of insured medications, the list of medications recommended by the AQPP and tailored pharmacist preferences. We used the list of medications included in the Vigilance Santé's software. If a product has a DIN number, it is in the software. This raised an issue: physicians could prescribe a product that the pharmacist's software would not recognize. This had not been planned and required modification to our communication protocol and to the pharmacies' interface. Third, we found out that the communication between the pharmacist's system and the MOXXI server was not as quick as we thought it would be. Retrieving a prescription could take up to a minute, and sending the pharmacological profile of all consenting patients was an operation taking up to 15 minutes, best done at the end of the day. Pharmacists with whom we did the tests did not seem to mind this performance so much, feeling that the benefits were well worth the wait. One vendor's software worked perfectly when within his lab, but did not work on the pharmacist's computer. The problem seemed to be a lack of memory, as the pharmacist's computer was approximately 10 years old. A computer was purchased and installed, and series of tests need to be rerun. These problems are currently being ironed out. They are exceptions but need to be dealt with as we cannot afford the pharmacist's system to fail because of an error in either our software or their provider's software.

2.7.3 Training

A. For Physicians: Familiarization with iPAQs

We gave the physicians their iPAQs in the spring of 2002 to allow them to become comfortable the tool prior to usage of the MOXXI prescription system. For those who so desired, we offered optional 3 hour training sessions, given by a representative of Compaq Canada. Training sessions were planned both in French and in English, and took place in the West Island to minimize transportation for physicians. Overall, 19 physicians out of 28 went to these training sessions. The Compaq user guide on the iPAQ was distributed, and was available both in French and in English.

B. Consent Entry Prior to Training on the MOXXI System

Initially, we had planned that patient recruitment would last the first 6 months of the project and would occur in parallel with physicians using the MOXXI system, which would have meant that physicians entered their patients' consents themselves. As this was not the case, we realized, prior to the first training session in December, that if physicians had to manually enter the accumulated consent forms they had gathered so far, they probably would not do it. We gathered all patients' consents in their clinics.

C. Training on the MOXXI System for Physicians

The first training session occurred during on December 12th, 2002. The following sessions were spread out over the months of January, February and March. We depended upon the physician's availability and capacity to closely follow up right after the initial training session, thus sessions could not be too close to one another, else it would not have been possible to give the expected level of support to physicians in the days following the training. Training sessions were highly tailored to each clinic. We usually tried to group the physicians in small groups of 4 to 7 people, but it was not always possible. What was more feasible was organizing training sessions per clinic, because physicians within a clinic were well aware of each other's schedule and could more easily get organised. Thus we did give training sessions to one or two physicians at a time. The trainer could adapt to the physician's need in a way that's impossible when teaching larger groups.

Most training sessions occurred at night unless otherwise requested by physicians. They lasted 3 hours, during which case studies were presented to illustrate the functions and the flow of the system, issues were discussed, questions were answered. Physicians used test iPAQs to do the case studies along with the trainer, practicing with the system. At the end of the training session, physicians were administered a standardized task during which the trainer could not help them. This last exercise was extremely useful, both for trainer and physician, as it pointed out where physician might subsequently need help. In turn, it seemed to help physician integrate the system more quickly. The trainer immediately scheduled his visit to the physician's office, as well as the date for the next standardized task. This on-site presence acted as an incentive for physicians to use the system rather than going back to the prescription pad.

D. Shared Care Training for Pharmacists

Training for pharmacists was held in the West Island as well. Trainers were pharmacists from the University of Montreal, accompanied by the pharmacist coordinator. Pharmacists were invited to a training session of an approximate duration of 2 hours. The small group format was preferred. 6 training sessions were organized, and nearly all pharmacists were trained. (4 pharmacists cancelled repeatedly as they met with last minute emergencies). The pharmacists were compensated for their time. The first hour consisted of a presentation of the tools, and the second half of the session was comprised of practical exercises. Following the shared care training sessions, a notebook including pictures, coordinates and specialties of all shared care pharmacists was distributed to physicians. The goal was to allow patient to identify his pharmacist more easily, but also to establish a closer contact between physicians and pharmacists.

2.8 Intervention

Table 2.8 - Task 8: Intervention

Task #	Task Name	Sub-tasks	Description
8	Intervention	Observation	Help desk logs, patient call centre activity, TAGGE usage logs, RAMQ data transmission logs, security audits
		Process Evaluation	User (patients, physicians, pharmacists) satisfaction surveys, usage logs, unscheduled down time, security audit reviews, random sample data quality, ongoing data analysis.
		Support	Help desk activity, patient call centre activities, website hits

2.8.1 Observation and Process Evaluation

Information was systematically gathered throughout the project to meet evaluation goals. We kept track of physicians' and pharmacists' requests, effectively setting up a help desk log. Data gathered is described in details in the evaluation report TAGGE logs. Usage logs are kept as part of the data gathered from physician's and pharmacist's systems. System operation logs were kept to track system down time. Due to the highly complex infrastructure of the MOXXI system which uses multiple service providers (such as Vigilance Sante, RAMQ, McGill University Health Centre) to enable the various components of the project, it was difficult to control and evaluate the source of unscheduled downtime. However, the system operation log, did track unscheduled downtimes for the local MOXXI server. Reliability was thus, evaluated based on the information extracted from this log.

Qualitative information about physician satisfaction was collected each time the physician's coordinator visited the clinics. A suggestion list from physicians was compiled and reviewed regularly; suggestions that were requested by many users and were technologically feasible without impacting our timeline significantly were implemented, improving the tool. MOXXInfo was used to provide the participants with feedback. In June 2003, all study physicians and pharmacists were invited to a cocktail, where they were provided with preliminary project results and where additional feedback was obtained.

2.8.2 Support

Throughout the project, the physician and pharmacist coordinators would regularly visit all sites to ensure communication, bring material and support users. Initially, support had been planned 'on demand', but this changed after we trained physicians and pharmacists.

To ensure that physicians used the computer-based prescription system, we provided intensive initial training, regular follow-up of physicians, and weekly usage monitoring through the central server to identify potential problems and follow-up with physicians who may be experiencing difficulty. Daily monitoring of prescription activity through the central server allowed the management team to quickly identify physicians with low levels of activity, and to provide onsite trouble-shooting and support. The same approach to monitoring compliance will continue throughout the follow-up period.

In our experience, we have found that it is most valuable to be out in the field providing support on a regular basis than waiting for physicians or pharmacists to telephone for assistance. This type of support was especially important since the rate of computer usage in clinical practice for the participating physicians, which we assessed through a baseline questionnaire, was low prior to the implementation of the MOXXI system. The face-to-face contact has been vital to identify technical problems, to obtain feedback about ways to improve the system, to keep the physicians abreast about the ongoing updates with the system and components to the project, and to provide verbal encouragement to continue to utilize the system. It appears that most participating physicians prefer meeting with the support staff in between their patients' visits rather than picking up the phone to resolve technical problems. At least one of our support staff of two persons is out in the field approximately 4 days per week with the goal of visiting all participating physicians every two weeks. With this type and frequency of support, physicians increased their rate of writing prescriptions using the drug management system from a steady rate of 17-25 electronic prescriptions per 100 visits (over 24 weeks) to 34 per 100 visits.

2.9 Follow-Up Assessment

Table 2.9 - Task 9: Follow Up Assessment

Task #	Task Name	Sub-tasks	Description
9	Follow-up Assessment	Data Collection	Extract data downloads from RAMQ, Tagge
		Data Analysis	Statistical modeling, analyses & reports
		Evaluation of Acceptability & Technical Performance	User surveys (patients, physicians, pharmacists, TAGGE, implementation & support teams)

2.9.1 Data Collection

Data collected came from the MOXXI server (consenting patient's prescription, log files, alerts), RAMQ, questionnaires participants have filled out, standardized tasks, and the time and motion study. These data were used to conduct the reported analyses outlined in the scientific evaluation.

2.10 Dissemination and Future Directions

Table 2.10 - Task 10: Dissemination and Future Directions

Task #	Task Name	Sub-tasks	Description
10	Dissemination & Future Directions	Scientific Evaluation	Analyses of intervention results (Use of baseline data & research outcome measures)
		Scalability and Sustainability Evaluation	Technical review committee in conjunction with the IRIS project
		International & Provincial Project Review	International task force combined with COMPETE and integrated with the IRIS future directions
		Lessons Learned and Suggested Modifications	User (patients, physicians, pharmacists, TAGGE, implementation & support teams) focus groups
		Project Administrative Reports to CHIPP	Reports submitted in a timely fashion according to Health Canada guidelines
		Financial Statement Reports to CHIPP	Reports submitted in a timely fashion according to Health Canada guidelines, audit reports
		Project Final Evaluation Report	Reports submitted according to Health Canada guidelines
		Communication of Results	Article submission to scientific journals, presentations (media, other), symposia, conferences, invited lectures, website.

2.10.1 Scientific Evaluation

Scientific evaluation was hindered by the delays and complications that occurred throughout the project. For example, to evaluate the prevalence of inappropriate prescriptions and the rate of aborted alerts, information from the MOXXI database was required. When extracting this data, it became apparent that both documentation of the fields in the database and pre-established logs of data required for evaluation are imperative to facilitate the extraction and analysis of the information provided by the database. Since these elements were not completely in place, data retrieval was onerous though attainable. Due in part to this experience, we have developed a strategy for documentation and the research team has prepared pre-established data requirements to provide to the IT team for development of the new infrastructure.

Baseline information was used to evaluate physicians' opinions regarding information technology and their current practices and to obtain pharmacists' satisfaction of their services. Follow-up questionnaires for physicians and pharmacists are scheduled to be administered in the near future so that post-implementation opinions can be assessed in comparison with baseline information. Changes in the access to health service information was evaluated with regards to overall treatment indications, indications for benzodiazepines in the elderly, and the relationship of access to information in the pharmaceutical profile and socio-economic status of the patients.

Evidence of improved linkages in community care was evaluated based on physician usage of the drug profiler (with compliance indicator), and percentage of patients eligible for referral to the shared care program (compliance support). Referral to the shared care compliance support service has only recently been available to the participating physicians. Comparative baseline data for compliance in the target population has been gathered and will be used for future evaluation in the post-implementation phase.

Health related impacts, such as emergency visits and hospitalizations, as a result of the ICT basic decision support for drug management, were too premature to evaluate at this stage in the project (one year post implementation). With regard to the advanced decision support, both the asthma and diabetes decision support components underwent thorough assessments by clinical and IT professionals after alpha versions of the systems were developed. Feedback from these assessments resulted in a complete re-development of the asthma decision support and a conclusion to not implement the diabetes decision support because of its complexities and the limitations of current technology. These necessary evaluative steps towards implementation of the advanced decision support components also resulted in delays for actual implementation. Therefore, the impact of the advance decision support on Asthma and Diabetes and compliance support services could not be evaluated. However, baseline comparative data has been collected for asthma control (% of patients taking > 500 doses of fast acting beta2-agonists in 6 months) and for emergency visits due to respiratory problems.

Quantitatively, cost-effectiveness can only be evaluated once the effectiveness of an IT intervention is shown and improvement in outcome is assessed. This process requires 3-4 years to complete. Since this time window was not provided in the planning of the CHIPP program, cost-effectiveness was examined qualitatively. That is, through a description of the project's components which essentially permit using fewer steps to provide quality service by communitybased physicians and through examples provided by physicians who are currently using the system. In the future, we will determine the average costs of the supply of target drugs dispensed for each patient.

2.10.2 Scalability and Sustainability

Ensuring that MOXXI was scalable and sustainable was of course a major concern. This was discussed by the international advisory committee; a study was mandated to the RAMQ to estimate the changes required to support up to 5 000 physicians and their patients. The current data transfer model we had with RAMQ was designed for a small number of physicians and it would not be possible to expand it much were we to have many more users. After 10 years of developmental work, culminating in the third phase of this project funded by the Canadian Health Infostructure Partnership Program, it was evident that the innovations developed in this decade of research experience could not be transferred or generalized on a broader scale without a business team. As a result, McGill University created a spin-off company, MOXXI Medical Inc., in the summer of 2003 and obtained funding from MSBI, a seed venture capital company that is co-owned by McGill University, the University of Sherbrooke, and Bishop's University.

2.10.3 International and Provincial Project Review

The international advisory committee met in February 2003. The attendees were Dr Robyn Tamblyn, principal investigator; Mr. W. Dale Dauphinee, Executive Director of the Medical Council of Canada; Dr. Mamoru Watanabe of Calgary University, Dr Robert Galvin of General Electric, Dr. Yves Lussier, Dr. David W. Bates of Brigham & Women's Hospital, Dr. Richard Dick, Dr. David Flaherty from Privacy and Information Policy Consultants, Dr. Allen Huang, investigator, Mr. Gerard Douville of IRIS Quebec, Mrs. Mélodie Faucher, MOXXI Project Manager, Dr. Gillian Bartlett, MOXXI evaluation coordinator, Dr. Robert Perreault, Mr. Denis Tremblay, of IRIS Quebec, Mrs. Nadyne Girard, analyst, and Mr. Matthew Mather. The objective of this committee was to ensure that the implementation and the process is congruent with other health and social policies, as well as the local professional and public environment. MOXXI's future was discussed, as were its integration with IRIS Quebec and the importance and strategies to adopt for data security. The international committee viewed the full integration of MOXXI with the drug insurance program and pharmacies as the clear business advantage that could bring value-added benefits to payers, professionals and patients.

2.10.4 Project Reports

Project reports, including the project final report, the evaluation report and the financial audits, were not handed to CHIPP on time. Delays were incurred because baseline RAMQ data necessary for evaluation was weeks late. The departure of the project's manager on maternity leave in October left an information void, which was eventually filled but created some delays.

There were also some misunderstanding regarding the financial audit that was to take place, as CHIPP had given verbal authorization for the project to use McGill's internal auditors, who were misconstrued as being accredited auditors. The evaluation and final reports were sent to CHIPP at the end of the month of January. At that time, the financial report had not been sent yet.

2.10.5 Communication Plan

The MOXXI project has generated interest both in the scientific world and in the community. Articles have been accepted for presentation at the international, U.S and Canadian health infomatics conferences. Numerous other presentations and public media coverage have occurred (see section 6.0. Communications).

The joint committee formed by the joint committee of the Collège des médecins and l'Ordre des pharmaciens requested a presentation from the project and the business team following the adoption of the law 90. We are also meeting the Conseil des medicaments during the month of January 2003.

In addition, MOXXI has been approached by the media on many occasions, and articles have been published about the project in the Time, the West Island Chronicle, Medical Journal. The project won the North American 3G CDMA A-List Awards and has been submitted to the OCTAS, which rewards innovative IT projects in Quebec. Direction Informatique has approached us about the project, as well as the television program La revanche des nerdz, an IT program for the general public.

The website of the project is regularly updated to keep participants aware of new development and inform the public about this innovative project.

2.11 Lessons Learned

2.11.1 Technological Performance

Physicians and pharmacists expected sub-second response times from applications they use. If the applications do not deliver the expected level of performance, it will be discarded in busy moments. The fact that physicians have consistently used the MOXXI system for the past year even though the PDA does not deliver this level of performance is a testimony to the fact that the information MOXXI delivers is truly precious to physicians. PDAs have not reached the level where they can in fact compete with computers in terms of speed and processing power. Our model required a thick client (meaning that a lot of data is residing on the physician's PDA) and we had to go to great lengths to ensure an acceptable level of performance. As PDAs get more powerful, performance will increase. We face the same challenge when using the wireless network: synchronization requires a lot of information to be transferred. As technology evolves (Bell already has a faster, more powerful network solution available), performance degradation produced by the telecommunication network should improve.

2.11.2 Partnerships

One of the benefits of having many partners involved in our project was that we derived benefit from a great wealth of knowledge, different perspectives and an influential network of people from different organizations. In fact, this project would not have been possible without the collaboration and cooperation of our multiple partners. There is a very challenging aspect to having multiple partnerships: it increases the complexity level associated to the project, not only in the decision making process but also the technological aspects of it.

We learned another lesson from our involvement with so many partners. It's important to include the potential players from the beginning, the beginning being the request for funding. This will not only give a chance for the participants to voice their opinions and concerns early, it will also allow them to include activities related to the project in their own timeline.

Potential partners may be repelled if they feel commercial interests are involved in the research project. Thus it's important to maintain the project's image as a university-based research project because people are usually well-disposed towards these interest-free projects.

Establishing trusting relationships with the participants is crucial and can be extremely challenging and time-consuming. Pre-existing relationships will shorten the time necessary to sign contract or obtain support. It is essential to plan sufficient time to establish the required partnerships. Also, being aware of the political context in which the partners are evolving and being aware of prior experiences in important. We blundered and created barriers with pharmacists because we initially were not aware of political factors between different professional and private organizations, which were then difficult to overcome.

2.11.3 Technology Obsolescence

During the project, Microsoft stopped supporting eVB (a stripped down version of Visual Basic) as a programming language. All subsequent operating systems would from then on support dot NET applications instead. Also, Hewlett Packard stopped building the PDAs we were using and favored newer models running PocketPC 2003 as an operating system. This meant that in case of equipment failure, it was impossible to change the physician's iPAQs as the operating system on newer PDA did not support the programming language we used. Simply put, the application would not run. It makes it important to have some IT expertise available to the project: Microsoft's move was relatively predictable in the IT world. Also, it demonstrated that the choice of the programming language should not only be left to the programmer. In our case, the initial programmer knew Visual Basic, and we did not foresee the problems in letting him use it. Rather, the choice should be based on best business practices. Finally, for sustainability purposes, it's highly relevant to have an application that can be migrated to newer machines.

Planning the implementation of an information system is difficult at the best of times. An IT project's complexity and length are systematically underestimated in every industry. Planning is even more difficult to do realistically without knowing the capabilities of the resource people. An experienced programmer who has worked on a similar project and who is using familiar tools will take less time in development than if he's using tools he's not proficient with. Experienced IT personnel do the best estimates of the time it will take to do the work. Plans should take into consideration environments that are particularly difficult (no debugging tools available, limitations to what's possible due to programming language limits, etc.).

2.11.4 System Analysis and Technical Requirements

Initial in-depth system analysis very early on in the project is essential. Having resources committed to these activities at the beginning of the project is likely to save time and money further on. This was an extremely ambitious project, having a tight timeline. Looking back, more time could have been spent on initial system analysis and technical specifications and documentation. Many modifications were made to the system and the databases after initial implementation. There is a balance to reach between initial planning, analysis and prototyping and actual implementation, as it is easy to spend a lot of time planning and analyzing and not producing anything useful.

Our experience with MOXXI II was invaluable, but the project evolved tremendously and the workload increased correspondingly, although the available resources did not.

2.11.5 Performance of Technologies

It is difficult and time-consuming to test performances of available technologies to simulate future system performance. In a context where sub-second response time is expected from physicians and pharmacists both, it may well be worth it though. Doing in-depth evaluation of technological capacities (simulating a complete database on an iPAQ and testing performance, simulating prescription transfer and database synchronization from the iPAQ to the MOXXI server, etc...) may help the project avoid performance pitfalls and point to high-performance technologies early on.

2.11.6 Prototype and Feedback on Interventions Prior to Implementation

Our experience with the initial asthma and diabetes algorithm suggested that a worthwhile strategy would be to test the intervention and the ideas on paper first, especially in highly experimental contexts. At the very least, it should be possible to have the prototype evaluated by potential users early on. If the user interface does not correspond to the workflow, this can then be identified prior to development. Designing a tool on paper prior to developing the system which users can try and comment on to see if it effectively supports their decision-making process, may help manage risk more efficiently. Prototypes should not be overly simplistic, they have to be representative of what is going to be delivered.

2.11.7 User Involvement

Prototypes reviews should not be limited to project team members: users should also test early beta versions. Following the same principle that the programmer does not test his own code, the team member and designer of the system should not be the ones evaluating the user-friendliness of the system, as they may not possess the same characteristics as users. This may be difficult, as physicians and pharmacists are busy users and may not have much time to assess systems.

2.11.8 The Difficulties of Selecting Vendors

It is not easy to select vendors who will deliver what is required, especially in a very experimental context. Chances are, no vendor will have prior experience in your exact field. A strategy we underused because of timelines was to use RFP to get the best vendor possible. Exploring possibilities is important - taking time to do this at the beginning may well save time in the end. If possible, ensure a service provider's competencies by giving them an initial contract for a part of the work and make sure they deliver in a timely and competent manner. Positive past experience with service providers is important but can't be used as the only reference: they may not have the required skill set required by the new tasks. Ensure that the service provider can guarantee a level of support that is sufficient (do not underestimate the needed support and input from participants involved in the development). Close geographical location helps - distance can become a hindrance and a problem as the vendor may not wish to spend money on travel. Take language and culture into account when choosing the provider. Include the terms of support in the contract.

2.11.9 Managing Contracts

Managing contracts can be difficult for a research project: companies may participate because it's a research project, but also the project may not be a business priority, or may lose its priority if it's late. Thus companies often carry out the requirements for a project as a favor and it becomes difficult to guarantee delivery dates.

2.11.10 Qualified Technical Personnel Recruitment

Whenever possible, ensure technical personnel are evaluated by an experienced peer before being hired. Our experience is that hiring a system's analyst only based on CV and on references may increase the risk factor, as it's difficult for human resource personnel to evaluate technical competencies. Giving the person a small but representative task at first and establishing a probation period may help minimize the risk.

Ideally, it is best not to depend on one single person to develop applications. We depended on a single individual for the integration of the pharmacies, the development of the MOXXI application, the maintenance of the application and the network, support for installation in the pharmacies, development of managerial applications (shared care reporter, consent helper system, TAGGE follow up interface). When the initial developer failed, we sub-contracted some tasks, with unfortunate results. Our new internal developer became responsible for them and competently fulfilled the requirements, but doing things this way set us back in our timeline as a single person was responsible for many key requirements.

2.11.11 Personnel Turnover

During the course of the project, the developer was replaced, the person assigned to the physician coordinator position changed 3 times, the evaluation coordinator changed, the pharmacist coordinator chose to go back to commercial ventures, and the project manager left on maternity leave during the last months of the project. Trying to minimize turnover is important, as some knowledge is always lost when people leave, no matter how well their job is documented. Also, it takes time for the new people to learn the ropes, particularly in a complex project.

2.11.12 Research Versus IT - Bridging the Communication Gap

Ensuring linkage between IT objectives (delivering a system corresponding to user requirements) and research objectives (gathering data to answer research question) may seem obvious but is challenging. Lack of a tight enough linkage between research objectives and IT goals increases the risk producing a system that answers user's requirements but not researcher's requirements. We hit that snag along the way; fortunately, it was early enough so that we could modify the application to permit the proper data to be gathered. It is important to be certain that IT team is highly aware of the researcher's detailed goals, as these evaluation goals are in part used to determine system requirements.

2.11.13 Project Management, Administration and Resources

The MOXXI project represents a highly complex undertaking, as it involves the management of a matrix of service providers, disparate jurisdictions, and knowledge sets. In addition to designing, developing and implementing an electronic prescription and integrated drug management system, advanced decision support systems and compliance support systems, the underlying research questions had to be answered. A major lesson learned is that a team of project managers is required to handle the complexities of the project. Awareness of the Zachman framework for system development is helpful. Some of the highlights of problems that will recur in similar complex projects include:

• Change requests need to be carefully managed and tracked
• Need to devote more effort to software and system design documentation
• Clinical informatics expertise is rare
• A mechanism to ensure input in disparate, poorly connected community-based care environment has not been established.

2.11.14 Multiple Partners, Multiple Environments

The MOXXI project provided a wonderful opportunity to study in-depth the working physicians' and pharmacists' physical environments and process constraints. One of the major challenges was to design solutions that work in very different operating environments. The pharmacists used a variety of software systems, older computer operating systems and PCs. Internet connectivity was also of variable speed, quality and security. The concept of 'one size DOES NOT fit all' was continually validated and creative work-arounds became routine. The independent physician offices, group clinics and pharmacies are a much more challenging environment to implement IT projects than the hospital centers, where corporate or institutional policies can be more easily controlled and enforced.

2.11.15 Sustainability

One important factor that improves future sustainability and continued uptake of IT in health care is to show continued small incremental successes. Managing the users' expectations of the new application is essential. The addition of new features or functionalities should improve response times and existing functions. End user evaluation is the gold standard determinant of useability. A solid business case had to be developed to ensure the financial sustainability of the MOXXI project. To this end a new for profit company MOXXI Medical Inc was created and a professionally generated business case finalized.

2.12 Conclusions

2.12.1 Responsiveness to Needs

The MOXXI project definitively gained a wealth of 'in your face', 'under your feet' experiences with health IT system implementation and support. This information can be reused to hopefully avoid the pitfalls in the continuation of this project and other similar complex health IT projects. The data can also provide research material to our colleagues in the Faculty of Management for continuing study of process re-engineering in health care when IT is introduced.

2.12.2 Further Development

The following list represents future activities that will further enhance and evolve the MOXXI project:

1. The MOXXI system will be re-designed and re-developed in a platform-independent environment in preparation for commercialization.
2. The creative solutions arrived at trying to retrofit modern applications and systems into outdated systems (eg Win95, DOS environments) need to be documented as a service and installation knowledge base for future reference .
3. Continued development of health IT expertise through university training programs. Need to lobby Canadian & provincial ministries to provide sustainable funding for new university training programs in health informatics to provide competent personnel in health informatics.
4. Continue research into the fields of knowledge transfer and translation.
5. Continue research into technology uptake into health care: determinants of barriers and successes.
6. Security, confidentiality and privacy services should be provided through a trusted third party, similar to the e-commerce use of electronic certificate authority services (eg. Verisign).
7. Form a 'users group' with membership from the end users. Ongoing information through a newsletter will keep users up to date with developments, problems, fixes, etc. A representative from that group should be involved in selected management meetings to provide direct feedback into decision points.

3. Products and Documents Generated

Document or Product Name	Available in Paper and/or Electronic Form	Licence Fee Required for use (Yes/No)	Previously Provided to Health Canada (Yes/No)	Appendix Name/Number
Recruitment material
Recruitment letter for physician (Lettre de recrutement des médecins)	Both	No	YES
Power point used for information sessions) Diaporamas utilisés lors du recrutement des médecins	Both	No	YES
Invitations to information sessions (Invitations aux séances d'information et indications)	Both	No	YES
Physician's brochures (Brochures médecins)	Paper	No	YES
Patient Brochures (Brochures patients)	Paper	No	YES
Pharmacist's brochures (Brochures pharmaciens)	Paper	No	YES
Formulaires de consentement médecin (physician's consent form)	Both	No	YES
Patient consent form and revocation form (Formulaires de consentement et formulaire de révocation de consentement)	Both	No	YES
Pharmacist consent forms	Both	No	YES
Shared care training material	Paper		NO	Appendix 1: 'soins partagés'
Shared care pharmacist guide	Electronic		NO	Appendix 2
IRB approval (Lettre d'approbation de l'IRB McGill)	Paper	No	YES
MOXXI posters (Affiches MOXXI)	Paper	No	YES
Pharmacie guide (Guide des pharmacies recrutées)	Both	No	YES
Procedure on patient recruitment (Guide sur le recrutement des patients)	Both	No	YES
Training material
Process and procedure for physician training session	Paper		NO	Appendix 3
Invitations to iPAQ training sessions (Invitations aux formations IPAQ)	Both	No	YES
IPAQ user guide (Guide d'utilisation du iPAQ)	Both	No	YES
Template for accreditation of MOXXI training sessions (Demande d'accréditation des formations MOXXI)	Both	No	YES
MOXXI system user guide (Guide d'utilisation MOXXI)	Both	No	YES
MOXXI training session power point slides	Both	No	NO	Appendix 4
MOXXI training session plan	Both	No	NO	Appendix 5
MOXXI training session invitation	Both	No	YES
MOXXI training session evaluation form	Both	No	YES
McGill's confidentiality agreement for physicians	Both	No	YES
Guide to on-line questionnaire	Paper		NO	Appendix 6
Communication tools
FAQs	Both	No	YES
MOXXInfo	Both	No	YES
MOXXI business cards	Both	No	YES
MOXXI postal cards	Both	No	YES
MOXXI Contact list	Both	No	YES
MOXXI website	Electronic		NO	www.moxxi.mc gill.ca
Templates
Templates for contracts	Both	No	NO	Appendix 7 - Templates for contracts
Job Descriptions and recruitment material			NO	Appendix 8 - Generic contract, recruitment material representant and instructor
Software information
Asthma and diabetes advanced decision support system algorithms	Electronic		YES
MOXXI system architecture	Electronic		YES
MOXXI application: database schematics, data dictionary	Electronic		YES
MOXXI release notes	Electronic		YES
Specifications for the software for the pharmacy interface (Specs Pharmacy Systems Interface)	Electronic		YES
RAMQ data extraction process (Processus Extraction Variables RAMQV9)	Electronic		YES
RAMQ support procedure (Procédure Soutien Client PMA 3)	Electronic		YES
Security guidelines about iPAQs	Electronic		YES
Security and confidentiality policy	Electronic		YES
Compliance computation algorithm	Electronic		YES
MOXXI suggested improvements list	Electronic		YES
Shared care process	Electronic		YES
Shared care referral specs (IPAQ)	Electronic		YES
Shared care paper schematics	Electronic		YES
Shared care paper intervention	Paper		NO	Appendix 9
TAGGE algorithm	Electronic		YES
Standard redeployment tests (redeployment_instal_Task 2)	Electronic		YES
Templates for testing
Templates for MOXXI performance evaluation tests	Paper		NO	Appendix 10 - MOXXI performance evaluation tests (excerpt of the file)
Templates for asthma algorithms tests	Paper		NO	Appendix 11 (legal paper)
MOXXI deployment procedure	Electronic		YES
Tools and procedures:
Protocol for calling study	Paper		NO	Appendix 12 participants randomized to home monitoring system
Standardized task protocol	Paper		NO	Appendix 13
Evaluation of TAGGE	Paper		NO	Appendix 14
Feedback to TAGGE about test results	Paper		NO	Appendix 15
Physician follow-up report (Rapport_prescription)	Electronic		YES
Support documentation (tool to gather additional data when on site with users)	Paper		NO	Appendix 16
Physician's time and motion protocol				Appendix 17
TAM questionnaire, call log for pharmacist, other questionnaires
Confidentiality and Privacy documents				Appendix 18

4. Main Impact

4.1 Human Resources Impact

4.1.1 New Skills for Physicians

The MOXXI project offered health care providers a unique opportunity to learn skills transferable in other context. One concern expressed by physicians that could influence uptake and acceptability is the increased time that may be required to use the system. User abilities may be a barrier to the adoption of an integrated electronic prescribing and drug management platform.

To assess the magnitude of this issue, a standardized task assessment was developed and used in training 28 family physicians for the MOXXI system. Participating physicians were provided with a personal digital assistant with the MOXXI software and completed a series standardized tasks using a pseudo-database. Times to complete tasks were recorded at training, 1 week (Time 1), 4 weeks (Time 2), and 6 months (Time 3) to assess improvement. The tasks that were evaluated include: 1) logging on to the system, 2) retrieving the records of a pseudo-patient, 3) changing the status for one of the patient's health problems, 4) adding an allergy to the problem list, 5) handwriting and electronically sending a prescription for three medications, including the drug, dose, duration and indication, and 6) re-prescribing (handwritten and electronically) eight medications. A research assistant recorded the number of seconds that were required to complete each task.

The results (Figure 4.1 & 4.2) show that improvement from baseline to the final training session was statistically non-significant (p-value ≥ 0.05) for Task 1 (log on). Some of the tasks increased in time after training but then significantly decreased for the final two sessions (Tasks 2, 4, 5).

The time to manually write new prescriptions (0.94 minutes, sd 0.24) is still faster than the average time to write a new electronic prescriptions (Task 5) at the final testing session (2.02 minutes, sd 0.97). However the fastest physicians can complete the task in 0.36 minutes (approximately 20 seconds) which is faster than the best time for handwriting 3 new medications (0.53 minutes). The greatest time improvement was seen for new electronic prescriptions with a mean time decrease of over half (1.58 minutes, p-value<.0001).

Figure 4.1 - Time for physicians to complete four standardized tasks on the MOXXI system at training, one week post-training (Time 1), one month post-training (Time 2) and 3 months post-training (Time 3)

Figure 4.2 - Time for physicians to complete the fifth and sixth standardized tasks on the MOXXI system at training, one week post-training (Time 1), one month post-training (Time 2) and 3 months post-training (Time 3)

In conclusion, there was significant improvement in performance (decrease time to task completion) for almost all tasks from baseline to 4 weeks. The biggest improvement in time was for Task 5 (prescribing 3 medications). For some of the physicians, prescribing three new medications is still slower to complete using the electronic prescription. However re-prescribing electronically is significantly faster (and neater) than handwriting the renewal prescription.

In the future, we plan to assess the relationship between the times on the standardized tasks with the utilization of the system, evaluate predictors of performance on standardized tasks (sex, age, utilization rates, computer fluency, prescribing habits), and evaluate the correlation between performance on standardized tasks and satisfaction with system

TAM Beliefs and Actual MOXXI System Usage

To investigate if any TAM beliefs at baseline was a significant predictor of overall E-Rx system usage from January to November 2003, three main TAM beliefs (Behavioral Intention, Ease of Use, and Perceived Usefulness) were assessed using multiple Poisson model. The model showed that Behavioral Intention was a significant positive predictor of actual usage (β=0.38; P<.001), followed by perceived usefulness (β=0.12; P<.001). However, perceived ease of use was not significant (β=0.01; P<.6).

MOXXI System Usage Trends by Behavioral Intention

MOXXI system usage trends over time were stratified by behavioral intention in Figure 4.3. A rating of Behavioral Intention was dichotomized at median score (66.67% vs. 33.33% for higher BI vs. lower BI). The e-Rx system usage trends, stratified by the degree of behavioral intention, showed that physicians with higher behavior intention (BI) used the e-Rx system consistently more, compared to the physicians with lower BI (Figure 8.7). This usage trend by level of behavioral intention at the baseline was constant throughout the time period from January to November 2003.

Figure 4.3 - Usage Trends by Behavior Intention

In summary, although study physicians reported a high level of satisfaction with their current practice, they also have high expectations for potential improvement, which can be addressed through the use of information technology. Physicians reported a commitment to use information technology, with the expectation that the technology should be easy to use and be useful. However, their relatively low level of current computer usage in clinical practice was seen as a potential barrier to manage the use of information technology during patients' visits. This information highlighted the importance of providing consistent support for physicians' skill acquisition that is necessary to manage an information technology system, such as the MOXXI system, in their practice.

Qualitative evidence shows that the MOXXI electronic prescribing system gives physician the information they need to better accomplish their work. Examples from study physicians reporting how the MOXXI system helped them deliver better care to their patients are numerous. Some physicians reported that having access to all patient's medication allowed them to have better tools to open a dialogue about such problems as compliance, which they were unable to do before. It also saved them time when updating their patients' medication history, especially when this patient saw multiple physicians or had been hospitalized. Follow up TAM questionnaires will be distributed to gather more solid evidence on this.

Physician's uptake

During the study period (January to November 2003) 7,949 electronic prescriptions were written by study physicians (mean [SD], 283.0 [304.3]). Within the same time period, a total of 35,734 visits by consenting patients were made to study physicians. Information from the log-file of the e-Rx system was used to measure the number of electronic prescriptions written each week. The number of medical visits made each week to study physicians by consenting patients was assessed using the medical services claims database. This database is provided by the provincial health insurance agency which provides comprehensive medical coverage for all Quebec residents. This database provides comprehensive inventory of all visits made by study patients.

Overall electronic prescription (e-Rx) system usage rate per 100 visits from January 15^th to November 30^th, 2003 were calculated for study physicians (Figure 8.6). Overall rate of the e-Rx system usage during the time period was 22.24 per 100 visits. At the start of the project implementation, the usage rate increased progressively until the 6^th week of usage (38.11per 100 visits). Between the 6^th and 10^th week, e-Rx usage rate fluctuated (18.41 and 34.86 per 100 visits at the 8^th week and at 10^th week, respectively). The 8^th week marked the arrival of a newly trained group of physicians who progressively began to use the system in practice. The e-Rx usage rate decreased from the 10^th week until the 16^th week (17.63 per 100 visits). This decline was influenced by a bug in the drug alerts system which was installed in March. From the 16^th week to 40^th week, the usage rates remained steady. The e-Rx usage rate then dropped slightly at the 42^nd week during the flu season, where visits to physicians were primarily for flu shots. The rate started increasing thereafter because of increased frequency of clinic visits by support staff (each physician seen at least once every two weeks in their offices for support). At the end of November 2003, the e-Rx system usage rate was at 33.33 per 100 visits.

Figure 4.4 - Electronic Prescription System Usage

4.1.2 Changes in Access to Health Service Information

A. Treatment Indication

Indications for prescribed medication are rarely documented in the chart, and they are also not generally known to the pharmacist responsible for dispensing the prescription. The absence of this information creates several problems. First, physicians cannot assess the on-going necessity of a patient's medication unless they know why the medication was originally prescribed. Even if a physician can recall the treatment indication for their own prescriptions, 69%-78% of elderly patients have more than one prescribing physician. Primary care physicians in particular, may also be asked to re-prescribe medication started in hospital or by other specialists without necessarily knowing why it was prescribed. Second, pharmacists are expected to advise patients on how to use the medications they dispense, yet with medications that have multiple indications, they cannot provide appropriate and relevant advice without knowing the reason for treatment.

Electronic prescribing provides an opportunity to resolve longstanding problems regarding the absence of treatment indication documentation. The development and implementation of a prototype integrated electronic prescribing and drug management prototype (MOXXI) allowed us to evaluate the feasibility of including treatment indication as a required field in the electronic prescription. The electronic prescribing pad was developed to record up to 4 treatment indications for a single prescription. To facilitate use, a drop-down menu of documented indications is presented for each drug that can be used to select treatment indication or the keyboard can be used to manually enter an Electronic Prescription Rate per 100 visits indication. As physicians may not be aware of the original indication for a drug prescribed by a prior physician, but may be asked to prescribe a refill, we had to add a "renewal" indication to the list for each drug.

A total of 24,119 medications were prescribed electronically by the 30 study physicians in first 11 months of the pilot project. The "renewal" indication was recorded in 22.1% of all electronic prescriptions. The medication with the greatest proportion of "renewal" indications was ASA (56.0%). ASA was also the medication with the highest proportion of indications entered manually on the keyboard. The high proportion of "renewal" was probably related to the absence of the indication "cardiovascular prophylaxis" from the drop down menu. The proportion of indications where "renewal" was selected varied by physician from 4.5% to 51.7%. This variation between physicians may be due either to an effort to mask the treatment indication field or a high proportion of patients in the practice whose prescriptions originate from other physicians. Unique information was generated for drugs where indication has traditionally been difficult to determine. For example, lorazepam, the most common benzodiazepine prescribed and 10th most common electronic prescription, was prescribed for anxiety (41.0%), insomnia (33.8%), renewal (22.1%), panic disorder (1.9%), convulsions (0.6%), nausea (0.3%) and cancer co-analgesia (0.3%).

Variation between physicians in the use of "renewal" as an indication requires further investigation. The proportion of prescriptions with valid indications may be increased by ensuring that all treatment indications including "prevention" are included in the drop-down list.

B. Improved Linkages in Community Care

The MOXXI project was created to foster integration of health information through its various components as illustrated in Figure 4.6. The drug decision support system and the compliance support programs are primary examples of an integrated drug management system that links information from primary care physicians, retail pharmacies, and patients with timely updates on current drug knowledge.

Figure 4.5 - The Electronic Prescribing and Integrated Drug Management System

Click on the image to enlarge or here

Figure 4.5 The drug management system is a light client, heavy server architecture. The beneficiary and drug database, and application are located on the MOXXI server, behind the McGill University Health Center firewall. A replication of the application, and encrypted drug and beneficiary database is also on the physician's pocket PC (iPAQ). Physicians connect to the server by wireless modem through the Bell 1x network (after device and password authentification) to automatically update the database of dispensed prescriptions, visits and new problems, and to send new prescriptions, and stop prescription orders via their iPac. Locally, a Bluetooth wireless network is used to print a copy of the prescription and Rx identification number. The pharmacist uses the Rx identification number to pull the prescription from the MOXXI server. The prescription is integrated into their clinical software so that it does not have to be transcribed. Information on dispensed prescriptions for the patient is sent back to the MOXXI server. Beneficiary names and RAMQ numbers, dispensed prescription and visit records for each physician's patient are sent from RAMQ in Quebec via the Internet SSL to the MOXXI server. Physicians can add new patients locally, and information is transmitted to the RAMQ via the MOXXI server, and drug and visit data is returned to the physician in 5 seconds.

C. Drug Decision Support

The drug decision support for prescribing and drug management provided on the physician's personal digital assistant (PDA) includes a drug profiler. The drug profiler provides a graphic representation of the list of current active medications of the consenting patients as well as historical medications over the past 12 months. The drug profiler permits identification of the drugs prescribed by the treating physicians and those prescribed by other physicians through a colourcoded diagram. This information is valuable given that 69%- 78% of elderly patients have more than one prescribing physicians. The drug profiler also illustrates dates of emergency visits or hospitalisation. New drugs that were dispensed upon discharge from the hospital are clearly visible on the profiler. For the community physician, the graphical representation is beneficial to understand the patient's complete pharmacological profile, which at times can be difficult to acquire from the patient.

In addition to the list of drugs, the profile graphically illustrates gaps and overlaps in the start and end dates of prescribed medication indicative of consumption of medication that is less than or more than the prescribed dose in the past 3, 6 and 12 months, colour-coded by prescribing physician. Primary noncompliance, drugs prescribed that have not been filled, is displayed as a grey bar of expected start and end dates for each electronic prescription written. By clicking on a specific prescription in the profile, physicians access detailed information on refills, average drug cost, and a refill compliance graph, depicting compliance in the past 3, 6 and 12 months. This compliance indicator provides physicians with information to discuss with their patients drug-specific compliance issues.

During a 4 month evaluation period (March to June, 2003), the participating physicians used the drug profiler a total of 1,704 times. The overall rate of drug profiler use was 15.2 times per 100 visits.

D. Compliance Support

For many patients, compliance with prescribed therapy is far from ideal. Approximately one-fifth of the prescriptions written are not filled. When a prescription is filled, estimated rates of noncompliance with treatment vary from 16% to 73%. Indeed, in Saskatchewan, only one-quarter of patients prescribed lipid reducing therapy will continue to use drug treatment by the second year, negating most of the possible benefit of the $6 million per year spent on drug treatment. Poor compliance with therapy compromises the potential benefits of treatment, and costs the Canadian health care system an estimated $3.53 to $4.49 billion per year.

The compliance rate for the target group of participants was calculated for the 6 month period prior to the implementation of the MOXXI compliance programs. Table 4a illustrates the percentage of patients with compliance rate of 70% or less for prevalent users of cardiovascular, diabetic and thyroid medications. Compliance ranged from 18.9% for thyroid replacement medications to 34.4% for diabetic medications, indicating a considerable problem with compliance with the targeted study group.

Table 4a - Compliance in Target Population of Patients Using Cardiovascular, Diabetic, Thyroid Medication (September 2002 - February 2003, MOXXI-III: 9,874 on target Rx)

Therapeutic Group	Prevalent Users
Cardiotropic	28.4 %
Hypolipidemic	25.9 %
Anti-Hypertensive	25.1 %
Diuretic	34.4 %
Oral Hypoglycemic	31.6 %
Thyroid Replacement	18.9 %

A component of the MOXXI project is the integration of compliance support programs in the drug management system. Participating patients who have chronic conditions such as asthma, diabetes, or are elderly and are taking cardiotropic, hypolipidemic, antihypertensives, diuretics, and/or thyroid drugs are randomised into 3 arms: automated telephone compliance support, shared care program, and usual care (Figure 4.6).

Figure 4.6 - Patient Compliance Interventions

The participants randomized to the automated compliance support system will receive automated telephone calls for refill reminders and an option for reminders to take their medications. At this point in time, the automated telephone compliance support system has been developed and is currently undergoing phase III testing. It is scheduled to go live in January, 2004. Because of the unanticipated delay in the implementation of this compliance support system, those randomized to this research arm are currently being called and notified that they will begin receiving calls from the automated system in the near future. The notification call provides the participants with a reminder about the project (as it may be one year since they signed the consent form), and the reason behind the automated compliance system.

4.1.3 Cost Effectiveness

Cost-effectiveness evaluation is only reasonable to conduct when the effectiveness of an IT intervention is shown. Further cost reductions can only be expected with improvements in outcome or avoidance of service duplication without deleterious effects. These outcomes can only be assessed over a 3-4 year period. The planning of the CHIPP program did not provide the time window required to accurately assess this outcome.

Qualitatively, the project's components such as the electronic prescription, drug profiler, and dates of hospital or emergency visits, can save steps necessary to provide service and consequently may be cost-effective. Electronic prescribing reduces the time needed for pharmacists to enter the prescription information onto the pharmacy computer system and possible calls to physicians for clarification of eligible prescriptions. For physicians, having the complete drug information on and dates of hospital or emergency visits can save time for those patients who have difficulty providing this important information to the physician. An example provided from a participating physician, was that of her patient who had schizophrenia and had come to her office. He was not able to provide her with the medications he was taking during that time. By accessing the drug profiler on the PDA, the physician had his complete drug profile available to her.

In the future, for each of the target medications, we will determine the average costs of the supply of drugs dispensed for each patient. Three components will be used in the estimation of drug costs. First, for each prescription dispensed, the cost per supply day will be determined by dividing the total cost of the prescription ingredients into the number of days of drug supply (duration field of prescription). Second, we will sum the supply day costs for each prescription included in the refill compliance measure for prescriptions dispensed in the follow-up period. This measure will exclude days after the drug was stopped as well as days of hospitalization. It will include supply day costs calculated from prescriptions dispensed prior to the study window but only for supply days in which the medication was available from the time of enrollment and inclusion in the study until the end of follow-up. The sum of supply day costs for all prescriptions dispensed as calculated for each prescription will be used to determine the value of total drug costs per drug within a therapeutic category for each patient. If a patient was taking more than one drug within the therapeutic target category, the sum supply days for all drugs dispensed within the therapeutic categories will be determined for each patient and the number of drugs used within the therapeutic category will be used as covariate in the analysis. To determine the overall impact of the compliance support programs on overall cost for target drugs within the study, we will, as a third measure, create a sum for all target drug supply days for each patient in the study population (assuming that many patients will be dispensed medications from more than one target class). For analysis of cost, the source of information for drug costs will be retrieved from Régie de l`Assurance-maladie du Québec prescription claims files for publicly insured patients and from the pharmacy's management system for privately insured patients. The cost-effectiveness of improving compliance with essential medication will be assessed by evaluating the impact on emergency visits and hospital admissions. The implementation the evaluation will be contingent on future funding.

For additional details on the impact of the MOXXI system, please refer to the evaluation report.

4.2 Privacy and Protection of Information

The MOXXI-III system created a model for integrated prescription drug management. This involved the sharing of information for optimal management of prescription medications between physicians, pharmacists and patients. The key elements of electronic health data included:

• Electronic prescriptions to be transmitted to community-based pharmacists
• The retrieval of information on drugs dispensed from the RAMQ prescription drug insurance database and the community-based pharmacists (for non-formulary drugs and for the privately insured)
• Diagnostic codes recorded for medical visits from medical services claims data
• Dates of hospital visits and emergency room visits

The first challenge was to define a model for data flow. This included how information should be transmitted, where it should be housed, and what the requirements were for access. To address these issues, the team conducted a privacy impact investigation in conjunction with key stakeholders and representatives of regulatory bodies. This included:

• The Order of Pharmacists
• The College of Physicians
• The Association of Pharmacy Banners and Chains
• The Association of Pharmacy Owners
• The Commission de l'accès à l'information
• The Régie de l'assurance maladie du Québec
• The McGill University Health Center Privacy Officer

A working group comprised of key stakeholders and the research team convened for a period of 8 months to map out usual work flow and defined the technology assisted model that would be used for transmitting electronic prescriptions, stop orders, and retrieving information on dispensed medication. The key components of the architecture for data flow that emerged from this working group were the following:

• Electronic prescriptions would be transmitted using a pull model. In a pull model, the electronic prescription is transmitted to a central server. The patient is provided with a copy of the prescription including a code number for the prescription that can be used to retrieve it from the central server. In this model, the patient is not required to select a pharmacy at the time the prescription is written. Although a push model, which would transmit the prescription electronically to the patient's pharmacy, would have been more efficient, it would have produced a substantial change in current practice as individuals would have to select the pharmacy to which the prescription would be sent in the physician's office. In contrast, a pull model fits current practice, whereby individuals may decide not to fill the prescription or to fill the prescription at a pharmacy that they select after they leave the physician's office. Thus the pull model was selected because: 1) it matched the current work flow, 2) it provided individuals with the same freedom to choose not to fill the prescription, 3) it allowed individuals to make last minute decisions as to their pharmacy of choice, 4) it did not provide pharmacies with information that they would otherwise not have if the individual chose not to fill the prescription at their pharmacy, and 5) it avoided a potential problem that was of concern to the pharmacists where prescriptions would be directed to a specific pharmacy by physicians.
• In contrast to the pull model for electronic prescriptions, the team decided to use a push model to facilitate efficient communication between the physician and pharmacist in relationship to decisions to stop or change an order for a prescription that had already been dispensed. We discovered that stop orders were rarely communicated to pharmacists, and generally, the pharmacist, if concerned that a medication had been stopped or changed, would consult the patient, and rarely call the physician. The regulatory guidelines for stop and change orders had not been clearly elucidated, a grey area that needs to be clarified in the regulations administered through the college of physicians and order of pharmacists. As a prescription that was being stopped or changed had already been filled by a selected pharmacy, the push model was selected because the individual had already selected the dispensing pharmacist, and it was the most efficient and safest way to transmit information to the dispensing pharmacist.
• The team decided that the central server that managed electronic prescriptions and records of dispensed medications would be housed behind the firewall of a public health care institution. Rather than creating a separate infrastructure to support privacy and security standards, the team elected to take advantage of the expertise infrastructure that already existed in the public hospital institutions. For this reason, the server that maintained databases for individual physicians in the community as well as information on new electronic prescriptions and records of dispensed medication was housed behind the firewall of the McGill University Health Center. In this way, the management of community-based prescription data benefited from the oversight of the security and regulatory requirements that pertained to electronic hospital-based information. As a result of this decision, some modifications were made in the data flow architecture on the basis of the recommendations and requirements of the McGill University Health Center privacy and security officer. Specifically, the computer telephony service that initially was responsible for data capture from the pharmacists and transmission to the MOXXI server was excluded from the architecture, and direct communication between the pharmacists and the MOXXI server was instituted. The computer telephony service, in turn, for consenting patients, would receive a selected subset of information necessary to conduct refill reminder calls. In addition, a separate firewall had to be created for the project as standards for efficient connectivity to the community-based health care system have not been established.
• Access to information on dispensed medication from the Régie de l'assurance maladie du Québec and the community-based pharmacists, required an electronic signal indicative of the presence of a written consent. In earlier phases of the MOXXI project, a written consent was obtained and a file of consenting patients was created that would be sent in batch form periodically to the RAMQ, after which physicians could have access to information on dispensed prescriptions. This was a cumbersome model that was extremely inefficient, and failed to provide physicians with information at the time of the visit. As a result, in MOXXI-III, we moved to a model whereby an electronic signal would indicate that the patient had a written consent in their file. Upon receipt of the electronic signal of a written consent, information about dispensed prescriptions would be transmitted electronically from the Régie de l'assurance maladie du Québec. Current legislation supports the concept of a cyber-notary. In this instance the physician is responsible for signally the presence of a valid consent, and the physician's identity is authenticated through both the specific pocket PC device coupled with the physician's password. This model for consent management was suggested by Mr. Jean Foisy at the CAI, and has since been used in a consent management module created by Andre Rousseau at the MUHC, that was acquired by the province for provincial roll-out of electronic health records data access control. For the community-based pharmacists, the written consent was also required in the patient's file for the pharmacist to transmit information on dispensed prescriptions to the MOXXI server. At the present time, the patient has to arrive at the pharmacy and deliver their consent for the information to be transferred from the pharmacy computer to the MOXXI server, and in turn, to the physician. This is a cumbersome model that will need to be modified for an efficient roll out in the future.
• In the MOXXI-III model, patients give their consent for a treating physician to have access to their prescription and visit profile. Patients in the current model are not authorized to provide consent to identify other prescribing and treating physicians or dispensing pharmacies. For this reason, in conjunction with the Commission de l'accès à l'information, the operating principle of the project is that the treating physician will be aware that other physicians are prescribing for the patient and other pharmacists may be dispensing medication but they will not be provided with information about the names and identities of other pharmacists and physicians. To obtain this information, the treating physician would typically ask their patient. This principle fits with current privacy legislation, but needs to be reviewed periodically as it may represent a barrier to appropriate communication and increase the risk of avoidable errors related to the inability of the treating physician to access and discuss potential problems with other prescribing physicians and dispensing pharmacists.

4.2.1 Barriers Created by Privacy Requirements

In the MOXXI-III model, data is encrypted prior to transmission and de-encrypted upon receipt. At the present time, the technology is insufficiently powered to maintain a reasonable performance speed and also support the encryption process. In the absence of a virtual private network, encryption/de-encryption was required and this created barriers to use by physicians as it lengthened response times in relationship to accessing patient data files.

Physicians found the consent process onerous. As a result, consent rates varied from physician to physician from 8% to 75%. There was a systematic trend, as outlined in the evaluation report for non-consenting patients to be persons of lower income and education. As a result, the consent process as currently constructed, may be increasing barriers for optimal care delivery for persons in lower income and education groups. Consideration needs to be given to other models for consent to access essential information for prescription drug management. These may include an acknowledgement and opt-out model such as that used in the B.C. Pharmanet.

4.2.2 Unexpected Findings

The drug profile provides information on dispensed medications, prescribed for a given individual, by the treating physician as well as other physicians. The profile not only provides the treating physician with better information on current prescription drug therapy to deliver optimal care, but it also is a mechanism of fraud identification. In an incident that involved one of the MOXXI-III physicians and patients, the physician and patient identified that they had neither prescribed the drugs nor had they received the medication listed in the profile. Indeed these medications were prescribed by other physicians and dispensed by other pharmacies. After preliminary investigation with the Regie, this situation was identified as a potential identity theft, whereby the patient's Medicare card was likely duplicated and was being used by others. The Régie de l'assurance maladie du Québec is investigating the situation. The opportunity to identify fraud, and/or drug abuse, was recognized by the Drug Enforcement Agency in the United States. The MOXXI-III team has been invited to provide a presentation at their next conference in San Diego in 2004.

4.3 Policy and Research Implications

The MOXXI project, as a primary goal and ongoing vision, aims to enable the best health care for the Canadian population. Since drug costs represent the largest expense in the health care system, improving the appropriate use of medication would generate global benefits. The earliest benefit of electronic prescribing is the decrease in medication errors caused by illegible scripts, incorrect doses and drug-drug interactions with potential for drug-related illness. These improvements in medication errors results in cost avoidance as an early return on investment. However an electronic prescribing system can also leverage improved medical care quality when drug-disease interactions and chronic disease management is added. The MOXXI results validate that the Institute of Medicine 2000 Report (Crossing the Quality Chasm) recommendations for the widespread deployment of electronic prescribing and eventual completely integrated computerized patient record are feasible and achievable goals.

4.3.1 Major Finding 1

An integrated drug management system that retrieves information on prescriptions dispensed to patients from community-based pharmacists (or insurers) is a necessary requirement for physicians to use an electronic prescribing system and to enhance the safety and quality of prescription drug therapy. Indeed, physicians are much more likely to consent to patients where information from an integrated drug management system is available versus not. However, there are considerable barriers to developing the interfaces necessary with community-based pharmacies to permit exchange of information between physicians and pharmacists. These problems include aging technology at the pharmacies, difficulty of gaining priority for interface development without it being a provincial requirement, the absence of a standard coding system for a posology (it is a text field in most software). Our study showed that in a metropolitan area over 60% of prescriptions obtained by patients were not prescribed by their primary care physician. Thus, for safety reasons alone, there is a need to insure that each and every physician prescribing for a patient has complete information on their current drug therapy.

Recommended Actions

Only some provinces have developed the capacity for an integrated drug management system through the creation of a Pharmanet. The two largest provinces, Ontario and Quebec, which hold over half of the Canadian population, do not have this capacity. It is strongly recommended that provincial and territorial governments provide incentives or legislative requirements to create an integrated drug management system that would permit electronic prescriptions and retrieval of current dispensed medications through a provincial Pharmanet. Further, such information should be accessible to treating physicians and dispensing pharmacists with proactive patient consent (as for the MOXXI project) or with patient notification and opportunity for opt-out (e.g. B.C. Pharmanet).

4.3.2 Major Finding 2: Primary Compliance

The rate of primary medication (filling first prescription) non-compliance was previously difficult to measure since tracking of an unfilled paper script was inaccurate. With the MOXXI e-scribe system this can now be tracked and interventions designed. Non-compliance with first prescriptions has implications for patient education about the disease and drug being prescribed. Public awareness campaigns may also be required in poorly-understood effective therapies.

4.3.3 Major Finding 3: Utility of Proxy Information Sources in the Feneration of Health Conditions

The MOXXI e-scribe system tested a new approach to generate individual patient health conditions automatically from existing electronic data sources (prescription and billing data). This approach as implications for the transition from paper-based health records to eventual computerized records. Using manual chart abstractions to pre-load health records are timeconsuming, costly and limited by interpretations during the abstraction process. The new method used in the MOXXI project needs further study to validate that the automated method is as good or better than the traditional manual approach. Embedding this or similar 'smart' information gathering function into clinical systems can improve the acceptance of IT by clinicians.

4.3.4 Major Finding 4: Private-Public Partnerships are Essential for Continuing Success

Success in a controlled project with manageable numbers can be achieved through highly dedicated and focused project team members. When a system such as the MOXXI e-scribe project is deployed to a larger group, business practice savvy and breadth of resources that exists in the private sector has to be tapped. IT support becomes a major undertaking. Outsourcing is not the only model to be used. Support may need to be tailored at a regional level, to respond to the strengths and weaknesses of the system infrastructure in each region (eg like the availability of high-speed ASDL or cable service is regional).

4.3.5 Major Finding 5: One Size Does Not Fit All

Out in the field there are myriad combinations of modern hardware running antiquated software, substandard infrastructure installations, unusual user preferences and habits. The clinical user may also have a different perception and expectation of what an application can and cannot do. This area of change management in physicians' office practice and its relation to clinical software design and implementation deserves further study. The implication of this finding is that more thought and resources need to be dedicated to the installation and training of clinical systems into these environments. It is easier to accomplish this in a health care institution since a corporate or institutional policy can be adopted and enforced.

4.3.6 Major Finding 6: Clinician Input and Feedback is Essential

Physicians' and pharmacists' ongoing input are required to provide feedback for quality assurance, auditing, and deign and implementation of future functionalities. Methods to assist this process may include: Website /paper /phone surveys, users groups, and focus groups.

5. The Future

One of the challenges in research and development projects is to transfer the innovations created through the research and development process into a solution that can be applied to the population and sustained. After ten years of developmental work, culminating in the third phase of this project funded by the Canadian Health Infostructure Partnership Program, it was evident that the innovations developed in this decade of research experience could not be transferred or generalized on a broader scale without a business team. As a result, McGill University created a spin-off company, MOXXI Medical Inc. in the summer of 2003 and obtained funding from MSBI, a seed venture capital company that is co-owned by McGill University, the University of Sherbrooke, and Bishop's University.

The CEO of MOXXI Medical, Monsieur Jacques Paquin, as well as a strong business team (Chief Financial Officer: Normand Rivard, Chief Technology Officer: Alain Simard, Marketing Director: Sean Dalton, Director of Public and Private Relations: Alain Denis) have developed the business plan for MOXXI Medical Inc. and are in the process of building strategic partnerships and fundraising for the purposes of implementing the MOXXI solution in Canada and internationally. Two patent applications were filed, one for the drug profiler and the second for the automated problem list. The research team will continue to develop and evaluate valueadded add-on modules for the MOXXI integrated drug management model. In addition to the asthma and diabetes support modules funded by CHIPP, the MOXXI research team anticipates developing additional evidence-based chronic disease management modules. Furthermore, we have identified the following target areas for research and development:

• The incorporation of adverse drug reaction reporting and pharmaco-surveillance into the stop and change order function of the MOXXI Integrated Drug Management system;
• The development and evaluation of a micro-policy platform for introducing multi-tiered first, second, and third line drug formularies into the electronic prescribing program;
• The implementation of pharmacogenomic screening to guide the selection of drugs and doses;
• The integration of lab and diagnostic information along with prescription drugs; and
• The development of an automated detection system for symptoms that may be related to an adverse drug reaction to provide an early warning system for physicians re: adverse drug effects in individual patients.

The CHIPP co-applicants intend to continue with the research and development aspects of the new company. New funding from CIHR and pharmaceutical companies has been requested to continue the current pilot project and expand to another geographic region within Montreal. A collaborative project with BC will be designed: to implement a MOXXI-like e-prescribing system integrated to the BC Pharmanet. Funding for this collaborative project will be requested from Canada Infoway Inc. The major challenge of the new MOXXI Medical Inc. company will be to secure sustainable funding from the provincial governments who are already paying for health care. A strong business case will be presented to the payors indicting where their return on investment will come. Our MOXXI team has been following closely the activities of Canada Infoway Inc and the Electronic Health Record Solution architecture efforts. Our design and principles are well matched to the interoperability proposed by the Infoway architecture. We feel confident that the CHIPP funding which as supported the MOXXI project has allowed the advancement of electronic prescribing and as a stepping stone to a full electronic health record.

6. Communications

The following table contains our communications effort during the course of the project. Documents and presentations are attached as appendix as requested.

Methods or Tools	Date	Targeted Audience	Documents or Presentations Produced	Appendix Name/ Number
Conferences/ Presentations	2004	COACH E-Health 2004 Victoria, BC	Using electronic prescriptions to document treatment indication: Evaluation of data quality in the MOXXI integrated prescribing and drug management system (R. Tamblyn et al.)	Appendix 19
	2004	National Association of State Controlled Substances Authorities Conference	Out with the old, in with the new: The impact of an integrated electronic prescription system on reducing inappropriate prescribing (G. Bartlett)
	2004	Division of Clinical Epidemiology, Royal Victoria Hospital Montreal, QC	Predictors of consent for an integrated electronic prescription project: Implications for pharmacoepidemiology (G. Bartlett)
	2004	AGS 2004 Annual Meeting Las Vegas, NV	Trends and indications for prescribing benzodiazepines for the elderly (A. Huang, G. Bartlett, R. Tamblyn)	Appendix 20
	2004	Medinfo 2004, 11th World Congress on Medical Informatics	Determinants of consent to participate in an integrated electronic prescribing project (G. Bartlett, R. Tamblyn, E. Laville)	Appendix 21
	2004	Medinfo 2004, 11th World Congress on Medical Informatics	Physician predictors of utilization of an electronic drug management in primary care (Y. Kawasumi, R. Tamblyn)	Appendix 22
	2004	VESPA	Evaluation of a computerized asthma management decision support system (R. Tamblyn, P. Ernst)	Appendix 23
	December 3, 2003	McGill University Faculty of Medicine Student Research Day Montreal, QC	Validating diagnostic codes in administrative databases with online diagnoses from primary care physicians (C. Sairam, G. Bartlett)
	November 8-12, 2003	American Medical Informatics Association (AMIA) 2003 Conference Washington, DC.	Evaluation of standardized tasks for primary care physicians using the MOXXI electronic prescribing and integrated drug management system (G. Bartlett, R. Tamblyn, A. Huang, Y. Kawasumi, L. Petrella, E. Dufour)
	November 2003	Ontario Hospital Association Annual Convention Toronto, ON	Enhancing optimal disease & prescription drug management (R. Tamblyn)	Appendix 24
	October 18, 2003	RAMQ Quebec City, QC	MOXXI Project Presentation	Appendix 25
	October 18, 2003	MSSS Gouvernement du Quebec Quebec City, QC	MOXXI Medical Inc.	Appendix 26
	October 14, 2003	OHA Privacy Workshop Toronto, ON	E-Health: Building a more effective healthcare system (R. Tamblyn)	Appendix 27
	September 29, 2003	1st Annual Maine Benzodiazepine Study Group Conference Bangor, Maine	MOXXI - The role of an integrated electronic prescribing system in pharmacovigilance and pharmacoepidemiology (G. Bartlett)	Appendix 28
	August 17, 2003	16th Annual CSEB Central Region Student Conference. Montreal, QC	To determine technical performance of automated voice recognition compliance support system - Pilot study (Y. Kawasumi R. Tamblyn, G. Bartlett)	Appendix 29
	July 29, 2003	UBC Division of Geriatric Medicine Academic Rounds Vancouver, BC	Rational prescribing for the elderly (A. Huang)
	June 19, 2003	CIHR Health Informatics PhD/Postdoc Strategic Training Workshop Victoria, BC	Clinical and health informatics research group activities (A.Huang)
	June 5, 2003	MOXXI Physician and Pharmacist Participants Montreal, QC	Preliminary results and feedback gathering	Appendix 30
	June 2003	McGill University Faculty of Medicine Advisory Board Montreal, QC	Addressing challenges in health care (R. Tamblyn)	Appendix 31
	June 2003	2nd National CSEB Student Conference, Halifax, NS	Physician preferences associated with the use of an electronic prescribing system (Y. Kawasumi, G. Bartlett, R. Tamblyn)	Appendix 32
	May 31, 2003	Canadian Pharmacists Association Annual Conference Vancouver, BC	MOXXI - the Medical Office of the 21st Century (R. Tamblyn)	Appendix 33
	March 13, 2003	McGill University Evening Lecture Series Montreal, QC	The dangers of drugs: Appropriate pharmacotherapy in the elderly (A. Huang)
	2003	Health Statistics Data Users Conference, Statistics Canada Ottawa, ON	Invited Round Table Discussant (R. Tamblyn)
	2003	Health Innovation, Wealth Creation and System Renewal Conference - Université de Montréal Montreal, QC	E-Health: Building a more effective healthcare solution (R. Tamblyn)	Appendix 34
	December 2002	Alberta Wellnet/Pharmanet	Evidence-based utilization of prescription drugs: Challenges and future directions	Appendix 35
	2002	Fifth International Conference on Electronic Commerce Research (ICECR-5) Montreal, QC	E-Technology introduction as value creation in integrated systems: A multi-level analysis using role theory	Appendix 36
	April 14- 16, 2002	The Canadian Association for Population Therapeutics Annual Conference (CAPT) Toronto, ON	Electronic prescribing and compliance support: Opportunities to enhance practice and research (R. Tamblyn)	Appendix 37
	February 12-13, 2002	4th Annual Chain Drug Conference Toronto, ON	Integrated health care delivery (R. Tamblyn)	Appendix 38
	2002	9th Annual Journées Francophones d'informatique médicale (SoQuibs) Conference Quebec, QC	Member of Scientific Committee and Invited Panel Moderator (R. Tamblyn)	Appendix 39
	2002	GlaxoSmithKline - Canada Health Infoway Forum Montreal, QC	Invited Round Table Discussant (R. Tamblyn)
	2002	Summer Program Presentations, Faculty of Medicine, McGill University Montreal, QC	Integrated health care and opportunities for future research (R. Tamblyn)
	2002	CIHI Board of Directors Meeting Kensington, PEI	Pharmaceutical Data and the implications for CIHI (R. Tamblyn)
	2002	Institute for Research on Public Policy (IRPP) Conference Toronto, ON	An evidence based presentation on national standards regarding utilization (R. Tamblyn)
	2002	Romanow Commission on the Future of Health Care in Canada & Queen's University McGill University Montreal, QC	Pharmacare - A dialogue on issues and solutions (R. Tamblyn)
	2002	Diagnostics & Solutions: Building Consensus for Health Care Reform in Canada McGill Institute for the Study of Canada Public Conference McGill University Montreal, QC	Pharmacare and drug therapy (R. Tamblyn)
	2002	Challenge 2002: Managing Health beyond Borders: Integrated Care for Integrated Health McGill University Montreal, QC	Invited Speaker (R. Tamblyn)
	October 10, 2001	Royal Victoria Hospital Medical Grand Rounds Montreal, QC	Salient issues in geriatric medicine (A. Huang)
	2001	9th Canadian Conference on Health Economics University of Toronto Toronto, ON	Invited Speaker (R. Tamblyn)
	2001	CHIPP National Workshop Ottawa, ON	Invited Panelist (R. Tamblyn)
	2001	Standing Senate Committee on Social Affairs, Science and Technology Montreal, QC	Invited Speaker (R. Tamblyn)
Media Events	December 2003	MUHC Ensemble	QUALCOMM 3G CDMA-List Award Article	Appendix 40
	November 20, 2003	MSBi Capital Press Release	MSBi Capital announces investment in MOXXI Medical Inc.	Appendix 41
	November 2003	Canadian Healthcare Technology	Wireless handheld computers enable doctors to prescribe at point of care	Appendix 42
	October 22, 2003	Innovations Report - Medicine/Health	Innovative health project wins international award	Appendix 43
	October 20, 2003	Qualcomm - Video Interview	CMDA-A List Award winner: Impact, non-profit organization
	September 16, 2003	Global TV - Live Interview	RE: Adverse Events article
	September 16, 2003	CBC TV - Live Interview	RE: Adverse Events article
	September 16, 2003	CTV - Live Interview	RE: Adverse Events article
	September 16, 2003	Montreal Gazette	Elderly get wrong drugs - Poor communication to blame for errors, McGill study concludes	Appendix 44
	September 2003	Pharmacy Post News	Still winding along road to eprescribing	Appendix 45
	August 26, 2003	Innovium Capital Corporation - Press Release	Innovium investment - ThinWEB Technologies signs contract for MOXXI Project	Appendix 46
	June 1, 2003	Canadian Healthcare Manager	Prescribing patient safety	Appendix 47
	May 29, 2003	National Post	Online prescriptions could save lives: report	Appendix 48
	April 22, 2003	The Medical Post	Avoidable drug interactions still causing hospitalizations	Appendix 49
	January/ February 2003	MUHC Health Perspectives	'Moxie' enough to go round	Appendix 50
	Winter 2002	Le Reseau Informatique, Volume 14	Feu vert quebecois pour les projets PPICS	Appendix 51
	September 23, 2002	TIME Magazine	Wired Canada	Appendix 52
	September 2002	MUHC Research Institute News, Volume 1, Issue 5	"The Cutting Edge" Time Magazine September 2002	Appendix 53
	December 2001	Healthcare Information Management & Communications Canada, Vol. XV, No. 5	Integrated provider solutions: Unleashing the power of technology for Canada's health care providers	Appendix 54
Publications	September 16, 2003	Canadian Medical Association Journal	The medical office of the 21st century (MOXXI): effective of computerized decision-making support in reducing inappropriate prescribing in primary care (R. Tamblyn et al.)	Appendix 55
	2003	Innovation: Essays by leading Canadian Researchers	The New Millennium Model for Health Care and Research (R. Tamblyn)	Appendix 56
	1998	Chapter in Encyclopedia of Biostatistics	Drug Utilization Patterns (R. Tamblyn & M. Abrahamowicz)	Appendix 57
Open House	October 22, 2002	All MOXXI Participants/Partners Montreal, QC	Project Presentation	Appendix 58
Marketing / Advertisement	November 2003	Open to public Montreal, QC	Book signing for Innovation: Essays by Leading Canadian Researchers	Copy of slide presentation attached
Website			www.moxxi.mcgill.ca
Other	2002	Primary care physicians	Physician Recruitment	Brochure and slide presentation attached
	2002	Patients	Patient Recruitment	Brochure attached
		Patients	Patient Consent Form	Consent forms attached