Atlantic Telehealth Knowledge Exchange (ATKE)

Health and the Information Highway Division, Health Canada 2003

Table of Contents

• 1. Background
  • 1.1. Background and Rationale
  • 1.2. Program Funding and Accountability
  • 1.3. Program Description
  • 1.4. Vision, Goals, and Objectives
• 2. Evaluation Approach and Methods
  • 2.1. Evaluation Approach
  • 2.2. Program Logic Model and Evaluation Framework
  • 2.3. Data Collection Methods
  • 2.4. Analysis and Report Writing
• 3. Evaluation Findings
  • 3.1. Developing a Collaborative Structure
  • 3.2. Knowledge Repository
  • 3.3. Information Sharing and Dissemination
  • 3.4. Sustainability
• 4. Conclusion and Summary of Evaluation Recommendations
  • 4.1. Conclusion
  • 4.2. Summary of Evaluation Recommendations
• References

Acknowledgements

The ATKE project and its process evaluation were funded through Health Canada's Office of Health and the Information Highway's Knowledge Development and Exchange (KDE) Applied Research Initiative program.

The ATKE Steering Committee would like to thank Tish Campbell, Debbie Gill of Sierra Systems Inc., John Buckley, formerly of Sierra Systems Inc., and Andrea Battcock of Tecknowledge Professional Group for their research and for the extensive material provided in various documents submitted to the ATKE project.

We would also like to acknowledge Paul Chaulk, Nishka Smith of Atlantic Evaluation Group Inc., and Patsy Beattie-Huggan of The Quaich Inc. for their valuable contribution to the evaluation of this project and for the creation of this final report for dissemination.

Executive Summary

Background

Atlantic Canada has some of the earliest, most comprehensive, well established networks and innovative applications for telehealth in the country. The region offers a wide range of models for telehealth in terms of management structure, co-ordination, funding, equipment, utilization, and telehealth applications. Collectively, this diversity, experience, and wealth of knowledge can significantly contribute to the development of a knowledge base for excellence in telehealth services. While telehealth itself is a mechanism for information and communications technologies to break down geographic barriers, there is little opportunity for telehealth professionals to go beyond their regional and / or provincial boundaries to exchange information, share their individually identified challenges and successes, and discuss issues of interoperability.

In 2002, the IWK Health Centre in Halifax, Nova Scotia and its project partners, the Nova Scotia Telehealth Network, Department of Health, Nova Scotia; River Valley Health, New Brunswick (Regional Health Authority # 3); Telehealth and Educational Technology Resources Agency (TETRA), Memorial University, Newfoundland and Labrador; and the Prince Edward Island Department of Health and Social Services received funding through the Knowledge Development and Exchange Program, which enabled the partners to undertake the applied research project called Atlantic Telehealth Knowledge Exchange (ATKE).

Project Description

The Atlantic Telehealth Knowledge Exchange (ATKE) project was developed to emphasize a 'bottom-up' strategy for breaking down barriers and facilitating communication and information sharing among stakeholders. It involved telehealth service providers and system managers, including: coordinators, provincial directors, and existing committees / forums (e.g., The Children's Telehealth Network Steering Committee and provincial meetings of telehealth coordinators, as applicable). The project was meant to complement existing regional (e.g., HIA) and national (e.g., Health Canada's OHIH) initiatives.

For the purposes of this project, telehealth activities included clinical consultations, health education and health administrative sessions conducted via videoconferencing, store and forward applications, web casting or the telephone. Telehealth activities did not include telephone advice programs, health information web sites, electronic health records and health information systems initiatives (unless integrated with telehealth activities or large and significant in nature).

The Vision for the ATKE project was to promote excellence in telehealth in Atlantic Canada and to contribute to telehealth knowledge and knowledge exchange at provincial, regional, and national levels. To achieve this vision, the ATKE project defined the following project goals:

• To initiate the definition and development of a sustainable Atlantic telehealth structure that will foster excellence in telehealth at the provincial / inter-provincial level in Atlantic Canada.
• To model, through the ATKE project process itself, an initial model / framework for Atlantic collaboration in telehealth.
• To leverage 'grassroots' input by involving all telehealth professionals / organizations currently involved in telehealth in Atlantic Canada in the applied research project.
• To produce a research report that captures the results of the ATKE applied research project.
• To complement the work of Health Infostructure Atlantic (HIA) in leveraging Ministerial support for the use of information and communications technologies in improving access and quality of health care services in Atlantic Canada.
• To capture and synthesize baseline information that can be leveraged to market and promote telehealth in Atlantic Canada.
• To establish a foundation for leveraging external funding opportunities in support of collaborative, pan-Atlantic telehealth initiatives.

The Steering Committee members were responsible for overseeing the project and the process evaluation. Sierra Systems Group Inc. and TecKnowledge Professional Group were contracted to implement the project.

Evaluation Approach and Methods

The ATKE Steering Committee contracted a partnership of two consulting firms, Atlantic Evaluation Group Inc. and The Quaich Inc. to conduct a process evaluation of the ATKE project; a formal, value-based, post-project assessment to determine value derived from the research effort and the potential for the Atlantic Telehealth Knowledge Exchange. A participatory evaluation approach was used with the involvement of the ATKE project Steering Committee at all stages of the evaluation in order to enhance the relevance and usability of the evaluation results.

A program logic model and an evaluation framework were developed. Data were collected through interviews and a document review. A semi-structured focus group / interview guide was developed for the collection of input from Steering Committee members, HIA, stakeholders, and project consultants. Three Steering Committee members, along with one stakeholder from PEI were interviewed during a focus group videoconference; one Steering Committee member was interviewed individually over the phone; and one Steering Committee member was interviewed as part of the HIA group interview. The 45-minute HIA group interview was conducted during a regularly scheduled in-person HIA meeting. Seven stakeholders -- one from PEI and two each from the remaining Atlantic provinces -- were individually interviewed over the telephone. Finally, one consultant from Sierra Systems Group Inc. and one consultant from TecKnowledge Professional Group were interviewed together in an audioconference.

Evaluation Findings

Developing Collaborative Structure

One of the major areas of activity for ATKE was the development of a collaborative structure for the sharing of information about telehealth:

• Objective 1: To establish a collaborative structure among telehealth initiatives in Atlantic Canada in order to facilitate the development and exchange of information on the Atlantic telehealth experience.

To achieve this objective, leadership was required in managing the project and in engaging partners and stakeholders. The ATKE project was successful in all areas of project management. This is attributed to: a) the adoption of a sound project management model provided by the consultants; and b) the commitment and experience of members of the ATKE Steering Committee. These factors enabled the project to overcome unforeseen events such as changes in consultants and committee members, as well as time delays. A different kind of working relationship evolved for the Steering Committee throughout the process of implementing ATKE; along with the high degree of commitment among Steering Committee members and the recognition of differences in each province it was seen as providing a strong foundation for future collaboration.

It would appear that the ATKE project was successful in engaging stakeholders and increasing awareness of the project. Some of the activities completed included: identification of potential partners / key stakeholders; contact with potential partners / key stakeholders to introduce them to the project and encourage their support and input; development of a communication plan to keep partners / key stakeholders apprised of progress; and hosting a workshop.

Knowledge Repository

One of the primary aims of the ATKE project was the establishment of a repository of baseline information on Atlantic telehealth initiatives and experience.

• Objective 2: To establish a repository of baseline information on the Atlantic telehealth experience, lessons learned, and best practices.

The key activity carried out so as to move this objective forward was research which focused on: the degree to which stakeholders felt that various knowledge, information, and data elements should be included in a telehealth data repository; the current level of stakeholder access to telehealth information and their preferred methods of accessing telehealth information; the types of elements that exist in their telehealth project / program and the availability of information on the project / program; the requirements needed to develop an operating model for ATKE; and gaps in telehealth knowledge, information, and data.

Although the development of a data repository is a key activity of ATKE, at the end of the project this activity was not perceived as the most useful function of the ATKE Steering Committee. Rather, it was concluded that ATKE should have more of an overseer role with regards to the data repository, so as to ensure that there is value in the process for those at the grassroots level who are contributing the information. It should also be recognized that there are other projects in Atlantic Canada, which are yet to be consulted.

Developing a repository for baseline data appears to be a viable option for ATKE. There is backing for the development of a web-based solution, and structures do currently exist that could support this initiative. The need to link to Health Canada's web site, the Canadian eHealth Initiatives Database, in order to support the knowledge repository was seen to be very important.

Information Sharing and Dissemination

Having a comprehensive dissemination plan to communicate research findings and project outcomes to stakeholders, partners, and others with an interest in telehealth was determined to be vital to increasing awareness of the project, fostering collaborations, strengthening partnerships, and helping to move the project forward. Objectives related to dissemination are:

• Objective 3: To generate and regionally disseminate findings and knowledge gaps identified from the baseline data.
• Objective 5: To disseminate the experiences and information on telehealth in the Atlantic region to relevant initiatives across Canada.

Because stakeholders were being consulted and engaged in the data collection process throughout the project, much of the information sharing took place as part of the communication and consultation efforts around these events. To date, dissemination of the findings has occurred in two formats: 1) a presentation during the stakeholder workshop; and 2) the videoconference presentation of the Sierra Systems Group Inc. (2003) report. The importance of broad dissemination to stakeholders was emphasized by all groups interviewed. A dissemination plan was developed and is included in the Sierra Systems Group Inc. (2003) report. Based on information-sharing activities carried out throughout the project, it would appear that the project has been successful in promoting the work of ATKE and increasing awareness of the project.

Sustainability

Because the work being carried out through the ATKE project is long-term in nature, a key part of the project was focused on sustainability and planning for the next steps.

• Objective 4: To identify strategies and seek ongoing regional commitment for further analysis, research, and / or policy development in the Atlantic region.

The activities carried out in order to move towards identifying strategies to sustain the work and seek ongoing commitment included: developing a value proposition; carrying out discussions regarding the requirements for ATKE, potential next steps, and funding options; linking with HIA; and identifying future research possibilities.

To date, the ATKE project has been successful in identifying some strategies for sustainability and in seeking regional commitment for further research. The various activities undertaken have resulted in: the development of a working relationship with HIA; the development of a value proposition identifying the relevance and value of developing an Atlantic Telehealth Knowledge Exchange; identification of some next steps / strategies for exploring relationship options with HIA; aligning with regional service delivery initiatives; aligning with provincial and federal service delivery strategies; aligning with emerging educational modalities for health care providers; and the identification of future research possibilities.

The ATKE project appeared to be effective in engaging stakeholders and raising the profile of telehealth in Atlantic Canada. The need for continued collaboration was identified and discussions took place on the future role of ATKE "to promote excellence in telehealth and to promote opportunities to work together collaboratively." In order to take on this role, ATKE will need to address some of the concerns, expressed during the evaluation by stakeholders and others, regarding barriers to collaboration; e.g., work / time requirements of stakeholders' own projects; timeliness of Atlantic collaboration, given factors such as provincial differences in program maturity; and appropriate structure for ATKE. It was suggested that there was a need for structure, but caution was expressed as to its extensiveness. Over time, support may build -- perhaps through a series of successful initiatives -- for a more formal structure for telehealth collaboration in Atlantic Canada. As well, ATKE could play a role in providing a link between federal initiatives and grassroots telehealth initiatives. This is an interesting concept for ATKE and other existing organizations to contemplate and make real, in terms of relationships and reporting mechanisms.

Summary of Evaluation Recommendations

In summary, it is recommended that the project partners:

• Keep the focus on what gives people passion about telehealth; i.e., improving access to quality care and decreasing isolation of caregivers;
• Work with grassroots to identify telehealth priorities for clinical applications, education, research, and policy issues;
• Work to integrate telehealth in the Atlantic Canada e-health agenda, in conjunction with HIA and other stakeholders;
• Engage with OHIH and others to explore the long-term potential of a web presence that meets the needs of 'grassroots' telehealth practitioners (as recommended in Sierra Systems Group Inc., 2003);
• Implement the dissemination plan in the immediate future, in order to maintain momentum for Atlantic telehealth collaboration;
• Gather feedback from a broad range of stakeholders on the research and evaluation findings, in order to help determine the level of support for future directions of Atlantic telehealth collaboration;
• Continue to dialogue with HIA;
• Engage a core group of individuals to provide leadership to ongoing collaboration among Atlantic telehealth initiatives;
• Build on low cost, informal mechanisms for communication and information sharing among grassroots telehealth practitioners;
• Build on the value proposition and seek funding for a business case; and
• Consider the value of submitting another proposal to further develop Atlantic telehealth collaboration.

1. Background

1.1 Background and Rationale

Atlantic Canada has some of the earliest, most comprehensive, well established networks and innovative applications for telehealth in the country. The region offers a wide range of models for telehealth in terms of management structure, co-ordination, funding, equipment, utilization, and telehealth applications. Collectively, this diversity, experience, and wealth of knowledge can significantly contribute to the development of a knowledge base for excellence in telehealth services.

While a number of telehealth initiatives have been implemented in Canada in the last five years, there are few studies on the implementation of this technology. As is the case with any new and evolving field, there is a need to review current initiatives, identify progress to date, and begin identifying effective and efficient practices.

While telehealth itself is a mechanism for information and communications technologies to break down geographic barriers, there is little opportunity for telehealth professionals to go beyond their regional and / or provincial boundaries to exchange information, share their individually identified challenges and successes, and discuss issues of interoperability.

At the National Telehealth Interoperability Workshop in February 2001, participants noted that there was a lack of communication between stakeholders with vested interest in all domains of interoperability (clinical / service, technical, operational), which created a barrier. Within the clinical / service domain, lack of knowledge among participants of related activities and isolation were the main reasons cited for this lack of communication. Workshop participants suggested that addressing this issue required both 'top-down' and 'bottom-up' strategies.

It is known that initiatives across Canada face similar issues in the operation, management, and technological implementation of telehealth services (Canadian Society of Telehealth, 2001; Jocelyn Picot, Infotelmed Communications Inc., 2001). Despite the existence of many telehealth projects, there are few established operations, and numerous problems related to integration between initiatives have been reported (Jocelyn Picot, Infotelmed Communications Inc., 2001).

While a number of organizations are defining structures for a coordinated health infostructure regionally and nationally (e.g., Health Infostructure Atlantic (HIA), Health Canada's Office of Health and the Information Highway, etc.), none is currently addressing the issues from a telehealth 'grassroots' level.

This need fit the requirements of Health Canada's Knowledge Development and Exchange Program, an initiative of the Office of Health and the Information Highway (OHIH). Specifically, there was an opportunity to apply for funding under that program. The objectives of OHIH Request for Proposals in 2001 were:

• "To engage stakeholders in undertaking applied research and developing practical and policy solutions related to the implementation of information and communications technologies (ICTs) in health and health care in Canada."
• "What are the best practices or lessons learned regarding...the development or implementation of telehealth projects and / or broader initiatives?"

1.2 Program Funding and Accountability

In May 2001, the IWK Health Centre in Halifax, Nova Scotia and its project partners (see Appendix A):

• the Nova Scotia Telehealth Network, Department of Health, Nova Scotia;
• River Valley Health, New Brunswick (Regional Health Authority # 3);
• Telehealth and Educational Technology Resources Agency (TETRA), Memorial University, Newfoundland and Labrador; and
• the Prince Edward Island Department of Health and Social Services

applied for funding through the Knowledge Development and Exchange Program. Notification of funding was received in May of 2002, which enabled the partners to undertake the applied research project entitled Atlantic Telehealth Knowledge Exchange (ATKE) (Sierra Systems Group Inc., 2003).

The Steering Committee members were responsible for overseeing the project and the process evaluation. Sierra Systems Group Inc. and TecKnowledge Professional Group were contracted to implement the project, and two consultants were contracted to carry out the process evaluation.

1.3 Program Description

The Atlantic Telehealth Knowledge Exchange (ATKE) project was developed to emphasize a 'bottom-up' strategy for breaking down barriers and facilitating communication and information sharing among stakeholders. It involved telehealth service providers and system managers, including: coordinators, provincial directors, and existing committees / forums (e.g., The Children's Telehealth Network Steering Committee and provincial meetings of telehealth coordinators, as applicable).

The project was meant to complement existing regional (e.g., HIA) and national (e.g., Health Canada's OHIH) initiatives. The ATKE Project Charter notes that the intention of the project was to closely align itself with the goals and objectives of Health Infostructure Atlantic and facilitate the sharing of telehealth practices / guidelines in specific specialties such as pediatrics, mental health, nephrology, cardiology, oncology, rehabilitation, and homecare. In addition to the data repository and report, a collaboration management approach that fosters cooperation and knowledge exchange among the project partners and telehealth initiatives throughout Atlantic Canada will be a key deliverable of the project.

The scope of the ATKE project included the following:

• Review of current telehealth initiatives in Atlantic Canada in order to establish baseline information. This review required input from telehealth service providers, telehealth system managers, funders, professional associations, and program suppliers;
• Development of an inventory of telehealth projects and programs within Atlantic Canada, which reflected clinical consultations, health education, and administrative meetings conducted via videoconferencing;
• Identification of currently available baseline information about existing telehealth projects and programs;
• Definition of requirements for ATKE;
• A gap analysis that identified the gaps in information about existing telehealth projects and programs within Atlantic Canada;
• Recommendation of a 'go forward' strategy for developing a data repository of telehealth information in Atlantic Canada; and
• Development of strategic and tactical initiatives for moving ATKE forward.

For the purposes of this project, telehealth activities include: clinical consultations, health education and health administrative sessions conducted via videoconferencing, store and forward applications, web casting or the telephone. The activities do not include telephone advice programs, health information web sites, electronic health records, and health information systems initiatives (unless integrated with telehealth activities or large and significant in nature) (Sierra Systems Group Inc., 2002).

1.4 Vision, Goals, and Objectives

ATKE Project Vision

The vision for the ATKE project was to promote excellence in telehealth in Atlantic Canada, and to contribute to telehealth knowledge and knowledge exchange at provincial, regional, and national levels.

ATKE Project Goals

To achieve this vision, the ATKE project defined the following project goals:

• To initiate the definition and development of a sustainable Atlantic telehealth structure that will foster excellence in telehealth at the provincial / inter-provincial level in Atlantic Canada.
• To model, through the ATKE project process itself, an initial model / framework for Atlantic collaboration in telehealth.
• To leverage 'grassroots' input by involving all telehealth professionals / organizations currently involved in telehealth in Atlantic Canada in the applied research project.
• To produce a research report that captures the results of the ATKE applied research project.
• To complement the work of Health Infostructure Atlantic (HIA) in leveraging Ministerial support for the use of information and communications technologies in improving access to and quality of health care services in Atlantic Canada.
• To capture and synthesize baseline information that can be used to market and promote telehealth in Atlantic Canada.
• To establish a foundation for leveraging external funding opportunities in support of collaborative, pan-Atlantic telehealth initiatives.

ATKE Objectives

1. To establish a collaborative structure (among telehealth initiatives in Atlantic Canada) to facilitate the development and exchange of information on the Atlantic telehealth experience.
2. To establish a repository of baseline information on the Atlantic telehealth experience, the lessons learned, and best practices.
3. To generate and disseminate regionally the findings and knowledge gaps identified from the baseline data.
4. To identify strategies and seek ongoing regional commitment for further analysis, research, and / or policy development in the Atlantic region.
5. To disseminate the experiences and information on telehealth in the Atlantic region to relevant initiatives across Canada.

2. Evaluation Approach and Methods

2.1 Evaluation Approach

The ATKE Steering Committee contracted two consulting firms to conduct a process evaluation of the ATKE project. The Atlantic Evaluation Group and The Quaich Inc. of Charlottetown were contracted to complete a formal, value-based, post-project assessment in order to determine the value derived from the research effort and the potential for the Atlantic Telehealth Knowledge Exchange.

A participatory evaluation approach was used with the involvement of the ATKE project Steering Committee at all stages of the evaluation in order to enhance the relevance and usability of the evaluation results. This participation was meant to ensure that the evaluation approach was consistent with the planning and decision-making needs of the Steering Committee. The Committee provided input to the development of the program logic model and the process evaluation framework, as well as to the development of the data collection instruments in order to ensure that questions were asked in a manner acceptable to participants, and instruments used would result in the gathering of necessary and appropriate information. In addition, the Steering Committee provided input to the final report in order to ensure that the results would be understood in their proper context and the developed recommendations would recognize the constraints of the operating environment.

2.2 Program Logic Model and Evaluation Framework

A program logic model was developed; it described the activities, target populations, outputs, as well as the short-term and the long-term outcomes for each of the four program components: developing a collaborative structure; information sharing / dissemination; knowledge repository; and sustainability. The program logic model included activities completed as part of the funded project and potential future activities in which the project might engage. An evaluation framework was developed and included indicators of success and data collection sources used for a series of evaluation questions under each program component. This evaluation framework served as a guide for the process evaluation. The program logic model and evaluation framework are found in Appendix B.

2.3 Data Collection Methods

Data were collected using the following methods (see Appendix C for a copy of these instruments):

• review of the various documents produced by or for the project;
• attendance at the videoconference presentation of the final report;
• focus group and interviews with Steering Committee Members;
• interviews with stakeholders in all four provinces;
• group interview with HIA members; and
• interview with a member of each of the consulting firms that implemented the project under contract with the ATKE Steering Committee.

The document review included the Sierra Systems Group Inc. project report; the Project Charter; status reports; Steering Committee / project team agendas and action items; evaluation team notes from meetings with the Steering Committee; presentation slides for the project report; and key notices / e-mails sent by the project coordinator.

A semi-structured focus group / interview guide was developed in order to collect input from the Steering Committee members, HIA, stakeholders, and project consultants. The consultant and stakeholder interview guides were reviewed by the Steering Committee and revised based on their input. Three Steering Committee members were interviewed during a focus group videoconference, along with one stakeholder from PEI; one Steering Committee member was interviewed individually over the phone; and one Steering Committee member was interviewed as part of the HIA group interview. The 45-minute HIA group interview was conducted during a regularly scheduled in-person HIA meeting. Seven stakeholders -- one from PEI and two each from the remaining Atlantic provinces -- were individually interviewed over the telephone. The stakeholders included three management persons (two from provider organizations and one representing a health informatics organization) and four representatives of telehealth initiatives. In terms of their involvement with ATKE, six of the stakeholders had participated in an interview; six had attended the workshop; five had completed the questionnaire; two had attended the videoconference presentation of the final report; and one had participated in a focus group. Finally, one consultant from Sierra Systems Group Inc. and one consultant from TecKnowledge Professional Group were interviewed together in an audioconference.

2.4 Analysis and Report Writing

All qualitative data was analyzed for key themes and organized according to the components and questions in the evaluation framework. The evaluation report presents the findings as they relate to the achievement of project objectives; i.e., the supporting factors, challenges, and outcomes of each area of project activity. A summary under each objective reflects on the achievement of project goals and makes recommendations for next steps.

3. Evaluation Findings

3.1 Developing a Collaborative Structure

One of the major areas of activity for ATKE was the development of a collaborative structure for the sharing of information about telehealth:

• Objective 1: To establish a collaborative structure (among telehealth initiatives in Atlantic Canada) in order to facilitate the development and exchange of information on the Atlantic telehealth experience.

To achieve this objective, leadership was required in managing the project and in engaging partners and stakeholders. As these two areas of activity were so essential to the success of the project, the report addresses them separately.

3.1.1 Project Management

The ATKE project was overseen by a Steering Committee comprising the following ATKE partners: the IWK Health Centre in Halifax, Nova Scotia; the Nova Scotia Telehealth Network, Department of Health, Nova Scotia; River Valley Health, New Brunswick (Regional Health Authority # 3); Telehealth and Educational Technology Resources Agency (TETRA), Memorial University, Newfoundland and Labrador; and the Prince Edward Island Department of Health and Social Services, in collaboration with the Queen Elizabeth Hospital in Charlottetown, PEI. The Steering Committee provided overall project leadership and helped the consultants to identify and collect relevant data (see Figure 1).

A project coordinator was assigned by the IWK Health Centre and ATKE partners to help guide the project, act as a liaison with the consultants, supervise the work of the consultants and sign off deliverables, serve as first contact for resolving routine issues, and report project progress to the project sponsors and to Health Canada. The project coordinator worked under the direction of the Steering Committee.

In September 2002, having secured funding from Health Canada, the ATKE Steering Committee contracted Sierra Systems Group Inc. and TecKnowledge Professional Services to manage the ATKE project and to carry out the research / consulting tasks. The role of project manager was assumed by consultants from Sierra Systems Group Inc. The project manager was responsible for managing the day-to-day activities of the project; i.e., project planning, guiding the work of the consultants, managing project resources, communication, and meeting timelines.

Figure 1: ATKE Project Organizational Chart
(Sierra Systems Group Inc., 2003)

Stakeholders with a wide range of backgrounds and areas of expertise also played a key role in the project, providing content expertise and confirming information on the status of telehealth in their region.

To carry out the research, a team of four Telehealth Consultants was contracted. The team worked in conjunction with the Steering Committee to develop research and process tools; to collect and synthesize telehealth data; to conduct the workshop; to establish baselines; to carry out the gap analysis; to develop strategies to move the project forward; and to write the final report. The Telehealth Consultants worked closely with the Technical Consultants, who reviewed the gap analysis findings and identified the data repository options.

In order to ensure that everyone involved in the project shared a common understanding of why the project was being conducted and how the project research findings were going to be utilized, a Project Charter was developed. The Project Charter outlined the roles and responsibilities of the Atlantic Telehealth Knowledge Exchange (ATKE) partners, the Steering Committee, the project coordinator, Sierra Systems Group Inc., and TecKnowledge Professional Services. It also outlined the project scope, project organization structure, roles and responsibilities, project management model, reporting mechanisms, project phases, work plan schedule, budget, and any major changes to the project.

To help manage the project, the project manager employed the Sierra Systems' Project Management Guidelines and formal management model. The Guidelines provided protocols for status reporting, change management, issue management / resolution, decision management, and risk management (Project Charter, 2002).

In order to ensure that all members of the project team were kept up to date on project progress, an internal communications strategy was devised, as outlined in the Project Charter. To keep the Steering Committee aware of the work being carried out by the consultants, bi-weekly status reports were provided using the Sierra Systems' standard form, and bi-weekly project status meetings were held among the consultants, project coordinator, and the Steering Committee. The telephone and e-mail were used to communicate between the bi-weekly meetings.

Between the bi-weekly project status meetings, the ATKE Steering Committee conducted bi-weekly Steering Committee meetings. These meetings included the project coordinator and, on occasion, members of the consultant team. Following each meeting, action items were distributed to all Steering Committee members via e-mail. Additional communication between Steering Committee meetings occurred via e-mail or the telephone.

Effectiveness in Achieving Objectives

A number of supporting factors and challenges to project management were identified during data collection for the evaluation.

Supporting Factors

Supporting factors included funding, project management model, and characteristics of the Steering Committee.

Funding

Overall, the Steering Committee noted that the substantial funding for the project provided by Health Canada was critical to moving the project forward in terms of data collection, communications, data analysis, etc. This funding also leveraged extensive in-kind contributions from the partners in each province (for Steering Committee members' time, travel expenses, etc.). Without the in-kind support, it would not have been possible to build so strategically on the work already begun by the telehealth champions in each province.

Project Management Model

The Steering Committee noted that the development of a formal project management model helped to define the project and keep it on track. This was the first time that such a model had been used in Atlantic Telehealth collaboration, and it provided the project with the structure required to move the project forward in an effective manner.

"The success of ATKE in terms of the 'how' from my perspective was the formal project management structure. It was the first time I was involved in a four-province telehealth project that used this approach...I think we can mirror this as we step in to do other initiatives... We had a process in place where we had to formalize all of our project phases from start to finish -- it had a structure around it that allowed us to keep on target."

(Steering Committee focus group / interview)

"The structured management approach helped us build collaboration among Steering Committee Members."

(Steering Committee focus group / interview)

The function of the project coordinator was also seen as important in keeping the process on track and in maintaining communication among ATKE participants.

"...I really think the project coordination from [Project Coordinator] was key in keeping the project moving forward and on track...without that, we probably would have let things go until we had time to get at them -- she really kept the process moving..."

(Steering Committee focus group / interview)

Characteristics of Steering Committee

In addition to acknowledging the value of the structured management approach adopted by the project, the consultants and the Steering Committee attributed much of the success in managing change, engaging key stakeholders, and overcoming obstacles to Steering Committee members' level of commitment to telehealth and their experience in the health sector.

"The history of Atlantic collaborations isn't good...there tends to be a lack of trust, and fear of other provinces taking over...that didn't happen with us and I think one of the reasons it didn't happen is because we had a good process... A lot of this collaboration gets right down to the personal level -- did I enjoy working with this team in a collaborative fashion? If that fails, nothing else will happen...we had a good Steering Committee that worked well together and we had a good process."

(Steering Committee focus group / interview)

"...I think the fact that we all had project management skills and leadership prior to coming into this project really helped -- it wasn't like we had to develop a whole bunch of new skills to participate in the project, we were already pretty skilled..."

(Steering Committee focus group / interview)

"The commitment and the feeling of everyone involved that there was a need to share and learn from each other."

(Steering Committee focus group / interview)

Stakeholders were also complimentary of this group, commenting on the members' drive and enthusiasm as factors to success.

"The ATKE team were persistent and hung tough. They found areas where they could make progress...and have done it well. They under promised and over delivered."

(stakeholder interview)

"Well I think the...representatives from the four Atlantic provinces are hugely committed individuals."

(stakeholder interview)

Challenges

The challenges included time factors / delay in funding; change in project team members; and vision of the final product.

Time Factors / Delay in Funding

During the focus group session, all Steering Committee members noted that having to deal with the heavy time commitment required by the project, in addition to their already full work schedules, was a challenge. This problem was exacerbated by the one-year delay in receiving project funding from Health Canada. Originally, Steering Committee members had considered and planned for the project time commitments and adjusted their work schedules accordingly. However, by the time the funding came through, those time commitments were no longer factored into their work schedules. This meant that Steering Committee members had to rearrange their schedules as best as they could and add the project work to their other work responsibilities. The consultants commented on this issue as well.

"When we applied for the funding, I had put aside time for this project...when the funding came a year late, I had to rearrange my schedule, and try and make time for it among my other priorities and commitments...despite this barrier, we did make the project a priority."

(Steering Committee focus group / interview)

Steering Committee members also noted that the one-year delay seemed to result in a loss of momentum. With so much time passing between submitting the project proposal and receiving the funding, they felt it was like starting all over again.

"...I think we lost momentum because of the one-year lag in funding. It almost felt like we had to start again -- had to pull it out and blow the dust off of it..."

(Steering Committee focus group / interview)

In addition, the delay in funding affected participation in the project. By the time funding was received, some participants who were originally involved in writing the proposal had moved on to other employment and were no longer available to participate in the project.

Change in Project Team Members

The turnover in project team members throughout the course of the project presented a challenge. Partway through the project, the original consultant from Sierra Systems Group Inc., who acted as project manager, had to be replaced by another consultant, due to changes in the Sierra company structure. Although Steering Committee members were happy with the work done by the replacement project manager, they felt that the change -- partway through the project -- was disruptive and resulted in their having to take on more work and responsibility; i.e., they had to dedicate more time to communicating / conferring with the consultants, guiding the process, carrying out the 'legwork' of the project, and helping with the report writing. They also felt it forced them to take more responsibility for the project's strategic direction.

In addition to the change in project manager, two of the Steering Committee members were also replaced. This was a challenge as it took time for the new members to 'get up to speed' on the project, build rapport with the other members, and develop a level of comfort in contributing to the project.

"...probably the barrier for me...was the change over of representatives part way through the project. I tried to work with [the original Steering Committee member] to keep things going when I came in, but it was a bit of a challenge for me coming into the project so late -- it was a challenge to stay focused and keep everything tied together because I really didn't know what my expectation was because I wasn't involved in the initial part of the project..."

(Steering Committee focus group / interview)

This change was also an adjustment for the remaining original Steering Committee members. They had to take the time to introduce the new Steering Committee members to the project and make sure that they understood their roles and responsibilities, and they had to readjust to working with new individuals.

Vision of Final Product

As the project unfolded, the vision of a structured organization and knowledge repository was challenged through the consultative process. This resulted in a change to the plan and expected outcomes, and made it more difficult for some participants to measure success by attainment of outcomes.

"A little bit of difficulty in sorting out the vision -- what the final outcome would be...I struggle with this. Where is it going to lead us...where are we going? Even after all this work, what is the final product now?"

(Steering Committee focus group / interview)

"I still feel a bit of dissatisfaction that we didn't come a bit further than that...but I think that comes with some of the changes we had to make, and the fact that we started a year late, and had a short time frame once we did get funding. The process was rushed. These things did impact on the results..."

(Steering Committee focus group / interview)

Outcomes

Success indicators for the project were identified during the consultation process between Steering Committee members and the evaluation team. The success indicators included: provision of resources to initiate the project; establishment of a Steering Committee; implementation of a project management model; development of a partnership with a qualified consulting company; development of terms of reference and expectations; development of a project charter; regular / open internal communications between project team members; and records of communication.

Based on the review of documentation and interviews with the Steering Committee and the stakeholders, the ATKE project was successful in all areas of project management. This is attributed to: a) the adoption of a sound project management model provided by the consultants; and b) the committed and experienced members of the ATKE Steering Committee.

These factors enabled the project to overcome unforeseen events such as the change in consultants and committee members, and time delays. In fact, the consultants interviewed noted the degree of development of the Steering Committee as positive for collaboration among Atlantic Canada telehealth initiatives. They felt that a different kind of working relationship evolved for the Steering Committee throughout the process of implementing ATKE; a relationship which, along with the high degree of commitment among Steering Committee members and the recognition of differences in each province, was seen as providing a strong foundation for future collaboration.

3.1.2 Increasing Awareness and Engaging Partners / Stakeholders

A key part of building the collaborative structure was working toward increasing awareness of the project among partners / key stakeholders in Atlantic Canada who are involved in telehealth and engaging them in the project. In order to help bring about this awareness, the ATKE project team developed a communications plan. The objectives of the plan were: to ensure that all stakeholders and partners with an interest in telehealth are aware of the research project and have an opportunity to participate and to benefit from lessons learned; to use communications to help break down barriers between and among telehealth providers; and to communicate to the public, to health sector stakeholders, and to private partners the process of the research and, ultimately, the findings.

Throughout the course of the project, several activities were carried out in order to communicate with and engage stakeholders. One of the key strategies was to have Steering Committee members make initial contact with identified stakeholders, in order to introduce them to the project and to set the stage for future communications. Overall, 98 stakeholders from across the Atlantic Region and with various backgrounds and expertise in telehealth were approached by Steering Committee members and forwarded information on the project. The information explained the rationale for the project, identified the ATKE partners, and outlined the project objectives.

The consultants from Sierra Systems Group Inc. then contacted the stakeholders individually by e-mail or by telephone and asked them to provide their input by completing a questionnaire or participating in an interview.

Of the 98 stakeholders identified by the Steering Committee, a total of 74 (76%) took part in the project by completing a questionnaire or participating in an interview (Sierra Systems Group Inc., 2003). A good cross-section of respondents with various backgrounds and expertise was represented; telehealth coordinators, telehealth program managers / directors, telehealth clinicians, program managers / directors, academics, government officials, CEOs / VPs, professional associations, IT service professionals, educators, and private sector providers.

Another activity carried out in order to increase awareness of the project and provide opportunities for stakeholders / partners to engage in the work was the day-long workshop hosted by the Atlantic Telehealth Knowledge Exchange (ATKE) Steering Committee on February 4th 2003 in Halifax, Nova Scotia. Overall, of the 20 invitees, 17 attended the workshop and one additional person participated via a videoconference; two persons were unable to attend, due to the weather. Participants included ATKE Steering Committee members, a number of stakeholders who had participated in the data gathering process, and others invited by the ATKE Steering Committee.

The workshop provided participants with opportunities to network, share ideas, and explore options for ATKE.

Throughout the course of the project, Steering Committee members also worked on increasing awareness of the project and engaging stakeholders by informally introducing / discussing the ATKE project with work colleagues and associates at various meetings, professional gatherings, and via e-mail communications.

"I would be going to meetings and at those meetings would informally start talking about ATKE and the work we are doing -- it gave us a voice and an identity. People became aware of us..."

(Steering Committee focus group / interview)

"I think the way I went about marketing or advocating for ATKE was to forward on e-mails to the telehealth people in the three health authorities that have telehealth coordinators...I always sent them along any information around ATKE that I thought was of interest, and told them that they could contact me if they wanted to learn more about anything...another link made was with the Department of Health. It really got started with the workshop -- that's where formal contact was made...I also presented an overview of the project to our key provincial IT group, and am hoping to be on their agenda in May to present on the ATKE results..."

(Steering Committee focus group / interview)

In addition, several articles were written and released throughout the course of the project. These articles provided an overview of the project, outlined the project purpose, identified the organizations involved, and summarized the project objectives. Articles published / released included: a November 19, 2002 news article which introduced the project; an article in the March 2003 issue of the Canadian Nurse, which profiled Chris-Anne Ingram from the IWK Health Centre; as well as an article in the Medical Post.

Effectiveness in Achieving Objectives

A number of supporting factors and challenges to increasing awareness and engaging partners / stakeholders were identified during data collection for the evaluation.

Supporting Factors

Supporting factors included personal communication with stakeholders, timing of the project, legitimacy of telehealth through funding for ATKE, and building on existing networks.

Personal Communication with Stakeholders

It was noted in the Sierra Systems Group Inc. (2003) report and by the project consultants and Steering Committee members that the initial contact by the Committee members with the stakeholders helped to increase awareness of the project, build rapport with stakeholders, and elicit support for the research being conducted. Project consultants and Steering Committee members felt that this approach had a major impact on stakeholder response rates and played a key role in introducing stakeholders to the project.

"...If we hadn't managed in the way we did, we wouldn't have gotten the response rate that we had...the way we managed it was we gave it a personal touch -- each of us personally contacted stakeholders before the consultants contacted them. We were opening the door for the interview or survey to take place..."

(Steering Committee focus group / interview)

Timing of Project

Given the global shortage of health professionals and the evolving importance being given to e-health throughout the country, the timing was right for telehealth to have an increased profile in provincial discussions. Given the momentum for regional collaboration on many fronts, Atlantic collaboration on telehealth was strategic. The Sierra Systems Group Inc. (2003) report, project stakeholders, and Steering Committee members noted that approaching the project from an Atlantic perspective was highly appropriate. This perspective was captured via many sources. Health Infostructure Atlantic (HIA) noted the added value in coming together in a formal way as four Atlantic provinces.

"Conceptually taking it to the Atlantic Region, a small part of country with a lot of innovative good work makes good sense. There are a lot of ideas that we are not adequately sharing due to a lot of disconnect with the larger systems -- health administrators and hospitals come to mind. There are pockets of knowledge but we have very few ways of assessing and identifying where and who has the learnings."

(stakeholder interview)

"Things have changed dramatically in New Brunswick since we started this project. I'd love to say it's all because of ATKE but I'm sure it isn't. For example, we just had a new e-health office announced that's going to have a telehealth arm to it."

(Steering Committee focus group / interview)

Legitimacy of Telehealth Through Funding for ATKE

The view expressed by the Steering Committee, by stakeholders and HIA is that telehealth has been on the margins of health care delivery and has not been included in the strategic plans of most health care organizations in Atlantic Canada. Having funding for ATKE allowed the development of a structure that gathered information about telehealth initiatives and served as a voice.

"ATKE became an identity -- it was something that people were able to reference. I think it's the identity and the communication back into the Ministries of Health -- it was a collective voice."

(Steering Committee focus group / interview)

"...being funded by Health Canada and supported by the provinces helped legitimize it, gave some validity to the group -- it wasn't just [individual Steering Committee member] and some informal group."

(Steering Committee focus group / interview)

HIA noted that because of the presence of ATKE, the issue of how to integrate telehealth was brought forward for discussion. HIA also observed that ATKE had done a good job of structuring the initiative and creating a group, using the funding from OHIH. The key question from the HIA perspective is how to move the telehealth agenda forward and how this agenda fits in with other priorities.

Building on Existing Networks

Stakeholders and Steering Committee members described previous involvement with other provinces via a range of academic research projects (for education) or via informal networks and commented on how ATKE helped build on these networks.

"We were able to build on the collaboration of the 4 to 5 organizations, which allowed others to see that there was a lot of activity going on..."

(Steering Committee focus group / interview)

Challenges

Challenges included the varied stages and structures of provincial development of telehealth; varied interpretations of telehealth; uncertainty regarding fit with existing 'system'; and time, money, and human resources.

Varied Stages and Structures of Provincial Development of Telehealth

One of the main challenges identified by the Steering Committee, by project consultants, one stakeholder, and HIA was the different stages of development of telehealth in the four Atlantic provinces, as well as the different structures or models of telehealth in each province. For example, when the project began, New Brunswick had a regionally-based telehealth program that was not province-wide; PEI was lacking a province-wide program; Nova Scotia had developed, as part of a publicly-funded system, a provincial program with regional responsibilities; and Newfoundland had a broader, well established program, developed on a cost recovery basis, which extended beyond the boundaries of health.

"...in New Brunswick we didn't really have a formal way to communicate with our Departments of Health when we first started this initiative because there wasn't the same presence in our Ministry -- we didn't have a key person who had a telehealth portfolio..."

(Steering Committee focus group / interview)

"One of the things that is important to note is the telehealth initiatives in each of the four provinces are at different levels of maturity and delivered in a number of different ways."

(Steering Committee focus group / interview)

Varied Interpretations of Telehealth

Another challenge noted by the Steering Committee was the fact that many programs / projects were not recognized as 'telehealth' projects. This created some difficulty in initially securing the involvement of some stakeholders and it may have resulted in skewed results. The sharing of a definition of telehealth in advance was considered very important in this regard. Nevertheless, even though a very detailed definition was provided to stakeholders, there still seemed to be some confusion about the term and about what stakeholders identified as 'telehealth.'

"...the other barrier I experienced from this region was the fact that a lot of the initiatives going on in telehealth weren't being recognized as telehealth. People were identifying them as a certain program, and weren't making the connection that they were actually telehealth initiatives -- for example, it's a telehealth activity even if it's called obstetrics rounds...we had to do a bit of explaining and exchanging of knowledge to get people on board, and make them aware that what they are involved in is telehealth...we had to give people a definition of telehealth, and expand more in the intro letter to get them engaged. There were challenges in trying to get people to think a bit differently, and look beyond what they saw around the activities..."

(Steering Committee focus group / interview)

"...telehealth is not recognized within the provincial health mandate, so is not clearly defined -- people are involved in a lot of initiatives and activities but don't think of them as telehealth...I think when people think about telehealth, they think of things like robotic surgery and such -- they don't think of all the other things they're involved in that are actually considered telehealth...I think we probably got a skewed result because people were not clear on what we meant by telehealth..."

(Steering Committee focus group / interview)

One stakeholder remarked that even within the telehealth field there are issues concerning the collection of information about current initiatives. For example, information requested in the survey was not sensitive to tele-homecare, where simpler technology is often better. The survey did not include fields for entering such data. HIA identified that telehealth is difficult to define and often cannot be viewed as a separate stream of activity.

Uncertainty Regarding Fit with Existing 'System'

It was noted in the interview with HIA that telehealth has been viewed as a totally separate stream of activity, when if fact it is just another mode of delivering services. The existence of the ATKE project has raised awareness of the need for an inter-provincial process for advancing telehealth. The need to consider where telehealth belongs in each provincial organization should be addressed, even if it is tackled differently in each province.

Time, Money, and Human Resources

Consultants felt that part of their role was to focus on higher-level issues, in order to help move the Steering Committee and the project forward. They remarked that time, money, and human resources were barriers to their greater collaboration in ATKE. They noted that Steering Committee members had to take time, over and above their already busy work schedules, to participate in the project. The Steering Committee members themselves also identified this as an issue, given the time commitment that ATKE required in comparison to provincial responsibilities.

"Telehealth initiatives were built in the lean years of government so we tend to be pretty small on the ground in terms of resources so our time is valuable and we do have to sit back and...ask how, after all the time that has been committed to it, what value has it brought at the end of the day."

(Steering Committee focus group / interview)

Outcomes

Based on the data collected through the evaluation, it would appear that the ATKE project was successful in engaging stakeholders and increasing awareness of the project. Success indicators used by the evaluators to review the data were: identification of potential partners / key stakeholders; contact with potential partners / key stakeholders to introduce them to the project and encourage their support and input; development of a communication plan to keep partners / key stakeholders apprised of progress; and hosting a workshop. Documentation indicates that all of these activities were remarkably successful.

Awareness and Information Sharing Among Stakeholders

The Sierra Systems Group Inc. (2003) report indicated that throughout the data collection process, stakeholders were interested in participating in the project and learning more about telehealth. It was also noted that the project served to heighten awareness about telehealth in Atlantic Canada and demonstrated, through the inventory process, that projects and programs were willing to share knowledge, information, and data about their telehealth experiences. The report also indicated that stakeholders felt that having access to telehealth knowledge would support them in developing or modifying their telehealth projects and programs, which would help to improve care.

The Steering Committee focus group / interview supported these positive findings. There was a sense that ATKE was useful in sparking discussion among stakeholders in the provinces and promoting interest in telehealth.

"I guess I'd have to say that I considered myself a messenger for ATKE...we talked with government and our colleagues, and tried to get the word out... It really sparked interest in the project -- people started asking questions about the project and Atlantic connections. It was good that way."

(Steering Committee focus group / interview)

Stakeholders who were interviewed also commented on the success of the Steering Committee in accessing a broad range of stakeholders and engaging new people in the telehealth network. One stakeholder observed that the project respected the different developmental stages of telehealth in each province, welcomed people, and brought them forward. Another stated that prior to the involvement with ATKE, there had been limited contact with other Atlantic initiatives.

"Usefulness was the opportunity to meet together at workshop for energy exchange. It demonstrated success in bringing together a wide range of stakeholders and let people come from where they were, and nurtured people on board."

(stakeholder interview)

"ATKE put me in contact with telehealth colleagues that I hadn't been in contact with before -- put me in contact with each of the telehealth leaders in each of the jurisdictions."

(Steering Committee focus group / interview)

"ATKE was successful in getting folks together around telehealth, which has not happened in the past -- talking about issues -- putting names to faces -- a success even if it goes no further."

(stakeholder interview)

"Not sure why I don't know what is happening in Atlantic Canada -- but I don't. We are not resourced to be out digging for information."

(stakeholder interview)

Four stakeholders indicated that as a result of their involvement with ATKE they had made contacts with other telehealth initiatives. One stakeholder noted that it helped foster government involvement. Four stakeholders expressed the view that there is a large potential for collaboration in the area of telehealth among the Atlantic provinces.

"Taken good steps to raise the profile of telemedicine -- just keep it up!"

(stakeholder interview)

Establishing a Collaborative Structure

The issue of whether the work carried out through ATKE to develop a structured way to link Atlantic Telehealth initiatives was useful or not was met with mixed reviews ranging from enthusiasm to hesitancy.

Progress and Potential

The consultants noted that more links were made with Ministries of Health and other organizations, and that while the project did not achieve the development of a formal collaborative structure, there was much value in the work completed to date in building collaboration in Atlantic Canada. Overall, the consultants felt that the project's first steps in legitimizing Atlantic collaboration on telehealth were successful.

Steering Committee members stated that although collaboration among its members was taking place prior to ATKE, the project confirmed their vision that bringing together members of telehealth initiatives in a forum to share information and perspectives was important. They saw stakeholders gravitate toward ATKE, as there was a lack of other formal structures. Having a place for sharing information -- without 'reinventing the wheel' -- was helpful.

Six of the seven stakeholders interviewed confirmed the observations of the Steering Committee and thought that the development of ATKE was a useful undertaking. One stakeholder noted that it is very important that individuals involved in telehealth in the four provinces continue to meet, talk, and share. Three saw collaboration with other initiatives as very useful for getting up-to-date on current activities, sharing similar experiences, and discussing similar issues. Two stakeholders noted that ATKE could make information more accessible -- thus avoiding tedious searching -- and could lead to the sharing of information without 'reinventing the wheel'. From a user perspective, one person commented on the fact that a structured organization could play a lead role in fostering the development of standards for interoperability, and two other persons commented on the role that ATKE could play in decreasing isolation.

"A network for telehealth is important since if you are reporting to traditional structure, it is difficult and very isolating. The notion of connecting with people who have enthusiasm for what you are doing is essential to sustain your energy."

(stakeholder interview)

In addition, stakeholders who held management roles saw ATKE as playing a strategic role in Atlantic Canada.

"With our small population we should be killing ourselves finding ways to collaborate and partner, which is why I supported ATKE."

(stakeholder interview)

"There is no downside to trying to get four provinces to cooperate and share. The chances of getting federal funding for some of these initiatives are tremendously enhanced if in fact you have a Maritime or Atlantic Canada collaboration in place."

(stakeholder interview)

"All kinds of potential for collaboration. When one surfs the net you see lots going on in Europe and all around the globe. We are all headed down the same path -- lots of opportunities economic and otherwise if we were to share."

(stakeholder interview)

Providers of telehealth identified numerous opportunities for Atlantic collaboration, and had suggestions for how to best structure these efforts.

"Potential possibilities are endless...we must deal with cross jurisdictional licensure. Nurses work within their own provinces; physicians need to deal with inter-provincial licensure and reimbursement issues."

(stakeholder interview)

"The opportunity is tremendous if I was a new employee with an empty desk. There is a great opportunity to network. I have some resistance to doing it because you get drawn into more things then you can afford to become involved with because of all of the other obligations that are present."

(stakeholder interview)

Formalized Structure -- Questions

During the evaluation, expressions of hesitation regarding a formalized structure were heard from HIA representatives, some Steering Committee members, and two stakeholders. The concerns ranged from lack of readiness to commit, to the need to establish a strategic fit with other telehealth initiatives, and to questioning the wisdom of becoming too large and bureaucratic. It was noted that there is a need to assess how to move forward on the telehealth agenda and whether that requires a formal telehealth structure or Steering Committee, and how telehealth fits into overall priorities.

"The next step I put forward with some caution, but there is a need to move from good will to something more structured. We do not need a super structure."

(stakeholder interview)

"I think it is a great goal to try to reach the Atlantic Canada side of things. I think it is a little early yet though to be looking for bucks from different governments to throw into a pot. I'm not so sure we're there yet. I think it's tremendously important that the individuals with tele-health in four provinces continue to meet, continue to talk, and continue to share. There is no doubt in my mind that I believe that a more formalized process down the road will help all of us."

(stakeholder interview)

"With four different provinces being at different levels of maturity with telehealth it is premature to talk about a formalized structure for ATKE. At the same time I see value as inter-provincial colleagues to continue to share in bringing our (provincial) networks forward."

(Steering Committee focus group / interview)

The suggestion was made that effort might be better aimed at constructing a Canadian network.

"...important to connect with colleagues and learn from colleagues and depending on the issue I might learn more from provinces outside ATKE."

(Steering Committee focus group / interview)

The above observations, obtained from stakeholders and Steering Committee members during the evaluation, support the conclusion of the Sierra Systems Group Inc. (2003) report, which recommended that ATKE "align with regional service delivery initiatives" rather than immediately establish an operational model (see Appendix D). That recommendation was based on focus group discussions held during the workshop, in which participants noted that building a case for ATKE was more appropriate and more realistic than focusing on developing an operational model.

The key issues discussed by workshop participants who supported that recommendation included:

• Although the evidence presented supported the notion of developing ATKE, workshop participants generally felt that ATKE needed to fit within the context of other initiatives, either planned or already underway, within the Atlantic region.
• The role of ATKE in relation to Health Infostructure Atlantic (HIA) was discussed. Participants were aware that HIA provided a letter of support for the ATKE project, that it was receiving regular updates on its progress and would be given a presentation of its results. In addition, the ATKE Steering Committee included one member from HIA. Workshop discussions focused on the evolving role of HIA and the potential fit of ATKE with HIA.
• Developing a business case for ATKE was also discussed during the workshop. Participants identified the need to build a case for ATKE that would demonstrate its value to the provinces.
• It was clearly identified during the workshop that ATKE should proceed from both strategic and tactical perspectives, and attention was given to identifying which strategic and tactical initiatives may be appropriate. After the focus group discussions, a general discussion was held in order to explore potential funding sources for the continuation of the ATKE project. The funding sources discussed included universities, transition funds, federal and provincial governments, foundations, pharmaceutical companies, etc. In addition to providing an opportunity to collect information and explore options, the workshop also provided an opportunity to increase awareness of the project, build rapport with stakeholders, strengthen collaborations, and increase buy-in.

Recommendations from this workshop set the course for a change in direction of the project. Based on the workshop findings, Steering Committee members determined that there was a need to step back and build a more strategic buy-in before moving to an operating model, and that it would be in the best interest of the project to shift the emphasis of the project work from identifying an 'operating model' for ATKE to identifying a 'go forward strategy', with appropriate linkages. The strategy proposed was meant to address:

• issues relating to 'authority and sanction' of such a body;
• HIA alignment;
• value proposition; and
• next steps (both strategic and tactical initiatives).

3.1.3 Summary and Recommendations Re: Developing a Collaborative Structure

Health Canada funding for ATKE and the commitment of partners and telehealth champions in four Atlantic provinces increased the profile of telehealth in Atlantic Canada by having ATKE become the focal point and the voice of 'grassroots'; i.e., individuals delivering telehealth services. A large number of consulted stakeholders were more than willing to participate in ATKE, in spite of the demands on their time. As a result of this effective effort to develop a sustainable structure for exchanging telehealth information in Atlantic Canada, many lessons are to be learned.

Lessons Learned

A key lesson learned is the importance of adequate funding, sound project management, communication and, most importantly, having passionate champions to drive the project through to completion. In an ideal world, there would be consistency of the project team from start to finish; however, in most projects, this is not the case. A lesson learned from this factor is the importance of building into the project a method for managing delays, change, and conflict. Another lesson learned is that it is helpful to revisit the project vision and repeatedly measure progress against initial goals and objectives. This would be of particular value when engaging new players in the project vision.

Additional lessons learned are the importance of knowing the right time to proceed and building on existing relationships. This latter lesson is imperative to the success of any project which aims to be sustainable. ATKE also teaches tolerance and respect for diversity of stakeholders and partners, celebrates common elements, and embraces people wherever they may be. Together, these lessons demonstrate exemplary project leadership.

In tackling the challenges of meeting the objective of creating a sustainable structure, ATKE also teaches about the need to acknowledge the problems and issues inherent in embarking on any project which will affect an existing system. Telehealth's lack of profile, clarity, and fit within the health system, and the ways in which these issues are resolved will provide lessons for other pioneer initiatives in the future.

Recommendations

The Sierra Systems Group Inc. (2003) report outlined specific strategic next steps for ATKE (see Appendix D). Evaluation findings have, for the most part, been supportive of these recommendations. Having been made simultaneously with the evaluation, some recommendations are already being acted upon. It will be important for ATKE to consider new developments in moving forward with HIA.

In light of the above, and given the definitive need for grassroots support and information sharing, ATKE should continue to address its needs and further the development of a sense of community and identity for the individuals involved.

Specifically, it is recommended that the project partners:

• Keep the focus on what gives people passion about telehealth; i.e., improving access to quality care and decreasing isolation of caregivers;
• Work with the grassroots to identify telehealth priorities for clinical applications, education, research, and policy issues; and
• Work to integrate telehealth into the Atlantic Canada e-health agenda, in conjunction with HIA and other stakeholders.

3.2 Knowledge Repository

One of the primary aims of the ATKE project was to establish a repository of baseline information on Atlantic telehealth initiatives and experience.

• Objective 2: To establish a repository of baseline information on the Atlantic telehealth experience, lessons learned, and best practices.

The key activity carried out in order to move this objective forward was research focusing on:

• the degree to which stakeholders felt that various knowledge, information, and data elements should be included in a telehealth data repository,
• their current level of access to telehealth information, and
• their preferred methods of accessing telehealth information;
• the types of elements that exist in their telehealth project / program, and the availability of information on that project / program;
• the requirements to develop an operating model for ATKE; and
• gaps in telehealth knowledge, information, and data.

3.2.1 Conducting the Research

Research Design

Before the research could be carried out, a research design process had to be developed. Both the Steering Committee and the consultants were involved in this process. The first step in the research design process was determining the scope of the research. Although a number of key elements to be researched were identified and discussed, the project time frame would not allow for all of them to be explored. It was determined that the elements most required for ATKE were knowledge, information, and data. The methods used to collect information on these elements included: a key stakeholder questionnaire, key stakeholder interviews, and a literature review. One data collection instrument was developed for both the questionnaire and the interviews, and was designed to: collect demographic information; identify the degree to which stakeholders felt that various knowledge, information, and data elements should be included in a telehealth data repository; determine how stakeholders currently access telehealth information and their preferred methods of accessing telehealth information in the future; identify which elements currently exist within their telehealth project or program, and in what form the information exists.

Research Team

A team of four telehealth consultants was contracted to carry out the research work. The team was responsible for collecting, synthesizing, and analyzing the telehealth data, conducting a workshop, establishing baselines, carrying out the gap analysis, developing strategies to move the project forward, and writing the final report. The telehealth consultants worked closely with the technical consultants, who reviewed the gap analysis findings and identified the data repository options.

Data Gathering Process

The Steering Committee selected stakeholders for participation in the study on the basis of their expertise, experience, and / or interest in telehealth. In addition to identifying stakeholders, the Steering Committee decided which stakeholders would receive questionnaires only, and which would be invited to participate in an interview with the consultants. In some instances, the ATKE Steering Committee members met with additional stakeholders within their organizations in a focus group format, in order to gather further information.

To help ensure a good response rate, it was decided that Steering Committee members would make the initial contact with stakeholders and send them information, via e-mail, that would introduce them to the project, identify the project partners, and outline the project objectives. Following the initial contact made by Steering Committee members, the consultants e-mailed or telephoned each stakeholder and invited him / her to participate in the research.

The data gathering process was completed between mid-December 2002 and mid-January 2003. Of the 98 stakeholders identified by the ATKE Steering Committee for completion of questionnaires or participation in interviews, a total of 74 (76%) responded. The stakeholders who responded represented a wide range of professions; telehealth coordinators, telehealth program managers / directors, telehealth clinicians, program managers / directors, academics, government officials, CEOs / VPs, professional associations, IT service professionals, educators, and private sector providers. Overall, the consultants and the Steering Committee indicated that the stakeholders were generally accessible and willing to participate in the research; although some declined to participate in an interview, most agreed to complete the questionnaire.

The data collected through questionnaires and interviews provided a great deal of valuable information on the telehealth needs of stakeholders and the types of telehealth initiatives currently underway in Atlantic Canada. When asked about their telehealth needs, the majority of stakeholders indicated that all the elements outlined in the data collection tool were important and should be included in a telehealth data repository. The top three preferred means of accessing telehealth information were electronic, via newsletters, and meetings. The top three preferred means of accessing electronic telehealth information were the web, a data repository, and a portal.

As regards the types of telehealth initiatives currently underway in Atlantic Canada, information was collected on 36 different projects / programs (see Appendix E). It was determined from the assembled information that most of these telehealth programs / projects contain information on: lessons learned; guidelines; policies; procedures; evaluation programs; needs assessment tools; education programs; quality programs; patient perspective; key contacts; roles and responsibilities; proposals; clinical consults; and case conferences. Unfortunately, even though stakeholders were willing to disclose information on their projects / program, much of the information was not in a form that could be easily shared or added to a repository. It is also important to note that the inventory only included projects that are currently underway and ongoing programs. It did not capture information on completed projects or discontinued programs.

In addition to the questionnaire and interviews, a literature search was conducted in order to collect information on telehealth initiatives in Atlantic Canada. To obtain this information, a web search was conducted and Modern Language Association (MLA) web search guidelines were followed. This literature search provided valuable information on various telehealth initiatives in Atlantic Canada. Overall, forty references were found and documented.

In addition to collecting information on the key elements (knowledge, information, and data), the research design also included a means of collecting information on the requirements for an operating model for ATKE. This was achieved by means of a workshop where Steering Committee members and key stakeholders discussed what would be required to make ATKE successful from the perspective of organizational structure, human resources, communication / dissemination, and data repository. Participating in this workshop were Steering Committee members, a number of stakeholders who had taken part in the data gathering process, and others invited by the Steering Committee.

The workshop began with introductions of the Steering Committee, followed by a presentation which described: the origins of the ATKE initiative; an outline of historical milestones; the ATKE project objectives; an overview of the organizational structure of the project; the phased approach to the project; project timelines; data collection methods; progress to date; and an overview of significant results. Following the presentation, workshop participants were divided into two focus groups. The purpose of the focus groups was to discuss the requirements for the Atlantic Telehealth Knowledge Exchange. Participants were asked to consider requirements, to attempt to prioritize those requirements, and to identify any key issues or obstacles.

The workshop resulted in a wealth of discussion, which focused on building the case for the Atlantic Telehealth Knowledge Exchange, rather than developing an operating model. In addition to the focus group discussions, a general discussion was held in order to explore potential funding sources for the continuation of the ATKE project; e.g., universities, transition funds, federal and provincial governments, foundations, pharmaceutical companies, etc. Overall, much valuable information was collected and many options were explored.

Gaps Analysis

Following the collection of data, the consultants conducted a gaps analysis in order to identify areas of gaps in telehealth knowledge, information, and data. It was found that many of the initiatives currently underway in Atlantic Canada are missing information on several of the key elements outlined in the data collection tool. For example, more than half of the stakeholders who participated in the questionnaire / interview indicated that the projects / programs in which they were involved are missing key elements such as site preparation documents, needs assessment tools, requests for proposals, and service provider agreements. It was also noted during the workshop that most of the requirements associated with organizational structure, communications / dissemination, and the data repository were also gaps, because none of the structures outlined currently exists in a formal context.

Effectiveness in Achieving Objectives

A number of supporting factors and challenges to conducting the research were identified during data collection for the evaluation.

Supporting Factors

Supporting factors included: quality of data collection tools and process, and grassroots approach.

Quality of Data Collection Tools and Process

It was determined from interviews with the consultants that the quality of the data collection tools was key to the success of the project. While the consultants noted that the extra time required to develop the data collection tools posed a challenge, they also viewed the quality of the tools as worth the investment. The consultants commented not only on the quality of the data collection tools but also on the benefits of the data collection process. They felt that the 'rapid' or 'real time' information gathering approach was very effective.

Grassroots Approach

The closeness of ATKE to the grassroots was seen as important in bringing information forward. The connection with grassroots was a theme articulated by Steering Committee members and by stakeholders during the evaluation process.

"Change begins at the grassroots level and moves from the informal to the formal."

(Steering Committee focus group / interview)

"Gathering of information has to come from the users -- it has to make a difference to the users. If not, it doesn't get used or it gets used poorly. If it makes a difference in helping people do what they do better in relation to the patient or the family or the community they are serving, it is positive, otherwise it's just keeping people busy."

(stakeholder interview)

Challenges

Challenges included: time consuming data collection process; difficulties completing the survey; participant selection process; intellectual property; state of information; competing demands; and project scope.

Data Collection Process was Time Consuming

One of the challenges noted by the consultants was the amount of time required to develop the data collection tools and to collect the data. A collaborative approach was used to develop the data collection tools; this proved to be very beneficial but also very time consuming, as it took time for Steering Committee members to review and provide input on the tools. In addition, issues with software and file conversion slowed down the data collection tool development process. With regard to the data collection, the project team had to work around the holiday season and the busy stakeholders' schedules, which also proved time consuming.

Difficulties Completing the Survey

In response to a question regarding the challenges stakeholders experienced in completing the survey, one stakeholder indicated that the questions were not that challenging, as respondents only had to indicate what they had to offer, rather than submit their data. However, there was a sense that the process may become more challenging in the next phase. Other respondents identified some of their challenges.

"As an educated guess I would say that some of our regional health authorities would have a tough time because some of them don't realize what they are doing and what they are not doing."

(stakeholder interview)

"Survey also asked questions regarding sources and where we would turn to. I was feeling inadequate as we have been tapping into so little -- difficult to know where to go."

(stakeholder interview)

Participant Selection Process

Although a good sample of stakeholders participated in the project, the consultants expressed concerns about how those stakeholders were chosen. The participating stakeholders were 'handpicked', which means that the sample could have been biased. In order to gain an accurate understanding of the telehealth needs of stakeholders and of the situation across Atlantic Canada, those stakeholders who were not invited to participate in the project need to be afforded an opportunity to provide their thoughts and input.

Intellectual Property

Some Steering Committee members and stakeholders raised questions regarding intellectual property. A few stakeholders expressed concern about submitting information for the repository and they questioned how the information was going to be presented, referenced, and used. Steering Committee members felt that this was an issue that needed to be looked into and explored.

"There is a guardedness as to how information is to be used. Will it be used competitively to get funding? How will sensitive information be handled -- critique of facilities, waiting lists. We'll need to manage expectations."

(stakeholder interview)

State of Information

Although many stakeholders who participated in the research indicated that they have information that could be included in a data repository, most noted that at the present time, the information is not available in a format that could easily be shared

"Expertise is there but resources are not being ear marked. There are not the bodies to do the work. We don't spend time promoting, publishing -- we're getting better, but we need to sell our successes. From a telemedicine perspective, there is lots we need to do but it is a resourcing issue."

(stakeholder interview)

"Also our results are not published. We have been too busy...need to look at data to see if it could be published, to see if it is accurate."

(stakeholder interview)

Competing Demands

Steering Committee members noted that one barrier which may have prevented some stakeholders from participating in the data collection process was the great demand on their time, as they are being approached by a number of telehealth research projects in North America and asked to provide input into research and complete surveys. Steering Committee members and consultants considered the personal contact with stakeholders important in addressing this barrier.

"I remember a major barrier that we all felt was 'how many more surveys could we send out to telehealth people and ask them to fill in?' There are such a small number of us in Atlantic Canada, and we are being bombarded by every research project on telehealth in North America for information or to fill in surveys. We were worried about the kind of reception we'd get. The fact that we [achieved] a 78% response rate is quite amazing, considering the number of requests these people have to fill in surveys or share information. It was a barrier that we anticipated, and had to think through and manage. If we hadn't managed it the way we did, we wouldn't have [received] the response rate that we had. The way we managed it was we gave it a personal touch -- each of us personally contacted stakeholders before the consultants contacted them. We were opening the door for the interview or survey to take place."

(Steering Committee focus group / interview)

Project Scope

One of the challenges noted by the Steering Committee members was trying to manage a project that focused on two key elements; the detailed data elements and the larger strategic element. With tight project timelines and the change in project management personnel, day-to-day operational details became the focus and strategic planning continued to be a challenge for the project. Although Steering Committee members were satisfied with the work completed in the end, it was suggested that the knowledge repository part probably received more attention than did the strategic elements.

"...The project had two sides to it, one was a very detailed side -- collecting data and getting the foundation for the inventory, micro level work -- and the other side had more strategic and collaborative pieces to it, macro level work. So we really had two working projects in a way, and we had to decide where the energy was going to be placed. ...The macro part really got short changed...if we had only been focusing on one or the other, it wouldn't have been so difficult...trying to balance the energy between both pieces of work was challenging..."

(Steering Committee focus group / interview)

Outcomes

Overall, it would appear that the research process was very successful. It provided valuable information on the telehealth needs of stakeholders, the types of initiatives that currently exist in Atlantic Canada, the requirements needed to develop an operating model for ATKE, and the gaps in telehealth knowledge, information, and data. The Sierra Systems Group Inc. (2003) report also stated that the data collection process and the ATKE workshop provided sufficient information to identify options and make a recommendation for a data repository that would support the ATKE vision of sharing information.

In addition, the research provided many opportunities to promote the project, build rapport with stakeholders, strengthen collaborations, and increase buy-in. It was noted in the Sierra Systems Group Inc. (2003) report that stakeholders were interested in participating in and learning more about telehealth. It was also noted that the project has served to heighten awareness of telehealth initiatives in Atlantic Canada and has demonstrated -- through the inventory process -- that projects / programs are willing to share knowledge, information, and data pertaining to their telehealth experiences. Stakeholders who participated in the data collection process and in the evaluation interviews indicated that they thought an Atlantic Telehealth Knowledge Exchange was a good idea. They stated that having access to telehealth knowledge would allow them to develop or enhance telehealth projects / programs and would ultimately have a positive effect on care delivery.

In general, most stakeholders who participated in the project were supportive of the idea of developing a data rep

"It can be virtual -- if there is a common framework and common standards it does not have to be in one spot... If office of health information [OHIH] has a platform that this could be glued on, it would give it the power and potential."

(stakeholder interview)

"Needs to have a wider focus than telehealth -- telehealth is only a small narrow part of the entire spectrum. We in telehealth are into health informatics and we are into e-health and so we have to be prepared to acknowledge that we are just a part of the larger scene which includes individual health record, individual clients accessing resources on their own, and transmission of medical data of all kinds -- EHR, public access, voice files, images, lab data, standard accessible, transmission of medical files -- not just teleheath consult. So I would be a little concerned if we narrow it in to telehealth as we normally speak of it."

(stakeholder interview)

"In time we may have people with their health records on the web -- if that was combined with a repository as to where you could find resources in Atlantic Canada it might be useful."

(stakeholder interview)

A key message from stakeholders is that the knowledge repository must reflect the needs of, and be available to, users of telehealth.

"At end of day it has to have value to users. Come at it with: Who are the fundamental users? What is the utility that they find in this model? How do we create a capacity to root that so that it continues to be used and grown because it makes a difference to the folks who use it?"

(stakeholder interview)

3.2.2 Summary and Recommendations Re: Knowledge Repository

Summary

The assumption of ATKE that there was a need for a knowledge exchange is well founded.

"Through our work it became painfully evident that information is required to plan and make decisions. Through this I became committed to ATKE... It's not fair to rely on one single person -- information exchange is extremely important. Information is out there, but it's tedious to search for it."

(stakeholder interview)

Although the development of a data repository is a key activity of ATKE, members of the Steering Committee felt that developing the repository was not the most useful function of the Committee. However, they also recognized that it is important not to pass off all responsibility for the data repository and that they should have more of an overseer role in that regard, in order to ensure that there is value in the process for those at the grassroots level who are contributing the information.

Overall, the Steering Committee felt some satisfaction with the accomplishments of the data gathering process, but also some frustration with not having gone further in the development of the data repository. However, this disappointment may have been due more to the growing expectations than to a measure of progress against goals and objectives set for the project, as the project did indeed gather baseline data from a wide range of stakeholders. The accomplishment acknowledges that other projects in Atlantic Canada are yet to be consulted, and recognizes the work as a good beginning.

Based on the research findings, the recommendation of the Sierra Systems Group Inc. (2003) report was "to develop a strategy for a web-based data repository." Developing such a repository for baseline data is a viable option for ATKE. The report also suggested that the inventory compiled as part of this project is an excellent starting point, and the focus group which discussed the repository during the ATKE workshop identified the requirements for such a repository. In addition, the Sierra Systems Group Inc. (2003) report made a number of suggestions regarding the development of such a web site.

Information collected from stakeholders during the data collection process also supports the development of a web-based solution; structures that could support this initiative currently exist (e.g., OHIH database). Some tactical next steps to move the process forward could include: exploring options for establishing a data repository working group; exploring the existing structures that could support this initiative (e.g., provincial web sites); and exploring options for integration with Health Canada's web site, the Canadian eHealth Initiatives Database. The need to link to such a structure in order to support the knowledge repository was seen to be very important.

During the evaluation, the consultants emphasized the need to determine the extent to which the repository or database will be collaborative / interactive versus read-only. They stressed that it is important to have a grassroots link to how it is being developed and to determine what information is needed. The need for the repository should be clarified, and its content and target audience defined.

Of the seven stakeholders who were interviewed, five indicated that their priority next step would be the creation of the data repository, and most attached additional conditions such as the need to decide, by first reviewing the evaluation report, what the repository should include. An additional stakeholder confirmed this approach and added that sessions were needed in order to discuss two issues: knowledge repository and collaborations. "What are the opportunities if we want it to go to the next level? What do we need to do?"

Further direction included: If there is agreement to proceed, engage the highest level of government support; start simply with the goal of having a resource that is dynamic and accessible to users.

Lessons Learned

A key lesson to be learned from activities related to the objective of developing a knowledge repository is that of the value of consulting stakeholders, both in terms of their needs and the availability of information. Too often, strategies are developed independently of this input and become useless to the people who need them the most.

Additional lessons learned are related to being open to factoring in the complexities of making adjustments along the way. In ATKE, these complexities include the state of information for sharing and concerns regarding intellectual property issues. Managing a project such as ATKE involved both macro- and micro-level detail, and in this regard lessons from ATKE would indicate a need for a broad enough skill set to be assigned to the project.

Recommendations

It is recommended that the project partners:

• Engage OHIH and others to explore the long-term potential of a web presence that meets the needs of 'grassroots' telehealth practitioners (as recommended in Sierra Systems Group Inc., 2003).

3.3 Information Sharing and Dissemination

Having a comprehensive dissemination plan to communicate research findings and project outcomes to stakeholders, partners, and others with an interest in telehealth was determined to be vital to increasing awareness of the project, fostering collaborations, strengthening partnerships, and helping to move the project forward. Objectives related to dissemination are the following:

• Objective 3: To generate and disseminate regionally the findings and knowledge gaps identified from the baseline data.
• Objective 5: To disseminate telehealth information and experiences in the Atlantic region to relevant initiatives across Canada.

In order to meet these objectives, information sharing occurred throughout the project. As well, a dissemination plan was developed for sharing research and evaluation findings.

3.3.1 Information Sharing and Dissemination of Findings

One of the key goals of the ATKE project was to communicate the research and evaluation findings to stakeholders involved in telehealth across Atlantic Canada. In order to communicate the findings, a dissemination plan was developed with the aim of informing a variety of groups / organizations, facilitating knowledge exchange, enhancing telehealth activity, and identifying potential options for further research and collaboration.

Throughout the course of the project, various communication strategies were used to share information on the project's process and to profile ATKE. As outlined in Section 3.1.2, 'Increasing Awareness and Engaging Partners / Stakeholders', some of the key strategies included: having Steering Committee members personally contact stakeholders to introduce them to the project; sending them project information packages via e-mail; hosting a workshop; and having Steering Committee members discuss the project informally with work colleagues at professional gatherings, meetings, and via e-mail.

With regard to dissemination of research findings, a summary report was developed by project consultants and Steering Committee members. It outlined the background of the project; project scope and research methodology; data analysis approach; research findings; conclusions; value proposition; data repository requirements / recommendations; next steps; and the dissemination plan. In order to communicate the research findings in the report, various dissemination methods will be used (Sierra Systems Group Inc., 2003):

• sharing of final report with agencies such as: Health Canada; Canadian Health Infoway Corporation; targeted Atlantic Canada telehealth stakeholders; HIA; administrators / health care professionals / senior managers involved in telehealth; Atlantic and National Research Councils, other Atlantic academic / research institutions; and telehealth champions within Atlantic Canada's universities;
• presentations to HIA, to policy makers within the provincial governments who have a vested interest in information or communications technologies, and to NB Provincial Healthcare Information Technologies Forum (HCIT);
• presentations at appropriate conferences and association meetings such as the Canadian Society of Telehealth Annual Conference General Meeting, the Canadian Organization for Advancement of Computers in Health (COACH) Annual Conference and General Meeting, and provincial professional associations currently engaged or interested in telehealth activity;
• posting of findings on appropriate provincial and health organizations' Internet web sites;
• sharing the highlights of ATKE and the potential benefits -- to the public -- of an enhanced collaborative structure for telehealth in Atlantic Canada; and
• publishing a peer-reviewed article in journals such as Telemedicine Today; Telemedicine Journal and E-Health (Official Journal of the American Telemedicine Association); or Canadian Health Care Technology.

As a first step in disseminating findings, a videoconference presentation summarizing the project progress and findings was given on March 26, 2003 for stakeholders, partners, and Steering Committee members. In addition, Steering Committee members have developed personal dissemination plans outlining how they will help communicate the research findings throughout their province.

"...As part of this project, we [Steering Committee members] sent [the project coordinator] a dissemination plan outlining what we thought we could do as individuals to disseminate ATKE findings and the report...I'll be going back to the IT forum in my province, and through that, get in contact with our Ministry of Health. At the local level, I'll be in touch with the regional telehealth coordinators and key stakeholders..."

(Steering Committee focus group / interview)

Other dissemination tools will include e-mail; media releases; conference / workshop presentations; postings on appropriate web sites; a published peer-reviewed article; formal and informal personal contacts / communications; and mail.

The evaluation report will also be used as a communication tool and will be disseminated to key stakeholders such as regional telehealth initiatives; regional health care facilities; provincial government departments of health; Health Infostructure Atlantic (HIA); Office of Health and the Information Highway (OHIH), Health Canada; Canadian Health Infoway Inc. (CHII); Canadian Society of Telehealth (CST); national health professional associations; training institutes involved in telehealth; private sector groups with an interest in Telehealth; and information technology professionals involved in telehealth.

Effectiveness in Meeting Objectives

A number of supporting factors and challenges to information sharing and dissemination of findings were identified during data collection for the evaluation.

Supporting Factors

Supporting factors included a receptive environment and stakeholder interest.

Receptive Environment

Given the current focus on telehealth and the amount of federal initiatives in place, the consultants, stakeholders, and Steering Committee members felt that the timing for the ATKE project was right. As a result, it was felt that findings from the ATKE project should be well received and seen as timely.

"...I think the timing was just perfect for an ATKE project in Atlantic Canada -- given the high profile that telehealth is now getting through the Romanow commission, Canada Health Infoway, and health reform..."

(Steering Committee focus group / interview)

"...I agree...I think the timing was right -- it was significant..."

(Steering Committee focus group / interview)

Stakeholder Interest

Stakeholders indicated that they were very interested in receiving feedback on the project and learning about future directions for ATKE. A few also indicated that there was interest in building / furthering collaborations with ATKE.

Challenges

Challenges included lack of reach to all telehealth stakeholders; loss of intellectual property; and selecting an appropriate format for a wide range of stakeholders.

Lack of Reach to all Telehealth Stakeholders

Although a fair number of stakeholders were contacted throughout the time span of the project, Steering Committee members and consultants noted that there are still many who have not been contacted and introduced to the project. There is a need to establish a plan, in order to ensure that these stakeholders are included in the dissemination efforts.

Loss of Intellectual Property

The consultants observed that one of the challenges to dissemination was concern about giving away too much information, which would diminish the potential for future research projects.

Selecting an Appropriate Format for a Wide Range of Stakeholders

Different audiences have different needs for information and ways in which they prefer to receive it. It will be a challenge for the Steering Committee to develop a variety of ways, appropriate to each audience, in which to communicate findings.

Outcomes

Because stakeholders were being consulted and engaged in the data collection process throughout the project, much of the information sharing effort took place as part of the communication and consultation efforts around these events. Based on information sharing activities carried out throughout the project, it would appear that the project has been successful in promoting the work of ATKE and increasing awareness of the project.

As the dissemination of the findings is a future effort, it is difficult to comment on all the success indicators that were developed around dissemination, other than to mention that the Steering Committee was successful in developing a dissemination strategy / plan to keep partners / stakeholders apprised of the project process and outcomes. Although not much dissemination has occurred up to this point, it would appear that stakeholders, based on the many suggestions they put forward regarding dissemination, see value in the work of ATKE and want to see the findings made available in a variety of ways to a broad group of stakeholders / partners. HIA also noted the need to disseminate results broadly.

3.3.2 Summary and Recommendations Re: Information Sharing and Dissemination

To date, dissemination of the findings has occurred in two formats: 1) a presentation during the stakeholder workshop, and 2) a videoconference presentation of the Sierra Systems Group Inc. (2003) report. The importance of broad dissemination to stakeholders was emphasized by all groups interviewed. A dissemination plan was developed and is included in the Sierra Systems Group Inc. (2003) report.

The Steering Committee is cognizant of the importance of ongoing communication, dissemination, and the sharing of project findings. As a result, the Committee has developed a preliminary dissemination strategy. As the lessons learned regarding information sharing have been encapsulated in an earlier section of this report and the dissemination of the findings of this project is in its early stages, a discussion of lessons learned will not be presented in this section.

Recommendations

It is recommended that the project partners:

• Implement the dissemination plan in the immediate future, in order to maintain momentum for Atlantic telehealth collaboration;
• Gather feedback from a broad range of stakeholders on the research and evaluation findings, in order to help determine the level of support for future directions of Atlantic telehealth collaboration.

3.4 Sustainability

Because the work being carried out through the ATKE project is long-term in nature, a key part of the project focused on sustainability and planning for the next steps.

• Objective 4: To identify strategies and seek ongoing regional commitment for further analysis, research, and / or policy development in the Atlantic region.

The activities carried out in order to move towards identifying strategies for sustaining the work and seeking ongoing commitment included developing a value proposition; holding discussions on the requirements for ATKE, on potential next steps and funding options; linking with HIA; and identifying future research possibilities.

3.4.1 Working towards Sustainability

It was noted in the Sierra Systems Group Inc. (2003) report that one of the major actions taken to help sustain the project was the development of a value proposition for ATKE. The following statements summarize the value proposition for the development of an Atlantic Telehealth Knowledge Exchange:

• Supports the identified need to share telehealth information;
• Builds on existing cross-jurisdictional Atlantic collaboration initiatives;
• Aligns with Health Infostructure Atlantic's vision of health care ICT collaboration in Atlantic Canada;
• Supports new and existing telehealth programs;
• Provides a grassroots approach to acquiring knowledge;
• Serves as a collaborative model for exchanging telehealth knowledge;
• Demonstrates interest and willingness to share;
• Enhances stakeholders' knowledge and practice;
• Offers potential to collaborate on national and international initiatives; and
• Offers potential to capitalize on telehealth stakeholder experience and expertise in education.

In addition to developing the value proposition, issues of sustainability were discussed at the workshop held on February 4, 2003. The focus group sessions held as part of the workshop were geared towards gathering information on the requirements for ATKE to move forward. Focus group participants were asked to consider various requirements, to attempt to prioritize them, and to identify any key issues or obstacles in regard to four specific topic areas: organizational structure, human resources, communication and dissemination, and data repository. From these discussions, various requirements regarding each topic were identified, key issues and obstacles were acknowledged, and tactical / strategic approaches and next steps for moving the project forward were suggested. From a strategic perspective, the major conclusions reached at the workshop were: a formal business case for ATKE is required; and provincial buy-in and governance need to be determined. With respect to tactical initiatives, it was determined that activities that are not too costly or do not require too much time to organize (i.e., further data collection and ongoing collaboration) could be undertaken concurrently with strategic initiatives.

Following the focus group discussion, a general discussion among workshop participants explored potential funding sources for the continuation of the ATKE project; i.e., universities, transition funds, federal and provincial governments, foundations, pharmaceutical companies, etc. In order to proceed with the ATKE project, additional funding sources will be required; therefore, it is important to be aware of all possible funding options and opportunities.

Another strategic approach for seeking ongoing commitment and support was the development of links with HIA. The Sierra Systems Group Inc. (2003) report states that the ATKE project is supported by HIA, and that it aligns with the HIA vision of collaboration on ICTs in health care in Atlantic Canada. HIA has been highly successful in acquiring funding to support the implementation of health care technologies across Atlantic Canada, and could be an important ally for ATKE in the search for support to continue the work. The consultants indicated that the links with HIA helped to raise awareness of telehealth and open the door for future collaboration.

The analysis of project data provided an opportunity to consider where future research opportunities in relation to telehealth knowledge, information, and data may exist, as well as having helped to identify a number of recommendations for 'next steps' to move the project forward. Some of the starting points for future research possibilities included looking at lessons learned and how these lessons have been used in telehealth projects; the types of continuous quality improvement initiatives in telehealth projects; the role of needs assessments in telehealth; the impact of services from a patient perspective; the extent of peer-reviewed telehealth research; the use of site preparation documents in the implementation of telehealth; issues with access to telehealth information; preferences for accessing telehealth / electronic telehealth information in the future; and how best-practices research is defined and captured for telehealth. The recommended strategic next steps included:

• Develop a strategy for a web-based data repository;
• Continue ATKE project Steering Committee members' meetings;
• Explore working relationship options with Health Infostructure Atlantic (HIA);
• Align with regional service delivery initiatives;
• Align with provincial and federal service delivery strategies; and
• Align with emerging educational modalities for health care providers.

Effectiveness in Meeting Objectives

A number of supporting factors and challenges to sustainability were identified during data collection for the evaluation.

Supporting Factors

Supporting factors included links with HIA; commitment from the ATKE Steering Committee; and timing of the project.

Links with HIA

Project consultants and Steering Committee members indicated that developing links with HIA helped to facilitate the project. They felt that linking with HIA helped to raise awareness of telehealth and build credibility. It was also noted by Steering Committee members and stakeholders that another benefit of working with HIA was the potential that this relationship provided for leveraging funding for the continuation of ATKE work.

Commitment from ATKE Steering Committee

From the Steering Committee focus group / interview it was noted that although this phase of the ATKE project is coming to an end, there is still strong commitment on the part of Steering Committee members to work together to help move the project forward to the next phase.

Timing of the Project

The Steering Committee thought that the timing of the project was good, considering the high profile that telehealth now has nationally. With telehealth starting to be identified as a key priority at a strategic level, the ATKE project is in a good position to be noticed and taken seriously.

Challenges

Challenges included lack of funding / resources to plan for next steps and potential loss of momentum.

Lack of Funding / Resources to Plan for Next Steps

One of the challenges identified by Steering Committee members is the lack of financial and human resource supports to help develop a plan for advancing ATKE to the next phase. Planning for future work takes a lot of time and resources, and at the moment there are no supports in place to help carry out this component of the work.

"I think the challenge is being concise in our next steps, and how we are going to position ourselves to go forward. It's going to take more resources to make that happen..."

(Steering Committee focus group / interview)

"...I'm wondering who's going to do the business case for ATKE, and where the funding is going to come from to do it...I don't think we can move ahead on many of our recommendations without another source of dedicated resources and funding. We can't sit down as a group and develop a business case -- we're going to have to have project support and money, as well as someone with the expertise to do it"

(Steering Committee focus group / interview)

Loss of Momentum

There is a fear among Steering Committee members that the lack of a dedicated financial and human resource support for moving the ATKE work forward could result in a loss of momentum. Securing funding for building the business case and developing a plan to move the project forward will take time, and there are concerns that a lag in funding / support could also result in diminishing the effects of the work completed to date.

Outcomes

To date, the ATKE project has been successful in identifying several strategies for sustainability and for seeking regional commitment for further research. The various activities that have been undertaken have resulted in the development of a working relationship with HIA and a value proposition identifying the relevance and value of developing an Atlantic Telehealth Knowledge Exchange. The activities have also resulted in the identification of some next steps / strategies for exploring relationship options with HIA, aligning with regional service delivery initiatives, aligning with provincial and federal service delivery strategies as well as emerging educational modalities for health care providers, and in the identification of future research possibilities.

3.4.2 Summary and Recommendations Re: Sustainability

In summarizing efforts related to sustainability, the discussion needs to focus on a consolidation of the points raised in different sections of this report. As the reader will note, throughout the report all stakeholders commented on the effectiveness of the ATKE project in engaging stakeholders and raising the profile of telehealth in Atlantic Canada. The issue debated was not in relation to the need for continued collaboration, but rather to the future role of ATKE. According to some stakeholders, that role should be more "to promote excellence in telehealth and to promote opportunities to work together collaboratively." In order to assume this role, ATKE will need to address some of the concerns expressed during the evaluation by stakeholders and others regarding barriers to collaboration (e.g., work / time requirements of their own projects; timeliness of Atlantic collaboration (given factors such as provincial differences in program maturity); and an appropriate structure for ATKE). Also expressed were the views that a structure was needed and that caution should be exercised as to the extensiveness of that structure.

The consultants viewed a role for ATKE as one providing a link between federal initiatives and grassroots telehealth initiatives; i.e., bridging the existing gap. This is an interesting concept for ATKE and other existing organizations to contemplate and make real, in terms of relationships and reporting mechanisms.

The Steering Committee felt that ATKE needs to continue as a champion for telehealth in the Atlantic provinces but that, at the same time, the value of the project needs to be weighed against the time commitment required, in relation to the partners' core mandate to deliver telehealth. The Steering Committee indicated that funding and structural supports are needed to move the project forward. The project needs the authority and the required resources to commit to the initiative on a full-time basis. On the other hand, informal communications may be a good starting point. Telehealth needs to be on the agenda of Ministries, the information management principal within each province, and other organizations involved in telehealth. A business plan outlining the next steps (not just financial requirements) needs to be developed. In the longer term, although the project is still far from that step, a second funding proposal would need to be developed, outlining the vision and goals for future work. The dissemination of the project's achievements to date is going to be critical to the advancement of the project. The interest, buy-in, and feedback from stakeholders (based on the dissemination of materials and lessons learned from the project) will be critical in determining whether there will be support for moving forward. The Steering Committee will need to look for opportunities that will allow it to move forward in alignment with the recommendations of ATKE. Over time, the support may build for a more formal structure for telehealth collaboration in Atlantic Canada; a series of successful initiatives are needed in order to build the case for such a structure.

One stakeholder felt that a proposal for funding at the Atlantic level needs to be developed. In the meantime, it is necessary and important to maintain the informal dialogue, and existing Steering Committee members must continue to meet and work together. Three stakeholders expressed the view that what is required is organizational and government support, as well as activities to endorse this support. Others remarked that the important point is to be meaningful to users and to meet their needs, in order to build commitment, and to work through existing organizations such as HIA; however, more time is needed for these measures to be developed. One stakeholder saw more potential for collaboration on education projects, while another indicated that there is a need to nurture and support champions. Two stakeholders expressed the need to investigate possible business / private sector opportunities related to ATKE.

The consultants recognized the importance of linking with HIA, which is tied to the Ministries of Health in Atlantic Canada. The continued involvement of an HIA member on the ATKE Steering Committee was considered to be important. The consultants viewed ministerial support as necessary for moving forward, while recognizing that Atlantic collaboration should not get delayed by bureaucracy. They felt that the key next steps were to work with HIA to obtain funding for another collaborative telehealth project in Atlantic Canada, and to develop links with federal initiatives to move the data repository forward. The consultants clearly differentiated between the strategic and the technical steps required.

They also suggested that the Steering Committee needs to consider whether it wants to move forward without any additional resources in place to support the effort, and pointed out the possibility of losing momentum if future funding / resources are not identified quickly.

Lessons Learned

As most of the work on sustainability is more future-oriented, little discussion of lessons learned has been put forward in this section. However, one of the key lessons learned during this evaluation process is the importance of engaging a larger number of people to take the work forward and defining the role of ATKE in relation to other e-health bodies within the health system. The benefits of such actions will have a positive impact on moving ahead toward a collective agenda.

Recommendations

It is recommended that the project partners:

• Continue to dialogue with HIA;
• Engage a core group of individuals to provide leadership to ongoing collaboration among Atlantic telehealth initiatives;
• Build on low cost, informal mechanisms for communication and information sharing among grassroots telehealth practitioners;
• Build on the value proposition and seek funding for a business case; and
• Consider the value of submitting another proposal to further develop Atlantic telehealth collaboration.

4. Conclusion and Summary of Evaluation Recommendations

4.1 Conclusion

The vision for the ATKE project was to promote excellence in telehealth in Atlantic Canada, and to contribute to telehealth knowledge and knowledge exchange at provincial, regional, and national levels. To achieve this vision, the ATKE project defined the following project goals:

• To initiate the definition and development of a sustainable Atlantic telehealth structure that will foster excellence in telehealth at the provincial / inter-provincial level in Atlantic Canada.
• To model, through the ATKE project process itself, an initial model / framework for Atlantic collaboration in telehealth.
• To leverage 'grassroots' input by involving all telehealth professionals / organizations currently involved in telehealth in Atlantic Canada.
• To produce a research report that captures the results of the ATKE applied research project.
• To complement the work of Health Infostructure Atlantic (HIA) in leveraging Ministerial support for the use of information and communications technologies to improve access to and quality of health care services in Atlantic Canada.
• To capture and synthesize baseline information that can be leveraged to market and promote telehealth in Atlantic Canada.
• To establish a foundation for leveraging external funding opportunities in support of collaborative, pan-Atlantic telehealth initiatives.

The ATKE project was discussed in great detail in previous sections of this report, and the documentation and perceptions of various committees and stakeholders were reviewed in relation to the specific objectives set by ATKE to achieve its goals. This concluding section will discuss the impact and status of the overall project in achieving its vision and goals.

As is evident in the report, great progress has been made throughout the project in increasing the profile of telehealth in Atlantic Canada by means of it becoming the focal point and the voice of 'grassroots'; i.e., those individuals who are delivering telehealth services. The research process has captured and synthesized baseline information on telehealth, and fostered collaboration among telehealth providers who might otherwise never have come together. Efforts to collaborate with HIA have been initiated, and communication with HIA has been a critical component throughout the process. Although the project falls short of achieving the fullest intent of its goal of developing a 'sustainable structure' in this funding period, it has mobilized great interest and a commitment to continued collaboration on the subject of telehealth. It has also brought to the discussion tables of governments and key committees a critical awareness of the value of telehealth in sustaining our health system, and has motivated decision makers to take action to integrate telehealth into health telematics discussions. The goals of ATKE may have been too ambitious for the time period allotted for the project, but they were significant and realistic in driving the telehealth agenda. The assumption of ATKE that there was a need for a knowledge exchange is well founded and should provide Steering Committee members with a strong basis for moving forward.

"Through our work it became painfully evident that information is required to plan and make decisions. Through this I became committed to ATKE...It's not fair to rely on one single person -- information exchange, extremely important Information, is out there, but it's tedious to search for it."

(stakeholder interview)

The importance of telehealth to sustaining the health system in Atlantic Canada was expressed by a number of stakeholders during the evaluation process.

"Down the road without tele-health in mind I don't think we can sustain health care in small rural provinces..."

(stakeholder interview)

"I support the construct and concept of telehealth to have capacity to provide care 'close to home' better for patients and families as the least intrusive service, and for recruitment and retention reasons...through access to telehealth they [professionals] can have access to colleagues, consultative advice and feel less isolated and more confident in staying in remote areas because they have support and important backup collegial support."

(stakeholder interview)

Stakeholders and HIA repeatedly expressed the view that telehealth needs to become more integrated into service delivery. This can be achieved by including telehealth among other services in strategic planning and policy; continued effort will be required to raise awareness and resource allocation.

"People know it exists but still treat it as if it is a bit exotic."

(stakeholder interview)

The view was also expressed that over time, telehealth will be looked upon differently. An example provided by HIA cited a national project, in which EHR (electronic health record) was combined with telehealth because the two could not be kept separately. This idea was supported by a stakeholder who described the health record in Singapore, in which all e-health elements are integrated, as the way of the future.

If telehealth is to have more legitimacy, a greater number of people need to become involved. This provides an enormous opportunity for ATKE. Although the provinces must organize their own systems before committing to funding for an organization such as ATKE, the consultants observed the need for more local collaboration within provinces while they are working on Atlantic-level collaboration. One member of the Steering Committee put forward the view that informal collaboration among stakeholders can accomplish a great deal, as change traditionally takes place at the grassroots level and moves from the informal system to the formal.

4.2 Summary of Evaluation Recommendations

Recommendations from each of the subsections of the Evaluation Findings are included here for ease of reading. In summary, it is recommended that the project partners:

• Keep the focus on what gives people passion about telehealth; i.e., on improving access to quality care and decreasing isolation of caregivers;
• Work with grassroots to identify telehealth priorities for clinical applications, education, research, and policy issues;
• Work to integrate telehealth into the Atlantic Canada e-health agenda, in conjunction with HIA and other stakeholders;
• Engage with OHIH and others to explore long-term potential of a web presence that meets the needs of 'grassroots' telehealth practitioners (as recommended in Sierra Systems Group Inc., 2003);
• Implement the dissemination plan in the immediate future, in order to maintain momentum for Atlantic telehealth collaboration;
• Gather feedback, from a broad range of stakeholders, on the research and evaluation findings, so as to help determine the level of support for future directions of Atlantic telehealth collaboration;
• Continue to dialogue with HIA;
• Engage a core group of individuals to provide leadership to ongoing collaboration among Atlantic telehealth initiatives;
• Build on low cost, informal mechanisms for communication and information sharing among grassroots telehealth practitioners;
• Build on the value proposition and seek funding for a business case; and
• Consider the value of submitting another proposal to further develop Atlantic telehealth collaboration.

References

National Telehealth Interoperability Workshop Report. Canadian Society of Telehealth (2001).

Picot, J. Telehealth Development in Canada: An Overview. Infotelmed Communications Inc. (2001).

Sierra Systems Group Inc. (2002). Consulting Services in Support of the Atlantic Telehealth Knowledge Exchange - Project Charter. IWK Health Centre, Halifax, Nova Scotia.

Sierra Systems Group Inc. (2003). Atlantic Telehealth Knowledge Exchange. A report to the IWK Health Centre. IWK Health Centre, Halifax, Nova Scotia.

Office of Health and the Information Highway (OHIH) (2001). Request for Proposals - Knowledge Development and Exchange (KDE) Applied Research Initiative. on May 6, 2003.