First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

(PDF Version - 249 K)

Management Action Plan - First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

Final Report

February 28, 2012

Table of Contents

• Executive Summary
• 1. Background
• 2. Program Description
  • 2.1. Program Components
  • 2.2. Program Resources
  • 2.3. Key Stakeholders and Partners
• 3. The FN&I eHealth Infostructure Program Evaluation
  • 3.1. Evaluation Objective and Context
  • 3.2. Evaluation Scope
  • 3.3. Departmental Assessment of Evaluation Risk
  • 3.4. Evaluation Approach, Design and Methodology
  • 3.5. Limitations and Mitigation Strategies
• 4. Key Findings
  • 4.1. Relevance
  • 4.2. Performance
• 5. Conclusions
  • 5.1. Relevance
  • 5.2. Performance
• 6. Recommendations

Acronyms

AANDC

Aboriginal Affairs and Northern Development Canada

AFN

Assembly of First Nations

AHTF

Aboriginal Health Transition Fund

BPMD

Business Planning and Management Directorate

CFOB

Chief Financial Officer Branch

DEP

Departmental Evaluation Plan

DPMED

Departmental Performance Measurement and Evaluation Directorate

EAC

Evaluation Advisory Committee

eHIAC

eHealth Infostructure Advisory Committee

eHIP

eHealth Infostructure Program

EHR

Electronic Health Record

EMR

Electronic Medical Record

F/P/T

Federal / Provincial / Territorial

FAA

Financial Administration Act

FEDAA

Federal Accountability Act

FN

First Nations

FNIH

First Nations and Inuit Health (Regions)

FNIHB

First Nations and Inuit Health Branch

FN/I

First Nation and Inuit

HC

Health Canada

HISAP

Health Infostructure Strategic Action Plan

HQ

Headquarters

HSIF

Health Services Integration Fund

ICTs

Information and Communication Technologies

IT

Information Technology

ITK

Inuit Tapiriit Kanatami

MAP

Management Action Plan

PAA

Program Activity Architecture

PMF

Performance Measurement Framework

PHCD

Primary Health Care Division

PHCPHD

Primary Health Care and Public Health Directorate

P/T

Provincial/Territorial

PTA

Project Technical Authority

RAPB

Regions and Programs Branch

RPP

Report on Plans and Priorities

RMAF

Results-Based Management and Accountability Framework

Executive Summary

The objective of the First Nations and Inuit Health Branch (FNIHB) eHealth Infostructure Program (eHIP) evaluation was to assess the relevance and performance of the program for the five-year period from 2006/07 to 2010/11. The results of the evaluation will provide the Health Canada Deputy Minister and senior management with a comprehensive and reliable base of evidence to support decisions regarding the eHIP's present and future initiatives. The evaluation is required by the Financial Administration Act and the Government of Canada Policy on Evaluation (2009). This evaluation was conducted in accordance with Government of Canada (2009) and departmental (2010) policies for evaluations. Further guidance was provided through a variety of templates (including a data collection matrix) and a National Evaluation Working Group (EWG).

The eHIP supports the use of health technology to enable First Nations and Inuit (FN/I) community front line healthcare providers to improve people's health through innovative partnerships, technologies, tools and services. It focuses on the adoption of modern information technology (IT) for the purpose of defining, collecting, communicating, managing, disseminating and using data to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities.

The eHIP is comprised of one national office and seven regional offices. In total, the eHIP's funding allocation (including full-time employees, operations and maintenance, and grants and contributions) for the five fiscal years included in the evaluation was $131,747,810.00. This amount, provided by CFOB, includes both A-Base funding as well as funding provided by other program areas outside the eHIP National office.

The eHIP is responsible for the following six program components which served to guide this evaluation:

1. Broadband Connectivity
2. Public Health Surveillance Systems
3. Telehealth
4. Electronic Medical Record (EMR) and Electronic Health Record (EHR)
5. IT Technical Support, Maintenance and Capacity Development
6. Information Management.

This evaluation sought to assess the relevance and performance of the eHIP against the following five Evaluation Core Issues:

Relevance:

• 1. Continued Need for the Program
• 2. Alignment with Government Priorities
• 3. Alignment with Federal Roles and Responsibilities

Performance:

• 4. Achievement of Expected Immediate, Intermediate and Long-Term Outcomes
• 5. Demonstration of Efficiency and Economy.

Key Findings and Conclusions

Overall, this evaluation has found that the eHIP is a highly relevant program that has demonstrated progress towards its stated outcomes, but it must improve its forward-looking strategic planning and implementation to ensure the achievement of the eHIP stated outcomes.

Relevance

The eHIP continues to address a significant and demonstrable need, and has been responsive to the needs of FN/I, though opportunities to become more appropriate and responsive to FN communities exist. In particular, eHIP helps meet a variety of specific FN/I health needs and eHIP's major business activities are relevant and appropriate. Recent disease outbreaks illustrate the importance of eHealth technologies (i.e., public health surveillance systems) that support and are responsive to FN/I communities.

The eHIP's objectives are well aligned and consistent with federal government priorities and departmental strategic objectives. Evidence includes "innovation and keeping pace with technology" as a federal government priority noted in the 2010 Speech from the Throne while Electronic Medical Records (EMRs) and Telehealth were identified as key national healthcare priorities by Canada Health Infoway. Various First Nations and Inuit Health Branch (FNIHB) documents outline objectives such as access to health services and communicable disease control, and ensuring the eHIP's historical and current outcomes are aligned with key stakeholders' expectations.

The eHIP is in alignment with federal roles and responsibilities, and it is appropriate for the federal government to be delivering this program. The vast majority of stakeholders believe the federal government should be involved in funding eHealth initiatives in First Nations and Inuit (FN/I) communities. This is consistent with the 2010 Assembly of First Nations Annual General Report which identifies partnerships with various Federal/Provincial/Territorial organizations and various eHealth projects as key priorities.

Opportunities have been identified to ensure the Program continues to meet the needs identified by FN/I communities.

Performance

There has been progress toward the achievement of most of the immediate outcomes, though none have yet been fully achieved. Many communities still lack access to many eHealth services. The most measurable progress has been made in the areas of improved access to eHealth infostructure services such as broadband connectivity and Telehealth. Significant improvement is required in the areas of a public health surveillance system and EMRs/EHRs. Although some training is being provided to increase the use of IT as part of service delivery, improved effort is required. Some communication strategies were identified to increase FN/I awareness of eHealth infostructure but there has been marginal increase in awareness of eHealth in FN/I communities over the past five years. There is evidence of the increased use of evidence-based information to inform eHealth planning and implementation. However, data was insufficient to analyze two of the eHIP's immediate outcomes: improved ongoing integrated planning and implementation of complex eHealth systems; and, greater use of policies, standards and guidelines for IT implementation and use.

Overall, stakeholders expressed the need for greater collaboration on how limited resources are allocated, based on strategic priorities and community needs. The lack of predictable funding impacts the sustainability of needed eHealth systems, services and tools. Greater collaboration is needed with all stakeholders to ensure funding priorities align with FN/I community needs and contribute to overall eHealth objectives, including interoperability with provincial/territorial system requirements. Assessment of expenditure allocations demonstrates a need for improved financial performance data and tracking. Theoretical modeling demonstrates the importance of system and tool up-take (utilization) to ensure cost effectiveness of eHealth strategies ensuring the achievement of overall value-for-money.

There has been some progress towards the achievement of intermediate outcomes. This is illustrated by the establishment of valuable partnerships to ensure stakeholders in FN/I health are engaged in the integration of eHealth services. The eHIP has demonstrated measurable progress in ensuring access to health information in the areas of broadband connectivity and Telehealth with less progress in the areas of a public health surveillance system and EMRs/EHRs. Data was not sufficient to analyze two of the eHIP's intermediate outcomes: increased First Nations and Inuit management of eHealth Infostructure; and, increasingly integrated information for continuous improvement in eHealth Infostructure.

There has been varied progress toward the achievement of most of the long-term outcomes however, it should be noted that these results are not to be fully expected until 2020. Progress has been made in such areas as FN/I capacity, capability and seamless integration with provincial EHR systems. There is no performance data on EMR implementation, little data on system integration performance, and low levels of satisfaction reported by survey respondents with respect to EMR/EHR availability. Significant levels of dissatisfaction were reported from community-level stakeholders with the governance of the program demonstrating a need to improve FN capacity to influence and/or control eHealth programs and services.

The results of this evaluation also indicate that community-level service providers do not believe many of the proposed key benefits of the eHIP have yet occurred; however, there have been demonstrated benefits such as increased access to educational opportunities and skills development, and making service delivery more efficient and effective. There are many opportunities to ensure long-term success.

Recommendations

Based on the findings of this evaluation, a number of recommendations have been provided to assist the eHIP in continuing to be relevant and improving on performance through the full achievement of expected results. The recommendations resulting from this evaluation are to:

Create a single comprehensive and detailed Business Case.

The eHIP should consider preparing a comprehensive and detailed business case, with a focus on increasing adoption of modern systems, change management strategies, and utilization within communities to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities. Findings suggest more detailed information regarding the eHIP's highest-level organizational goals (and how it will achieve them) is required. The development of a business case would provide this level of detail and assist the eHIP with its "improved ongoing integrated planning of complex eHealth systems" immediate outcome, as well as its "use evidence-based information to inform eHealth planning" immediate outcome. Consideration of best practices and building on past successes must also be incorporated.

Prioritize investments and further clarify funding policies and practices.

The eHIP needs to ensure funding is adequate for both implementation projects and ongoing operations through a review and identification of funding priorities within the eHIP component areas. Since eHealth priorities and progress vary significantly between communities, the eHIP should be flexible in working with communities to understand their unique needs and develop solutions. Investments should be based on the program components, regional needs and identified funding priorities.

Build eHealth capacity of FN communities.

Ongoing support to external stakeholders will identify areas for investing and sustain First Nations and Health Canada governance, planning and strategy development as well as First Nations human and organizational capacity development. Readiness assessments will facilitate each FN/I community to assess gaps in eHealth knowledge, identify training needs of service providers in their community, and develop a strategy that is tailored to meet their specific needs.

Implement a comprehensive Communications Approach.

The development and implementation of a Program-wide, multi-pronged, communications approach will increase awareness of eHIP's activities across all Regions and communities (one of the eHIP's immediate outcomes). It will also facilitate dialogue between stakeholders at various levels of government, as well as external stakeholders, and will provide greater opportunity for stakeholders at the community level to be involved in eHealth program planning and decision-making.

Implement a refined Performance Measurement Strategy.

The refined strategy will ensure: the collection of better quality performance data (including financial data); improved consistency of data collected across jurisdictions; better measures of success against established targets; support for informed decision making; and facilitate future evaluations.

1. Background

The objective of the FN&I eHealth Infostructure Program (eHIP)^{Footnote 1} evaluation was to assess the relevance and performance of the program for the period of 2006/07- 2010/11. The evaluation will provide the Health Canada Deputy Minister and senior management with a comprehensive and reliable base of evidence to support decisions regarding the continued implementation of the program's present and future initiatives. The evaluation will also identify any gaps, barriers to success, areas of concern, and success stories related to eHealth at the community, regional and national levels.

This evaluation is required by the Financial Administration Act and the Government of Canada (GoC) Policy on Evaluation (2009). As per Health Canada's 5-Year Departmental Evaluation Plan (DEP), the evaluation of the eHIP is required to be completed in 2011/12.

2. Program Description

The eHIP supports the use of eHealth technology to enable First Nations and Inuit community front line healthcare providers to improve people's health through innovative eHealth partnerships, technologies, tools and services. It focuses on the development and adoption of modern IT systems for the purpose of defining, collecting, communicating, managing, disseminating and using data to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities.

The program evolved out of the need to align with First Nations' ehealth strategies, health plans and policy directions, as well as the movement by provinces and territories and the health industry towards increased use of information and communications technologies (ICTs) to support health service delivery and public health surveillance. A brief history of the evolution of the eHIP from 1999 to present is outlined below:

1999 - 2002: First Nations and Inuit Health Information System (FNIHIS)

The original FNIHIS application traces back to the 1980s, when the FNIH Ontario Region developed a health information computer system at a time when there was no existing electronic means for collecting health data of First Nations living on-reserve. The system was jointly owned by Health Canada's FNIHB and the Chiefs of Ontario. By 2002, FNIHIS was running in health facilities in 65% of the communities served by FNIHB as a case management and planning tool for public health nurses and other health professionals.

2002 - 2010: e-Health Solutions Unit (e-HSU)

The FNIHIS Project Team became the core of a newly created organizational unit in FNIHB called the e-Health Solutions Unit. The e-HSU was created in 2002 and evolved out of the need for FNIHB to align with First Nations' eHealth strategies, health plans and policy directions, as well as the movement by provinces/territories and the health industry towards increased use of IT to support health service delivery and public health surveillance.

2010 - Present: eHealth Infostructure Program (eHIP)

Following the Health Infostructure Strategic Action Plan (2010) and under the new Program Authorities, the e-Health Solutions Unit transitioned into the eHealth Infostructure Program. The emphasis of the new program is on: Service Provision Capacity Building; Stakeholder Engagement and Collaboration; Data Collection, Surveillance, and Research; and Policy Development and Knowledge Sharing. This will provide support for Electronic Health Record (EHR) capacity and capability for First Nations in the future.

Health Canada is committed to achieving a fully integrated, sustainable health service for First Nations and Inuit community members that gradually adds more community-level eHealth services, and enables front-line health care providers to use innovative technologies and services to improve health care. Guided by FNIHB's Health Infostructure Strategic Action Plan (HISAP), work towards this vision is continuing in close partnership with other federal departments such as Aboriginal Affairs and Northern Development Canada (AANDC) (formerly Indian and Northern Affairs Canada), Canada Health Infoway, provincial governments, private sector organizations and First Nations and Inuit leadership. FNIHB and FNIH Regions support the delivery of public health and health promotion services to First Nations on-reserve and in Inuit communities.

The eHIP logic model (see Figure 1) is derived from the HISAP, outlining the main activities, outputs and expected outcomes of the eHIP. The eHIP also plays an important role in FNIHB's public health plans, evidenced by the Five-Year Strategic Framework for FNIHB's Public Health Role in First Nations Communities. The eHIP's portfolio of program components, described in Section 2.1, are derived from these various strategies and plans.

Figure 1: FN&I eHealth Infostructure Sub-Sub-Activity Logic Model

2.1 Program Components

The eHealth Program is responsible for the following six components:

1. Broadband Connectivity
   Sustainable Broadband Connectivity is the key basic element for modernizing community level health service delivery in First Nations communities. Working in partnership with First Nations and Inuit, AANDC leads a Government of Canada approach to community broadband connectivity with Infrastructure Canada, other federal departments and provincial governments. FNIHB works on achieving "last mile" solutions to First Nations/Inuit health facilities and supporting the sustainability of broadband for health business. FNIHB works with AANDC on capacity development strategies to maintain the connectivity infrastructure for the benefit of all public services at the community level.
2. Public Health Surveillance and Related Systems (i.e., Panorama, or equivalent)
   Public Health Surveillance is understood to be the on-going, systematic use of routinely collected health data to guide public health action in a timely fashion. The goal of FNIHB and FNIH Regions is to work with the provinces towards a system that ensures adequate public health surveillance for First Nations and Inuit peoples across the country that is comparable to public health surveillance in place for other Canadians.
3. Telehealth
   Telehealth is the use of information and communications technologies (ICTs) to deliver health services and transmit health information over both long and short distances. FNIHB and FNIH Regions work with First Nations and Inuit regionally to access key services within established provincial and territorial Telehealth Networks, forming partnerships for integrated service delivery and cost support. Telehealth services aim to be responsive, flexible, accessible, and cost effective for communities. This includes providing supports and tools for professionals and community members to establish long-term sustainable funding for telehealth infrastructures and service delivery programs.
4. Electronic Medical Record and Electronic Health Record (EMR/EHR)
   FNIHB works with FN/I to address privacy concerns and seeks to meet the principles of ownership, control, access and possession (OCAP) where applicable, while ensuring that all provincial and federal legislative requirements are satisfied. At the community level, the use of the EMR/EHR will be based on the community's health needs assessment, and local health management priorities.
5. Information Technology (IT) Technical Support, Maintenance and Capacity Development
   FNIHB works with First Nations/Inuit to harmonize community IT policy, support strategies, and leverage approaches and resources for efficient community level IT public services support. Capacity development and support around ICT management at the community level is a longer term investment in both time and resources and this will be coordinated with other key federal departments, especially AANDC, who shares similar community ICT support challenges.
6. Information Management (IM)
   Quality health information is essential for: planning and implementing health policy and programs;informed decision making; providing information on health events and their causes and impacts; integration of information that can be passed on to individuals and communities to help them protect, improve and maintain their health;identifying research hypotheses and assistance in research studies; and detection of unusual events, epidemics, or other health changes. IM tools, procedures and legislation (federal/provincial/territorial) apply to all health information, regardless of whether it is in paper or electronic form.

2.2 Program Resources

The eHIP is comprised of one national office (FNIHB-HQ) and seven regional offices, all with varying resources. Table 1 outlines eHIP allocations for the fiscal years included in the evaluation.

Table 1 - Spending and Fund Transfers by Major Group ^{Table 1 footnote *}

	2006/07	2007/08	2008/09	2009/10	2010/11	5 Year Total
Full-Time Employees	$3,408,701	$3,174,834	$4,256,142	$4,499,676	$4,944,304	$20,283,657.00
Operations & Maintenance	$4,834,615	$2,938,646	$5,980,966	$3,831,520	$3,580,191	$21,165,938.00
Minor Capital	$708,822	$573,985	$0	$0	$103,077	$1,385,884.00
Grants & Contributions	$19,391,720	$21,672,803	$14,603,349	$15,238,578	$18,005,881	$88,912,331.00
Total	$28,343,858	$28,360,268	$24,840,457	$23,569,774	$26,633,453	$131,747,810.00
Table 1 footnotes Table 1 footnote * Note: Total expenditures include both A-Base funding as well as funding provided by other Program areas outside of the eHealth Program's National Office. Financial information provided by CFOB. Return to table 1 footnote * referrer

Sources of Funding

The table above illustrates total investments of $131M which includes regional allocations provided by other federal program sources. Some of these other sources may include the Aboriginal Health Transition Fund, AANDC, etc. Expenditures may also include one-time investments by the department.

FNIHB National Office primarily funds FNIH regional offices that, in turn, fund First Nations and Inuit communities and regional organizations using various funding models. In addition, some FNIHB funds are used to support targeted projects with a national scope designed to examine innovations for possible application to national programming.

2.3 Key Stakeholders and Partners

The following are some of the eHIP's key stakeholders and partners:

• First Nations on-reserve (may include First Nations regardless of where they live)
• First Nations and Inuit Health Regions
• First Nations and Inuit Health Branch
• Assembly of First Nations
• Regional First Nations Organizations
• Departmental Performance Measurement and Evaluation Directorate
• Primary Health Care and Public Health Directorate
• eHealth Infostructure Advisory Committee
• Community Programs Directorate
• Strategic Policy, Planning and Analysis Directorate
• Aboriginal Affairs and Northern Development Canada (formerly INAC)
• Public Health Agency of Canada
• Industry Canada
• National e-Health Advisory Committee
• Others, as required

3. The FN&I eHealth Infostructure Program Evaluation

3.1 Evaluation Objective and Context

The objective of the FNIHB eHIP evaluation was to systematically collect and analyze evidence of the eHIP's results to assess the relevance and performance of the program for the period of 2006/07- 2010/11. The evaluation will provide Health Canada's Deputy Minister and senior management with a comprehensive and reliable evidence base to support decisions regarding the continued implementation of the program's present and future initiatives. The most recent program review was done in 2006 covering the period between September 2002 and December 2005.

This evaluation was conducted between June and December 2011.

3.2 Evaluation Scope

All six eHIP components listed in Section 2.1 are in scope for this evaluation.

The eHIP has planned to undertake two separate phased evaluation activities as shown in the eHIP Evaluation Plan (Figure 2). Phase One of this process is this departmental evaluation, and Phase Two, which is not in scope for this report, will be conducted in the near future. Phase Two will examine international best practices, potential business models for eHIP, and the costs to operate the program at both the national and international levels. This evaluation will support Phase 2 as a line of evidence in order to recommend best practices and potential future business models.

Text description

The eHIP has planned to undertake two separate phased evaluation activities as shown in the eHIP Evaluation Plan (Figure 2). Phase One of this process is this departmental evaluation, and Phase Two, which is not in scope for this report, will be conducted in the near future. Phase Two will examine international best practices, potential business models for eHIP, and the costs to operate the program at both the national and international levels. This evaluation will support Phase 2 as a line of evidence in order to recommend best practices and potential future business models.

For this evaluation, the eHIP evaluation falls under 3.3 Health Infrastructure Support, 3.3.2 Health System Transformation, and 3.3.2.2 Health Infostructure of the Program Activity Architecture (PAA), and includes performance indicators relevant to the eHIP as described therein.

As specified in the eHIP Evaluation Framework, this evaluation assesses the relevance and performance of the eHIP against five core issues.

Table 2 - Evaluation Core Issues

Evaluation Core Issues	Description
Relevance
Issue #1 Continued Need for Program (Maps to Relevance Question R1)	Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
Issue #2 Alignment with Government Priorities (Maps to Relevance Question R2)	Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
Issue #3 Alignment with Federal Roles and Responsibilities (Maps to Relevance Question R3)	Assessment of the roles and responsibilities for the federal government in delivering the program
Performance (effectiveness, efficiency and economy)
Issue #4 Achievement of Expected Outcomes (Maps to Performance Questions P1, P2, P3)	Assessment of progress toward expected outcomes (including immediate, intermediate and long-term outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
Issue #5 Demonstration of Efficiency and Economy (Maps to Performance Questions P4, P5, P6, P7)	Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes

The following evaluation questions were created to gather data about the Evaluation Core Issues. The eHIP Data Collection Matrix identifies questions for each of the specific indicators for Relevance, and outcomes and indicators for Performance.

Relevance Questions R1-R3

• Core Issue #1- R1: To what extent does the eHealth Program continue to address a demonstrable need and is responsive to the needs of FN/I Canadians?
• Core Issue #2- R2: Are there linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes?
• Core Issue #3- R3: To what extent is the eHealth Program aligned with federal roles and responsibilities?

Performance Questions P1-P7

Core Issue #4:

• P1: To what extent have the immediate outcomes been achieved?
• P2: To what extent have the intermediate outcomes been achieved?
• P3: To what extent have the long-term outcomes been achieved?

Core Issue #5

• P4: How has the eHealth Program optimized the overall quantity, quality, blend of products and/or services to facilitate achievement of its expected outcomes?
• P5: Are there alternative methods which ensure the same achievement of immediate expected results?
• P6: Has the eHealth Program minimized allocated resources while maximizing outputs?
• P7: Were the eHealth Program's resources managed to facilitate the achievement of relevant immediate outcomes?

3.3 Departmental Assessment of Evaluation Risk

Health Canada assesses the evaluation risk to determine an evaluation approach and the level of effort required to complete the evaluation. The overall risk ranking level for the eHIP, as determined in the HC Departmental Evaluation Plan 2011/2012, was "Low". The low ranking was considered in the design of this evaluation.

3.4 Evaluation Approach, Design and Methodology

3.4.1 Evaluation Approach

The evaluation approach for this assessment was to examine the achievement of expected outcomes, that is, the results achieved by the program based on its logic model (see Figure 1). This summative evaluation focus was on immediate outcomes given the program's long-term outcomes are not expected to be achieved until 2020.

The evaluation also included a participatory approach, that is, the inclusion of internal and external stakeholders in the development of the evaluation framework to ensure the relevancy of the evaluation. This included an Evaluation Working Group (EWG) which was co-chaired by the eHIP Program Liaison and a Departmental Performance Measurement and Evaluation Directorate (DPMED) Senior Evaluator. Membership of the EWG consisted of the eHIP Program Liaison, DPMED, eHIP program coordinator, FNIH regional representatives, an Assembly of First Nations representative, and a consultant.

3.4.2 Evaluation Design

The Government of Canada Policy on Evaluation (2009) and the HC Evaluation Policy (2010) were reviewed to receive guidance on evaluation design and data collection best practices. An Evaluation Framework was developed to guide the evaluation including an examination of the logic model, its context and position within the department's Program Activity Architecture (PAA); an assessment of the logic model's validity in this context and the expected results chain; the use of comparison data; and baseline data when available. This was a non-experimental evaluation. However, a theory-based approach and cost utilization analysis were included in the design to support the assessment of economy and efficiency.

3.4.3 Data Collection Methods

A Data Collection Matrix was developed as part of the Evaluation Framework to guide the development of the evaluation data collection strategy. The Core Issues outlined in the GoC Policy on Evaluation (2009) include the integration of program performance and evaluation measures, methodologies and other elements that strengthened this evaluation^{Footnote 2}.

The methods used in this evaluation included the use of surveys (2) and an extensive document and literature review. Multiple lines of evidence were gathered from different sources and through the methods described below to allow for data comparison, and to support evidence-based conclusions.

3.4.3.1 Documentation and Literature Review

Program documents were obtained from FNIHB eHIP. These included annual reports, work plans, program files and other relevant material that both described and documented the eHIP's progress over time.

Internet and literature searches were conducted to identify other relevant Canadian eHealth reports, strategies, and evaluations, including non-First Nations initiatives. Documents and literature were reviewed by the consultant, and data relevant to the indicators was extracted.

A total of 184 documents from 2002 to present were systematically reviewed to identify relevance and performance data using data collection templates provided by DPMED. These standardized grids documented and mapped the relevant data to specific performance indicators and assisted in the overall analysis based on the evaluation questions.

3.4.3.2 Stakeholder Surveys

Two separate surveys were developed and administered: one for community-level service providers (herein referred to as "Community survey"), the other for Federal government management (National office and FNIH regional offices), herein referred to as "Management survey". The decision to create two separate surveys was based on input from the EWG, as it was felt that survey questions needed to be tailored to specific recipient groups. Surveys were piloted within the target audience communities.

Survey Question Development

Survey questions were developed based on indicators and designed to be an additional data source and line of evidence, as well as to capture information not expected to be found during the document review, such as stakeholder opinions. Survey questions were reviewed and revised extensively by members of the Evaluation Working Group (EWG). The surveys were adjusted to an appropriate readability level, and were translated into French.

Survey Administration

Names and contact information for community-level service provider recipients were solicited from the members of the EWG. Examples of recipients at the community-level are Health Directors and eHealth Coordinators of community health facilities. The names and contact information for 302 community-level recipients, and 112 internal management recipients were provided to the consultant. All recipients identified were sent the respective survey.

A Canadian-based online tool, Fluid Surveys, was used to administer the surveys. Recipients were given five weeks to complete the survey.

Survey Response Rates

Overall, survey response rates were lower than expected, but on par with average response rates of other Health Canada surveys within First Nations and Inuit communities. The response rate for the Management survey was 32% (N=112), with 36 recipients completing all questions in the survey. Of the 302 Community survey recipients, 40 completed all questions in the survey, resulting in a 13% response rate. However, 13 recipients notified the consultant that they had no involvement in eHealth activities and thus opted-out of the survey. The adjusted response rate is 14% (N=289).

It was determined that of those who responded, despite response rate variances from region to region, respondents reflected a fair cross-representation of both management and community participants. The implications of overall low survey response rates to data quality and general evaluation results and strategies for improving response rates in future evaluations are discussed in the Limitations section (Section 3.5).

Table 3a - Regional Distribution of Community Respondents

Response	National Percentage	Count/Total Survey Recipients
Pacific	20%	8/44
Alberta	0%	0/14
Saskatchewan	22%	9/94
Manitoba	20%	8/47
Ontario	10%	4/16
QuebecTable 1 footnote *	2%	1/43
Atlantic	25%	10/44
Total Responses		40/302
Table 3a footnotes Table 3a footnote * The First Nations of Quebec and Labrador Health and Social Services Commission compiled their responses into one survey. (Source: AFN) Return to table 3a footnote * referrer

Response rates may also have impacted the data in that, Regions with more community-level service providers currently active in eHealth activities and projects may be overrepresented in the sample. Recipients who felt their communities and Regions were not currently engaged in eHealth projects fully, may have opted-out of the survey skewing the overall general interpretations of findings.

Table 3b - Regional Distribution of Management Respondents

Response	National Percentage	Count/Total Survey Recipients
Pacific	3%	1/44
Alberta	3%	1/7
Saskatchewan	14%	5/22
Manitoba	28%	10/17
Ontario	17%	6/17
Quebec	6%	2/7
Atlantic	14%	5/19
National Capital Region	17%	6/19
Total Responses		36/112

3.4.4 Data Analysis Methods

The data collected was analyzed using the following methods:

• Systematic review of data extracted from the documents was conducted, summaries were created, and conclusions were drawn based on summaries (Data Summary templates provided by DPMED);
• Statistical analysis: Statistical analysis was conducted to analyze quantitative survey data, and appropriate charts created;
• Trend analysis: A method of time series data analysis (information collected in sequence over a period of time) that involves comparing data for the same indicator, over a period of time, to determine whether a relationship exists between the variables pertaining to that specific indicator;
• Thematic analysis: Qualitative data from open text survey questions was analyzed using a thematic analysis technique, where responses were systematically reviewed and emergent themes were identified and categorized; and
• Comparison of data gathered from document reviews and stakeholder surveys was conducted to synthesize data from disparate sources, and validate trends as part of the findings of this assessment.

3.4.5 Multiple Lines of Evidence

Gathering multiple pieces of corroborating evidence helps improve the quality of certain data. As described above, the evaluation methods relied on more than one line of evidence. The majority of evaluation questions were addressed through multiple lines of evidence, as determined through a cross-walk and data collection template.

3.4.6 Ethical/Human Subject Protection Issues and Protocol

Ethical and human subject protection principles were upheld in survey administration, data management, and reporting processes. A Canadian-hosted online tool (Fluid Surveys) was used to administer the surveys as per standard HC evaluation guidelines. Participation in the survey was voluntary.

Metadata was only provided to the contractor, and all personal or identifying information was kept confidential. Responses are presented in summary form within this Evaluation Report. The information collected was not disclosed to external third parties, as specified by the Privacy Act.

3.5 Limitations and Mitigation Strategies

The following limitations were observed during the data collection process. Potential impact on how the findings and conclusions are interpreted, and risk mitigation strategies that were used in this evaluation are discussed below.

3.5.1 Low Survey Response Rates

Several strategies were used to solicit the highest response rate possible, including:

• Survey questions were developed in close consultation with the EWG to ensure questions were appropriate for the audience;
• Community-level service provider recipients were identified by the EWG, management-level recipients were identified by the eHIP, and validated by the EWG.
• Valid email addresses were obtained for all contacts;
• Potential respondents were provided with an estimated amount of time needed to complete the survey;
• Participants were given five weeks to complete the survey. The survey was executed in September 2011 to reduce the risk of recipients being away on summer holidays;
• Reminders were sent out at mid-point, and two days before survey closure; and
• Regional eHealth Evaluation Leads were asked to remind recipients in their Regions to complete the survey.

Low survey response rates may be attributed to a statement in the survey introduction asking participants with no eHealth involvement to not complete the survey and notify the consultant to be removed from the recipient list. Participants may have self-assessed their involvement in eHealth activities and decided to opt-out because they felt they lacked eHealth knowledge and experience. In addition, recipients may not have been comfortable completing an online survey due to computer literacy, privacy, and/or accessibility issues. Furthermore, surveys were sent via email from the consulting company and recipients may not have recognized the sender, deleting or ignoring the email.

3.5.2 Data Limitations

Low survey response rates can negatively impact data quality, as a smaller sample size is less likely to represent the overall population. However, a low response rate does not guarantee lower survey accuracy; it simply indicates the risk of lower accuracy. The distribution of responses is more critical for data interpretation, as data trends are more difficult to identify with smaller sample sizes, and the risk of misinterpretation increases. As such, results of this survey must be interpreted carefully, with special care in generalizing the findings. In an analysis of the data by region, no meaningful trends were identified likely due to low response counts by Region. Drawing conclusions and comparing findings across Regions is not appropriate considering some Regions had very few or no respondents.

Gaps in Regional performance data existed in documentation provided to the contractor. These gaps included: limited or inconsistent availability of performance data over the full period of the evaluation; inconsistent availability of data between program areas; and, inconsistent reporting practices between Regions. As such, the performance data extracted from program documentation was insufficient to support nine indicators related to Evaluation Core Issue #4.

The strategies used to mitigate the risk of poor data quality in this evaluation included:

• For all questions where over 20% of respondents responded 'N/A' or 'Don't know', data was disqualified due to a reduction in small sample, and potential misinterpretation of findings;
• For questions where over 20% of respondents reported a neutral response, the impact on the findings is discussed for each instance;
• Comparison with data extracted from other sources was employed where possible to validate conclusions drawn from Community survey data;
• The contractor identified gaps in documentation early on in the project to allow sufficient time for documents to be requested from appropriate sources; and
• Survey questions were designed based on evaluation indicators and with consideration of data that may not be contained in documents provided for review.

With the use of several of these mitigation strategies, data was analyzed and interpreted using methods to increase the credibility and reliability of the findings presented in this report.

3.5.3 Assessing Economy and Efficiency

Setting the context of the efficiency and economy aspects of this evaluation in relation to the Government of Canada (GoC) Evaluation Policy (2009) is important. The five years of the eHealth Program evaluated were implemented prior to the recent GoC Policy on Evaluation. Thus, specific requirements for defining and operationalizing efficiency and economy analysis were not set out for the Program's performance measurement strategy - it did not define efficiency and economy performance measures, definitions or indicators.

As such, there is a lack of "object costing", consistent financial performance data tracking as it relates to program activities and outputs. Similar to other programs within Health Canada, financial performance data was not consistently collected across the Program's national office and regional counterparts, nor was there consistency of program reporting in terms of components or specific component-related activities. Changes in the program's overall structure, financial allocations, operational priorities, and expected results have also complicated the assessment of economy and efficiency.

Shifts in financial reporting over the past number of years, including redefining the Program Activity Architecture (PAA), the 'clustering' model within FNIHB, and how the eHealth Program results align with overall FNIH Branch expected results, all contributed to an inability to explicitly measure economy and efficiency for this evaluation.

Although these limitations present a challenge in providing an overall assessment of the economy and efficiency of the eHIP, the evaluation did attempt to provide a general sense of resource utilization by comparing resource allocation/expenditure data with program results. As well, stakeholder opinions provided an additional line of information on the appropriateness of resource utilization. This was then supported through an economic modeling exercise that examined cost effectiveness and cost utilization as elements of ensuring eHealth effectiveness by demonstrating the importance of tool and service utilization as a key factor for ensuring economy and efficiency within the program's components.

Combined, these assessments indicate where programmatic successes (in terms of financial effectiveness) have been made, where greater successes can be made in the future, and whether the Program has provided economy and efficiency in addressing the eHealth needs of FN/I communities in Canada.

4. Key Findings

4.1 Relevance

4.1.1 Core Issue #1: Continued need for program

To what extent does the eHIP continue to address a demonstrable need, and is responsive to the needs of FN/I communities?

Generally, findings from both the survey and document review indicated that the eHIP continues to address a significant demonstrable need, and has been responsive to the needs of FN/I communities. Some opportunities for improvement were observed.

"Bottom line, this program is the only opportunity for eHealth initiatives to grow within First Nation territories. I encourage Health Canada and the Treasury Board to fund these current initiatives."
Community-Level Service Provider Survey Respondent

A number of documents indicated FN/I health needs that eHIP helps meet, such as access to care through Telehealth programs, and other needs to improve health and save lives through its various other eHealth programs and infostructure services. Once a public health surveillance system/tool is integrated with its provincial system, stakeholders feel that communicable disease control, administration and tracking of immunizations will improve.

The majority of survey respondents agreed that the eHIP's major business activities - which comprise a broad spectrum of the eHIP's activities - were relevant, appropriate and responsive (see Tables 4a and 4b).

• More than 83% (N=36) of management-level and more than 55% (N=40) of community-level respondents agreed that the various "management, governance and communications" activities (such as supporting needs assessments for eHealth projects, developing and sharing eHealth best practices, etc.) performed by eHIP are relevant and appropriate.
• More than 69% (N=36) of management-level and more than 57% (N=40) of community-level respondents agreed that the various "infostructure readiness" activities (such as internet connectivity, videoconferencing services, etc.) performed by eHIP are relevant and appropriate.
• More than 80% (N=36) of management-level and more than 54% (N=40) of community-level respondents agreed that the various "eHealth projects" (such as broadband connectivity, public health surveillance, telehealth, etc.) performed by eHIP are relevant and appropriate.

Low agreement that certain activities were relevant and useful (appropriate and responsive) were found (e.g., expanding the number of tripartite data sharing and storage agreements, website hosting, lab information systems, drug information systems, diagnostic imaging, and registries). This indicates areas the eHIP should further analyze for possible refinement.

Mixed results were found regarding program-funded services and projects and their relevance, usefulness and appropriateness to addressing the needs of the First Nations and Inuit.

Table 4a - Relevant, Useful and Appropriate Agreement Rates of Community-level Service Provider Respondents (n=40)

Program Component	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree	Don't Know	N/A
Broadband Connectivity	2%	20%	10%	30%	18%	2%	18%
Public Health Surveillance Tool (i.e., Panorama, or equivalent)	5%	20%	25%	8%	12%	12%	18%
Telehealth	2%	12%	2%	38%	22%	8%	15%
EMR/EHR	5%	28%	5%	20%	15%	10%	18%
IT Technical Support, Maintenance and Capacity Development	5%	20%	10%	22%	25%	2%	15%
Information Management	8%	18%	15%	25%	18%	5%	12%

Table 4b - Relevant, Useful and Appropriate Agreement Rates of Management-level Respondents (n=36)

Program Component	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree	Don't Know	N/A
Broadband Connectivity	0%	11%	0%	28%	50%	0%	11%
Public Health Surveillance Tool (i.e., Panorama, or equivalent)	0%	6%	8%	28%	44%	3%	11%
Telehealth	0%	0%	8%	19%	58%	0%	14%
EMR/EHR	6%	6%	8%	11%	47%	8%	14%
IT Technical Support, Maintenance and Capacity Development	3%	8%	0%	25%	42%	8%	14%
Information Management	0%	8%	8%	28%	39%	3%	14%

Low agreement scores above may indicate a variety of issues, such as flawed strategy to poor data quality. For public health surveillance in Table 4a, the neutral response rate may be attributable to a lack of awareness of current activities or little-to-no implementation and/or participation at the community level. Further study of the areas is required.

Management-level stakeholder survey results indicated that there is a continued need for investment in eHealth since it helps provide more efficient and effective service delivery (63% agree, N=36), supports innovation in the delivery of health services (69% agree, N=36), offers improved access to primary care (55% agree, N=36), reduces patient travel times and costs (55% agree, N=36), and increase access to educational opportunities and skills development for providers (63% agree, N=36). In particular, Telehealth was found to increase FN/I access to specialized health services, reduce transportation costs, and provide a number of other positive benefits.

Survey results indicated that community-level service providers do not believe that many of the proposed key benefits of the eHealth Program occurred during the evaluation period. In addition, discrepancies emerged between the management and community groups related to satisfaction with the progress of the eHealth Program.

Key Findings

• At the community-level, there was generally low agreement that many of the proposed benefits of the eHealth Program occurred during the evaluation period (FY 2006/07-2010/11).
  • There was low agreement (12% agree, N=40) that eHIP helped decrease adverse events following immunization, which is likely due to the lack of available tools/systems currently in place. According to the findings, for eleven of the eighteen areas surveyed, the majority of responses were in the negative (disagree or strongly disagree with realization of benefits - such as improved availability of primary health care services, reduced patient travel time and costs, improved retention of health services professionals).
• There was strong agreement that the eHIP helped increase access to educational opportunities and skills development (58% agree, N=40), made service delivery more efficient and effective (47% agree, N=40), enhanced prevention and health promotion initiatives offered via Telehealth (48% agree, N=40), and supported innovation in the delivery of health services (42% agree, N=40).
• 37% (N= 40) of community-level respondents and 53% (N=36) of management respondents indicated they were satisfied with the general progress the eHealth Program has made in achieving its outcomes; however, 30% (N=40) of community respondents and 19% (N=36) of management respondents indicated they were unsatisfied.
• The following themes emerged as challenges related to achieving the expected results in First Nations and Inuit communities:
  • Lack of available funding for eHealth capacity, implementation and sustainability;
  • Inadequate infrastructure to support eHealth projects;
  • Fragmented First Nation healthcare governance structures; and
  • Insufficient communication between FNIHB, FNIH Regions and First Nations about eHealth projects, planning and new and innovative approaches to technology.

4.1.2 Core Issue #2: Alignment with government priorities

Assessment of the linkages between program objectives and (i) federal government priorities, and (ii) departmental strategic outcomes

Generally, findings^{Footnote 4} indicated that the program's outcomes are well aligned and consistent with federal government priorities and departmental strategic objectives.

Federal government priorities include:

• "Innovation and keeping pace with technology", which were identified in the 2010 Speech from the Throne, and the eHealth systems the eHIP is helping implement are complex and innovative modern health technology.
• EMRs and Telehealth, identified in Canada Health Infoway's 2004 document called National Healthcare Priorities as two keys to health system renewal (particularly for Canadians living in rural and remote areas), are two of the eHIP's prime areas of focus.

Departmental strategic objectives include:

• The FNIHB 2010 Health Infostructure Strategic Action Plan (HISAP) identifies the following as FNIHB strategic objectives: ensuring availability of, and access to, quality health services; support greater control of the health system by FN/I; and support the improvement of FN health programs and services through improved integration, harmonization and alignment with F/P/T health systems.
• The 2008 Interim National Directives for the FN&I eHealth Program - e-Health Solutions Units identifies key objectives, including communicable disease control.
• The FNIHB 2009 document called Five-Year Strategic Framework for FNIHB's Public Health Role in the FN/I Communities identifies a vision of "all First Nations Reserve Communities [being] served by an integrated, comprehensive public health system".

The majority of management-level survey respondents (52-78%, N=36) agreed or strongly agreed that the eHIP is well-aligned with its vertical and horizontal stakeholders and partners, including Health Canada's FNIHB and Regions and Programs Branch (RAPB), Aboriginal Affairs and Northern Development Canada (AANDC), and Canada Health Infoway.

Findings indicated the majority of management-level survey^{Footnote 5} respondents (93-95%, N=36) agreed that the eHIP's historical and current goals and priorities were also consistent with their goals and priorities including:

• an electronic health record capacity and capability for all FN people by 2020 (83%);
• seamless integration of First Nations Infostructure with provincial EHR Systems (97%);
• meaningful, standardized information for decision support available to First Nations, Regions and FNIHB (89%); and
• collaborative and sustainable partnerships between FN, Provinces and the Federal Government (97%).

4.1.3 Core Issue #3: Alignment with federal roles and responsibilities

Assessment of the roles and responsibilities for the federal government in delivering the program

Generally, findings indicated that the program is in alignment with federal roles and responsibilities, and it is appropriate for the federal government to be delivering the program.

Management-level survey respondents (97%, N=36) indicated that the federal government should be involved in funding eHealth initiatives in FN/I communities. In addition, as stated in the findings for Core Issue #2, survey respondents agreed there was strong consistency between the eHIP's components and the priorities of a number of other federal government organizations, and that the eHIP's stated current and historical goals align with the priorities of many of their federal-level stakeholders.

For example, the 1979 Federal Indian Health Policy identifies that one of the pillars of the policy is the responsibility of the Federal Government to maintain a health system that supports public health activities on reserves, health promotion, and the detection and mitigation of hazards in the health environment. And further, in March 2010, the Assembly of First Nations identified collaboration and partnership between First Nations, Canada Health Infoway and provincial governments on Telehealth and public health surveillance projects.

Some differences were noted by management-level survey respondents in the alignment of federal government roles and responsibilities (i.e., F/P/T partners' goals and priorities in relation to the eHIP outcomes).

4.2 Performance

4.2.1 Core Issue #4: To what extent have immediate outcomes (short-term goals) been achieved?

Overall, findings from the document review and surveys indicate that there has been progress towards the achievement of most of the immediate outcomes.

Immediate Outcome 1: Improve Access to eHealth Infostructure Services

As implementation of eHealth tools into FN communities relies heavily on a community's readiness (connectivity, capacity), evaluation results note that there has been varied progress in increasing and/or improving access to eHealth infostructure services across the country. Findings indicate the most significant measurable progress has been made in the implementation of broadband connectivity in health facilities in FN communities; Telehealth; videoconferencing implementations; and e-SDRT (electronic service delivery reporting template). In contrast, low or zero implementation progress data has been reported for EMR, Public Health Surveillance Tools^{Footnote 6} (i.e., Panorama, or equivalent) and Drug Information Systems.

There was a lack of performance data contained in the FNIH Regional Year-End Reports to assess the availability of eHealth infostructure services in FN communities, with the exception of Broadband Connectivity and Telehealth.

Key Findings

Broadband Connectivity:

• Across all Regions, the percentage of health facilities in First Nations communities with Broadband, Dial-up or High Speed Connectivity increased from 2006 to 2008.
• 85% of health facilities had some form of Connectivity in 2006-2007, and 96% in 2007-2008.
• Specifically, availability of Broadband Connectivity across all Regions has improved, most notably in Quebec (from 15% of health facilities with Broadband in 2007 to 100% in 2008), and Saskatchewan (from 7% in 2007 to 89% in 2008).

Telehealth:

• Across all Regions, the number of Telehealth sites grew from 165 sites in 2008 to 284 sites in 2010 (representing 45% of the 626 health facilities in FN Communities across all regions).
• The majority of community and management respondents were satisfied or very satisfied with the availability of Telehealth (56% (N=40) and 70% (N=36) respectively).

Electronic Medical Records (EMR):

• The number of EMR implementations in FN communities is not well documented, which is likely due to this being a fairly recent activity.
• Of the community-level survey respondents 45% (N=40) reported that they had assessed the feasibility of implementing an EMR, and 25% reported that an assessment had not yet been conducted. Of those who reported that an assessment was complete, 40% (N=25) indicated that an EMR had been implemented, 45% indicated planning was still in progress, and the remaining 15% indicated that an EMR had not been implemented.

Community-level service provider respondents were asked to indicate the current stage of implementation of various eHealth tools. Although not all of the tools identified below are managed by the eHealth Program, they each have an electronic health component. Reported implementation progress for various eHealth tools by community-level service providers can be seen in Table 5. The high number of "Don't Know" responses may indicate a lack of awareness of tool status or direct involvement in some areas at the community level.

Table 5 - Reported Implementation Progress for eHealth Tools (n=40)

	Planning in Progress	Implemented and Still Using	Implemented but Not Using	Not ImplementedTable 1 footnote *	Don't Know
a. Public Health Surveillance Tool (i.e., Panorama, or equivalent)	35%	0%	0%	38%	28%
b. Electronic Medical Record (ex. EMR in a public health clinic)	12%	22%	0%	48%	18%
c. Drug Information Systems	5%	0%	0%	57%	38%
d. Diagnostic Imaging	5%	18%	0%	50%	28%
e. National Native Addictions Information Management System (NNAIMS)	10%	8%	2%	35%	45%
f. Home and Community Care/Aboriginal Diabetes Initiative (HCC/ADI)	10%	45%	0%	15%	30%
g. Medical Transportation Records System (MTRS)	8%	18%	0%	40%	35%
h. First Nations and Inuit Health Information System (FNIHIS)	5%	15%	12%	38%	30%
i. Community Reporting	22%	25%	0%	20%	32%
j. Electronic Service Delivery Reporting Template (e-SDRT)	0%	68%	2%	8%	22%
k. Email	0%	79%	0%	5%	16%
l. Internet Access	0%	32%	0%	5%	15%
m. Videoconferencing
i. Administrative purposes	15%	52%	0%	15%	18%
ii. Family encounters	15%	15%	2%	30%	38%
n. Telehealth
i. Health Promotion (client education)	5%	42%	0%	32%	20%
ii. Clinical consultations	12%	38%	0%	32%	17%
iii. Education sessions/training (professional development)	8%	60%	0%	15%	18%
Table 5 footnotes Table 1 footnote * Some respondents may not be aware of "Planning in Progress". Return to table 5 footnote * referrer

Immediate Outcome 2: A workforce that is increasingly comfortable using IT as part of service delivery

Progress has been made towards this outcome. Some training is being provided to promote a workforce that is increasingly comfortable using IT. The same goal will be accomplished as more eHealth services are implemented and used by the workforce over the course of time.

Key Findings

• The majority (55%, N=40) of community level survey respondents indicated the eHealth Program has not been successful in developing eHealth skills and knowledge in FN/I communities.
• Telehealth, videoconferencing and e-SDRT training has been consistently offered in communities since 2006. Some eHealth tool training has recently been added as the Program continues to evolve.
• 33% (N=40) of community respondents indicated that the eHealth Program has been moderately or very unsuccessful in supporting First Nations and Inuit communities to develop eHealth education, skills and experience. 25% reported it has been moderately successful or successful.
• More than half (55%, N=40) of community respondents indicated that their community or organization does not have an eHealth Tool or Technology training strategy in place.
• 25% (N=40) indicated that they were satisfied and 2% (N=40) very satisfied with the eHealth tools training or skills development received by their community. (It is important to note that 32% of respondents indicated N/A for this question, thus data quality is low.)
• Gaps in performance data contained in the FNIH Regional Year-End Reports related to eHealth tool training and eHealth capacity building existed. Comparisons and aggregation of data across Regions and fiscal years could not be conducted.

Immediate Outcome 3: Increased First Nation awareness of eHealth Infostructure

Progress has been made towards this outcome. From FNIH Regional Program documentation, it appeared that a formal communications strategy is lacking. Overall, a need was identified to increase frequency and level of communication with FN communities and organizations as well as other partners including provincial jurisdictions. Respondents indicated a need to develop an effective communications approach that keeps information flowing to and from the eHIP to First Nations communities and FNIH Regions.

Key Findings

• Types of communication that were reported by Regions include: quarterly on-site meetings to discuss projects, short and long-term goals, and problems encountered; presentations regarding the FN eHealth Infostructure to communities; and monthly phone conversations regarding project progress and barriers being encountered.
• Examples of communications described by Regions were a conference/forum held in Saskatchewan in 2010 providing HISAP workshops, and; a presentation by Quebec Region in Kahnawake on the FN Health infostructure initiative (March 2009).
• 48% (N=40) of community-level respondents indicated that there has been an increase in awareness of eHealth in their community over the last 5 years, 18% indicated there has been no increase, 22% didn't know, and 12% indicated N/A.

Immediate Outcome 4: Improved ongoing integrated planning and implementation of complex eHealth systems

There were no specific performance indicators or data collected to analyze this outcome.

Immediate Outcome 5: Increased use of evidence-based information to inform eHealth planning and implementation

Progress has been demonstrated, particularly related to planning at the national level; however there is a lack of evidence that demonstrates the use of evidence-based information to inform planning and implementation at the community and regional levels.

Key Findings

• The 2008 Interim National Directives for the FN&I eHealth Program - e-Health Solutions Units document and the 2010 HISAP document are strong examples of the use of evidence in eHealth Program planning and decision-making. Both documents draw on findings from literature reviews and relevant reports on eHealth, health of First Nations, and health system development.
• 44% (N=36) of management respondents indicated that they agreed or strongly agreed that there has been increased use of evidence-based information to inform eHealth planning and implementation.

Immediate Outcome 6: Greater use of policies, standards and guidelines for IT implementation and use

There were no specific performance indicators or data collected to analyze this outcome.

4.2.2 Core Issue #4: To what extent have intermediate outcomes (medium-term goals) been achieved?

Findings indicate there has been some progress towards the achievement of most of the intermediate outcomes. Significant gaps existed in the data sources for the performance indicators in this section.

Intermediate Outcome 1: Key stakeholders in First Nation and Inuit health are engaged in the integration of health services

Overall, it was found that progress has been made in the eHealth Program, FNIH Regions and FN communities in the development of partnerships and collaboration between stakeholders in implementing eHealth strategies and projects. From the data, it is apparent that the level of integration between FN/I eHealth stakeholders is high. Additionally, engagement in partnerships with federal and provincial governments, FN organizations, eHealth agencies, and regional health authorities emerged as the most frequently cited alternative method or approach communities explored to make their eHealth Program or projects successful.

Key Findings

• For the fiscal years 2006/07 to 2010/11, the eHealth Program was engaged in 2 Interdepartmental Letters of Agreement (ILA), 11 Contribution Agreements (CA), and 9 Memorandums of Agreement (MOA).
• Partnerships between the FNIHB eHIP and (1) Canada Health Infoway and other Federal Health Care Departments, (2) AANDC and other Federal Aboriginal Organizations, (3) Provincial Governments, and (4) FN/I Communities, Health Organizations, and NGOs have been established over the past few years, and are outlined in the 2008 document Interim National Directives for the FN&I eHealth Program - e-Health Solutions Units.
• At the community level, 50% (N=40) of respondents reported participation in regional/provincial committees discussing the integration of eHealth services, and 28% (N=40) indicated they did not participate. At the management level, 75% (N=36) responded that they did participate and 19% (N=36) indicated they did not.

Intermediate Outcome 2: Increased First Nation management of eHealth Infostructure

There were no specific performance indicators or data collected to analyze this outcome.

Intermediate Outcome 3: Increasingly integrated information for continuous improvement in eHealth Infostructure

There were no specific performance indicators or data collected to analyze this outcome.

Intermediate Outcome 4: Access to health information

There is overlap between the indicators in this outcome and those of outcome P1.1 "Improved access to eHealth Infostructure services".Please refer to Section 4.2.1 for findings related to the number and type of eHealth information tools implemented across FN communities.

4.2.3 Core Issue #4: To what extent have long-term outcomes (long-term goals) been achieved?

There has been varied progress towards the achievement of most of the long-term outcomes. However, it should be noted that these results are not to be fully expected until 2020.

Long-Term Outcome 1: An EHR capacity and capability for First Nation and Inuit by 2020 and seamless integration with provincial EHR systems

An EHR capacity and capability for all First Nations and Inuit with seamless integration with provincial electronic health systems has achieved moderate progress. As previously discussed in Section 4.2.1, there has been some progress towards achieving this vision in terms of EMR planning, however satisfaction levels are low regarding the availability of EMR/EHR and results are not fully expected until 2020.

Long-Term Outcome 2: Promote innovative integrated health governance relationships

Good progress towards this outcome has been demonstrated. Findings indicate that innovative and integrated health governance relationships are being built between the Regions, communities, provincial governments, the federal government and other relevant stakeholders.

Key Findings

• The Pacific and Saskatchewan Regions appear to be on the forefront of building innovative partnerships.
  • BC is engaging in several innovative and integrated health governance relationships to jointly plan, deliver, and fund integrated services for the First Nations and Inuit (eg. Tripartite First Nations Health Plan established in 2007, BC First Nations eHealth Centre of Excellence, FN Telehealth Expansion projects, and eHealth Strategy Plan Project).
  • In August 2008, a tripartite MOU on First Nations Health and Well-Being in Saskatchewan was signed between The Federation of Saskatchewan Indian Nations, the Government of Canada and the Government of Saskatchewan with several goals, one being to adapt and better integrate health and wellness programs of all jurisdictions by eliminating duplication, closing gaps and improving the coordination and efficacy of the health care systems.
• For fiscal years 2006/07 to 2010/11, the eHealth Program has had partnerships and agreements with 22 groups/organizations.

Long-Term Outcome 3: Improved FN capacity to influence and/or control (design, deliver, and manage) eHealth Programs and services

Little progress has been made for this outcome as demonstrated by high levels of dissatisfaction reported among community respondents with the governance of the program components. Stakeholders want to be more involved in the planning and/or control of the major eHealth Program components. Many stakeholders are unclear about their current role in the planning or control of the eHIP components. This needs to be addressed in order to improve overall FN capacity for influencing and/or controlling programs and services within the context of eHealth.

There were gaps in the program documentation for the performance indicators in this section.

Key Findings

• 50% (N=36) of management respondents believe the eHealth Program supported their organization's ability to influence and/or control eHealth programs or services; community stakeholders are less in agreement, with 32% (N=40) believing they are being supported (32% no, 18% didn't know, and 18% N/A).
• On average, 20 - 33% of community respondents are not satisfied with their role in planning eHealth activities as demonstrated in Table 6. In addition, a similar number of respondents indicated that they were dissatisfied or very dissatisfied. This would indicate the need to strengthen a community's role in this area.

Table 6 - Community Respondent Satisfaction with their Current Role in the Planning and Implementation of eHealth Program Components (N=40)

	Very Unsatisfied	Unsatisfied	Neutral	Satisfied	Very Satisfied	Don't Know	N/A
a. Broadband Connectivity	2%	20%	25%	20%	8%	5%	20%
b. Public Health Surveillance Tool (i.e., Panorama, or equivalent)	2%	22%	32%	20%	2%	5%	15%
c. Telehealth	5%	15%	28%	30%	2%	2%	18%
d. EMR/EHR	2%	15%	25%	25%	0%	8%	25%
e. IT Technical Support, Maintenance, and Capacity Development	0%	25%	30%	22%	2%	2%	18%
f. Information Management	2%	25%	25%	25%	0%	2%	20%

4.2.4 Core Issue #5: Assessment of Economy and Efficiency

The assessment of economy and efficiency demonstrates that further emphasis on increased utilization of eHealth services is required to ensure overall eHIP cost effectiveness.

The evaluation attempts to assess efficiency and economy under Core Issue #5: Demonstration of Efficiency and Economy as outlined in the 2009 TB Policy Directive on Evaluation. That is, an assessment of resource allocation and utilization in relation to the production of outputs and progress toward expected outcomes.

The Government of Canada (GoC) Policy on Evaluation (2009) defines the demonstration of efficiency and economy as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Within the realm of program activities and FNIHB activities in general, there is considerable difficulty in measuring economy and efficiency in terms of comparison, alternative approaches, and attribution of the outcomes.

Specifically, the evaluation framework's matrix for economy and efficiency outlined four questions and set out to measure economy and efficiency using a standard set of performance indicators.

Those questions are:

• How has the eHealth Program optimized the overall quantity, quality, blend of products and/or services to facilitate achievement of the program's expected outcomes?
• Are there alternative methods which ensure the same achievement of immediate outcomes?
• Has the eHealth Program minimized allocated resources while maximizing outputs?
• Were the eHealth Program's resources managed to facilitate the achievement of relevant immediate outcomes?

However, due to a lack of financial performance data, the following approach was taken to assess resource allocation and utilization using four methods outlined below. They included:

1. addressing questions about economy and efficiency at the level of program implementation and delivery stage (activities, outputs) through an assessment of available financial data (expenditure allocation review);
2. obtaining clarification of data expenditure trends through interviews with key program staff (HQ);
3. obtaining and reviewing qualitative data through key stakeholder surveys (community and management-level) on opinions regarding the factors affecting and/or influencing the achievement of long-term outcomes as they relate to resource availability and/or resource allocations; and
4. theoretical modeling to examine whether telehealth services aimed at First Nations and Inuit communities in Canada are cost effective; whether electronic health and medical records (EHR/EMR) improve service efficiency; and whether public health surveillance tools (ie., Panorama, or equivalent) are cost effective.

4.2.4.1 Resource Allocation and Utilization

Financial data and related analysis provides an overall assessment of the impact of expenditure allocations in the context of resource utilization. This analysis included an examination of program delivery costs (direct and indirect salary, operating/maintenance costs and grants and contributions), including cost drivers, resource allocations (by program component) as well as cost/output ratios. In most cases, trend data was reviewed to understand how expenditure allocations affected program delivery and potentially, expected outcomes.

Total resources of $131,747,810 include the combined eHIP funds received by FNIHB National Office, FNIH Regional offices and distributed to FN/I communities. Contributions to the total funding may include resources made by other FNIH programs, regional funds re-allocated within given fiscal years and one-time special investments made by the Branch.

On average, the overall program expenditures per year were $ 26,349,561.38 across the 5 years reviewed (2006/07 - 2010/11). There was little variance from year to year of total expenditures (average 2%) as illustrated in Figure 3.

Text description

Figure 3 illustrates that, on average, the overall program expenditures per year were $ 26,349,561.38 across the 5 years reviewed (2006/07 - 2010/11). There was little variance from year to year of total expenditures (average 2%).

For the five fiscal years evaluated, expenditures by FNIHB and FNIH Regional offices, varied greatly, with the greatest percentage of expenditures in Ontario, Alberta and Pacific regions, as indicated in Figure 4.

Text description

Figure 4 indicates that, for the five fiscal years evaluated, expenditures by FNIHB and FNIH Regional offices, varied greatly, with the greatest percentage of expenditures in Ontario, Alberta and Pacific regions.

For most FNIH Regional offices, as well as the NCR national office, year-over-year expenditures varied by only <10% (on average) with the exception of Ontario and Pacific where the greatest fluctuations in expenditures were in 2008/09 and 2009/10 varying as much as a decrease of between 50 - 80%. Greatest fluctuations were in expenditures associated with contribution agreements. Fluctuations exist given that funding allocations for community-based projects are based on a number of criteria that impacts the fiscal allocations for each FNIH Region. These criteria include: allocations provided based on number and size of proposals approved; previous year funding allotments and priorities for longer-term project funding; and impact of future strategic priorities and funding cycles. Population size within each region also impacts on funding requests and allocations. The Northern Region received funding only in 2007/08.

Across the five years evaluated, total expenditures were tracked by salary, operating and maintenance, capital (minor) and contributions (agreements through FNIH Regional offices to FN/I communities and/or NAOs). Contributions comprised the majority of total expenditures (67.5%) while operating and maintenance costs comprised 16% and salaries 15.4% of total expenditures (see Figure 5).

Capital expenditures were made only in the first two years of funding (2006/07 and 2007/08) in the Atlantic, Manitoba, Alberta and Pacific Regions. The capital costs represent less than 1.1% of the total expenditures across the five years examined. Capital costs were initially covered by HQ but subsequently through contribution agreements within FN/I communities.

Text description

Figure 5 illustrates that, across the five years evaluated, total expenditures were tracked by salary, operating and maintenance, capital (minor) and contributions (agreements through FNIH Regional offices to FN/I communities and/or NAOs). Contributions comprised the majority of total expenditures (67.5%) while operating and maintenance costs comprised 16% and salaries 15.4% of total expenditures.

Despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expenditures (Figure 6 - Actual Costs and Figure 7 - Actual Costs as a Percentage) for comparative purposes. Nearly half of the investments in eHealth over the past 5 years were made in Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building.

A quarter of investments were made in Information Management (Business Strategy, Management and Service Manager and Policy and Program Development). The remaining 25% of investments were split between Public Health Surveillance, Telehealth and Emerging Technologies (including EMR/EHR and emergency mobile technology).

Text description

Figure 6 illustrates that, despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expenditures, in particular, the actual costs to the eHIP program. Nearly half of the investments in eHealth over the past 5 years were made in Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building.

Text description

Figure 7 illustrates that, despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expeditures, in particular, the actual costs as a percentage of overall costs for Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building. A quarter of investments were made in Information Management (Business Strategy, Management and Service Manager and Policy and Program Development). The remaining 25% of investments were split between Public Health Surveillance, Telehealth and Emerging Technologies (including EMR/EHR and emergency mobile technology).

4.2.4.2 Review of Stakeholder Opinions

Key Findings

Through the key stakeholder's survey (community and management-level), questions regarding the likelihood of achieving long-term outcomes raised a number of issues specifically as they relate to funding availability and/or funding allocation. These issues were identified in open-ended questions regarding challenges and/or barriers to the Program's success.

For community respondents funding was the most frequently discussed topic with a majority of respondents (N=40) indicating that funding is insufficient, and this lack of funding is slowing progress on outcomes.

Funding sustainability was consistently identified by community level stakeholders and was evident throughout the program documentation as the primary barrier to implementing and maintaining eHealth tools in FN communities. Funding needs to be adequate for both implementation projects and ongoing operations. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate.

Stakeholders indicated that funding needs to be appropriate to build required capacity, and, that more clarity is required regarding what initiatives will be funded, how they will be funded, when funding will be received, and stipulations and requirements around funding.

4.2.4.3 Theoretical Modeling

Key Findings

Several eHealth services were included in theoretical modeling to determine their cost effectiveness. These include telehealth, electronic health records and electronic medical records, and public health surveillance systems (i.e., Panorama, or equivalent). For each component, the process required a review of current literature to derive a cost effectiveness model, and outlined the implications for the Canadian healthcare system. It is shown that it is possible for these services to be cost effective, but that the cost effectiveness is strongly dependent on the utilization of the service.

Telehealth

Many telehealth programs have been implemented worldwide to provide service to remote populations. Medical benefits to the service users, as well as cost effectiveness have been shown in many instances. A model developed to gauge cost effectiveness showed that a telehealth system can be cost effective if the product of the fraction of patients who use the services and the fraction of users who avoid travelling is greater than 0.5. The breakeven point is slightly greater than 0.5 by a value determined by the cost of running the service, the money spent on medical travel, and the size of the population that uses the service. Analysing several past projects, it is found that their cost effectiveness can be explained by this model, and that as a whole, telehealth projects in Canada can be cost effective.

Electronic Health Records and Electronic Medical Records (EHR/EMR)

Electronic Health Records and Electronic Medical Records (EHRs and EMRs) have the potential to improve the efficiency of the Canadian healthcare system. Implementation in other countries has been shown to be financially viable. A model to determine the cost effectiveness has been derived, but data is deficient from Canadian EMR and EHR systems. An EMR/EHR system can be cost effective if utilization is high and if the costs of running the system are less than the money saved by using electronic rather than paper records. It is recommended that data be gathered in order to better determine the cost effectiveness.

Public-Health Surveillance and Related Systems (i.e., Panorama, or equivalent)

Public Health Surveillance is understood to be the ongoing, systematic use of routinely collected health data to guide public health action in a timely fashion. Surveillance is descriptive in nature, reporting on occurrence of injury or disease and their treatments as well as broader factors that determine health or factors that put some people at increased risk of injury or disease. Such systems have been shown to be cost effective in other countries. Using the recent SARS outbreak as an example, the potential savings are in the hundreds of millions of dollars.

The outlook of cost effectiveness of a Public Health Surveillance system is favourable. Similar projects in other countries have been shown to be economically favourable and simple economic models used to predict cost effectiveness show that it can be achieved with realistic parameters.

5. Conclusions

5.1 Relevance

Findings from this evaluation of the FN&I eHIP suggest that it is highly relevant, but improvement is required particularly in EMR/EHR, and IT capacity building program components to facilitate further progress and the achievement of expected results. FNIHB and FNIH Regions continue to make progress towards integration. As a result, there is strong evidence that supports continued investment in eHealth.

Evaluation findings suggest that the eHIP continues to address a demonstrable need and is responding, albeit with varying success across FN/I communities, to the healthcare and health technology needs of FN/I. The priority areas of the eHealth Program are well aligned with the needs of FN/I communities, and the projects and program components are useful and relevant to the needs of the FN/I communities.

The eHealth Program's outcomes are well aligned and consistent with federal government priorities, roles and responsibilities, and departmental strategic outcomes. The federal government considers eHealth a priority, and the evaluation findings illustrate that the eHealth Program and its business activities and outcomes are in alignment with various federal organizations including RAPB and AANDC.

Some areas for improvement were noted such as addressing concerns about lack of sustainable funding mechanisms to support projects and, ensuring that the Program supports increased community-based governance of eHealth projects, and eHealth capacity building in FN communities.

5.2 Performance

There has been varied progress towards the achievement of the eHIP's immediate, intermediate and long-term outcomes. However, inadequate and inconsistent Program data for many performance indicators makes it difficult to confidently and accurately draw comprehensive conclusions regarding the achievement of outcomes. Long-term outcomes are not expected to be fully realized until 2020.

Immediate (Short-term) Outcomes

There is evidence indicating progress towards improving access to some eHealth Infostructure services (e.g. high-speed connectivity, Telehealth and e-SDRT), while more effort is required in other services such as public health surveillance tools and EMR/EHR.

Integration of a Public Health Surveillance System (i.e., Panorama, or equivalent) is dependent on provincial selection and implementation of their respective tool.

Continued effort to ensure effective connectivity, support, maintenance, capacity development and information management are all components of the core infrastructure required to provide effective eHealth services. Findings suggest that Telehealth has been the most successful component. Further effort is required to ensure that a public health surveillance tool (that is compatible with its respective provincial system) will improve the management of public health information. EMR/EHR is a leading F/P/T priority.

Progress has been demonstrated in the use of evidence-based information to inform eHealth planning and implementation, particularly related to planning at the national level (i.e., in the development of strategic plans such as the HISAP and interim directives), however there is a lack of evidence that demonstrates the use of evidence-based information at the community and regional levels, particularly in the areas of improved capacity and capability of the work force to use IT as part of service delivery.

Intermediate Outcomes

One of the major strengths of the eHIP is progress made towards the development of partnerships and collaboration between stakeholders in implementing eHealth strategies and projects. From the data, it is apparent that the level of collaboration between FN/I eHealth stakeholders is high. Additionally, engagement in partnerships with federal and provincial governments, FN organizations, eHealth agencies, and regional health authorities emerged as the most frequently cited alternative method or approach communities explored to make their eHealth Program or projects successful.

There is progress being made towards improving access to health information, as assessed by the number and type of eHealth information tools implemented across FN communities. However, this progress varies across each eHealth tool and from FN community to community.

Long-Term Outcomes

Progress has been demonstrated towards building innovative and integrated health governance relationships between FNIH Regions, communities, provincial governments, the federal government and other relevant stakeholders through the establishment of key effective partnerships.

Although the focus of this evaluation was not on the long-term outcomes, there is evidence to show that some progress has been made in many areas including developing partnerships, planning for EHRs, and integrating public health surveillance systems with the provinces. There remains a need to focus on assessment planning and full implementation of the eHealth program components in order to achieve the desired long term outcomes.

Overall, these conclusions highlight one of the main issues to emerge from this evaluation - the eHIP is highly relevant to addressing the needs of the FN communities, and aligns well with federal priorities, however continued progress will be achieved by addressing:

• Insufficient or unavailable program and financial performance data to support performance measures;
• Program outcomes that better align with activities and progress occurring at the community and Regional levels. The eHIP's current program outcomes are mostly IT driven, focusing on the progress of deployment/implementation of eHealth tools and infostructure development. However, if outcomes were to be more reflective of improvements made in the eHIP's business processes (which is occurring extensively across Regions and communities as they prepare and adapt to changes such as privacy policies and procedures, information management and data sharing agreements, etc.), this would facilitate the achievement of outcomes;
• A need for mechanisms specifically required to address improving FN community capacity; and
• A need for mechanisms to ensure Program-wide communications that increase awareness of eHIP's activities, expected outcomes and, opportunities for stakeholders to be involved in both planning and decision-making to support the achievement of the expected results.

Funding sustainability was consistently identified by stakeholders and was apparent throughout program documentation as the primary barrier to implementing and maintaining eHealth tools in FN communities. Funding needs to be adequate for both implementation projects and ongoing operations. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate. More clarity is required regarding what initiatives will be funded, how they will be funded, when funding will be received, and stipulations and requirements around funding.

Based on an analysis of expenditure investments across the six program components, effort should be made to re-examine where future investments are made to ensure appropriate resource allocations that maximize the achievement of program outcomes. For example, interoperability with provincial/territorial systems requires significant investment, and so a possible re-examination of both when and how resources are allocated should be incorporated into a long-term business case.

Theoretical modeling has shown that connectivity can improve both the efficacy of a health care service as well as the cost effectiveness. Cost effectiveness depends strongly on utilization. It is paramount that any technology-supported health information system that is implemented be used as much as possible (use maximization), or else it may not be cost effective.

6. Recommendations

Recommendation #1 - Create a Single Comprehensive and Detailed Business Case

The eHIP should consider preparing a comprehensive and detailed business case, with a focus on increasing adoption of modern systems, change management strategies, and utilization within communities to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities. Findings suggest more detailed information regarding the eHIP's highest-level organizational goals (and how it will achieve them) is required, and such a plan would provide this level of detail.

A comprehensive and detailed business case would: allow eHIP to align and tie together important concepts and issues; form a complete picture of their Program (components, partners/stakeholders); help the Program anticipate financial requirements well into the future; ensure the Program's roles and responsibilities are clearly identified; provide an opportunity to track and closely study the national and international eHealth landscapes; and provide an opportunity to track and closely study the continuously evolving needs of the program's end users.

The business case would help the eHIP achieve the "improved ongoing integrated planning of complex eHealth systems" immediate outcome, as well as the "use evidence-based information to inform eHealth planning" immediate outcome. Consideration of best practices and building on past successes must also be incorporated into the business case.

eHealth and the integration of eHealth services is a very complex undertaking. A single comprehensive strategy and planning document will help to bring cohesion and alignment to this complex subject.

Recommendation #2 - Prioritize Investments and Further Clarify Funding Policies and Practices

Due to the high levels of relevance and alignment of the eHIP with stakeholder needs and federal government priorities, it is recommended that there is a continued investment of resources in all six eHIP components as identified in the HISAP in order to facilitate the achievement of long-term outcomes.

eHealth projects and services are capital and HR intensive to execute, implement and operate on an ongoing basis. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate. The eHIP needs to ensure funding is adequate for both implementation projects and ongoing operations.

The eHIP should continue to monitor the readiness of FNIH Regions, communities and provinces in each program component to determine annual investments. In order to distribute funding accordingly, each FNIH Region should continue to provide an individualized annual workplan to identify the status of each program component and the planned goals/activities for the following fiscal year. Analysis should be performed on an ongoing basis as priorities shift and provincial public health surveillance implementation plans are advanced.

Funding should be aligned with needs and priorities of FN communities through a readiness assessment. Since eHealth priorities and progress vary significantly between communities, the eHIP should be flexible in working with communities to understand their unique needs and develop solutions.

Findings indicate more clarity is required by stakeholders regarding what initiatives will be funded, how they will be funded, when funding will be received, and other stipulations and requirements around funding.

Recommendation #3 - Build eHealth Capacity of FN Communities

The development and implementation of a community-based training strategy will improve the performance satisfaction levels of stakeholders with eHealth tool training available, and furthermore will support the FN&I eHealth Program's outcome of improving the capacity and proficiency in the use of eHealth applications in FN communities.

A community-based strategy allows for each FN/I community to assess gaps in eHealth knowledge, to identify training needs of service providers in their community, and to develop a strategy that is tailored to meet their specific needs. FN/I communities should be provided with sufficient guidance and support in the development and execution of their eHealth tool training strategy from Regional and/or Federal bodies to ensure consistency in the quality and availability of training across communities. A community-based training approach will also support greater involvement of FN/I communities in the planning and governance of eHealth.

Innovative methods of training delivery should be explored to improve availability of training opportunities, such as partnerships between FN/I communities in sharing training resources, and leveraging videoconferencing systems to support remote training.

Recommendation #4 - Implement a Comprehensive Communications Approach

The development and implementation of a Program-wide, multi-pronged, communications approach will increase awareness of eHIP's activities across all Regions and communities (one of the eHIP's immediate outcomes). It will also facilitate dialogue between stakeholders at various levels of government, as well as external stakeholders, and will provide greater opportunity for stakeholders at the community level to be involved in eHealth program planning and decision-making.

Keeping community level service providers and other stakeholders informed of the status of projects, upcoming initiatives, staffing changes, etc. is important to facilitate collaboration and build a positive work environment. Stakeholders have indicated their support for increased frequency and level of communication with FN communities and organizations as well as other partners including provincial partners.

Messaging in the communications should also seek to provide information about the benefits that have been realized from implementing eHealth services, about training and education opportunities, employment opportunities, eHealth progress related to goal achievement, and progress of eHealth tool implementations and services (e.g. lab information systems, public health surveillance systems, Telehealth, etc.).

Recommendation #5 - Implement a Refined Performance Measurement Strategy

The eHIP should refine and implement a performance measurement strategy to:

• Collect better quality performance data including financial data that supports economy and efficiency in the context of value for money;
• Improve the consistency of data collected across Regions;
• Better measure success against established targets;
• Support informed decision making; and
• Facilitate future evaluations.

Components of this strategy could include:

• Aligning Regional reporting requirements with program performance indicators;
• Adding an element of time to outcomes to allow for better measurement of progress and achievement of outcomes;
• Developing protocols for data collection to be used at all levels of program delivery; and
• Implementing a performance measurement dashboard.