Evaluation of the Health Information Initiative 2012-2013 to 2014-2015

Prepared by Office of Evaluation - Health Canada and the Public Health Agency of Canada

December 2014

Executive Summary
1.0 Evaluation Purpose
2.0 Program Description
- 2.1 Program Context
- 2.2 Program Profile
- 2.3 Program Logic Model
- 2.4 Program Alignment and Resources
3.0 Evaluation Description
- 3.1 Evaluation Scope, Approach and Design
- 3.2 Limitations and Mitigation Strategies
4.0 Findings
5.0 Conclusions
- 5.1 Relevance Conclusions
- 5.2 Performance Conclusions
6.0 Implications for the Future
Appendix 1 - Logic Model
Appendix 2 - Coverage of CIHI data holdings
Appendix 3 - Summary of Findings
Appendix 4 - Evaluation Description
Appendix 5 - Jurisdictional adoption of Case Mix-based funding
Appendix 6 - Stakeholder Engagement Framework

List of Tables

Table 1: Limitations and Mitigation Strategies

Executive Summary

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Health Information Initiative (HII) for the period 2012-2013 to July 2014. The evaluation follows a supplement to an independent evaluation commissioned by the Canadian Institute for Health Information (CIHI) that was approved by the Deputy Minister, Health Canada in July 2013. The supplemental evaluation focussed primarily on relevance to comply with requirements of the Financial Administration Act.

A full evaluation, i.e. an evaluation covering all core issues required under the Treasury Board Policy on Evaluation (2009).

Program Description

CIHI was created in 1994 in response to the recommendations of the National Task Force on Health Information (Wilks Report) which were presented to the Conference of Deputy Ministers of Health in 1991. The Task Force suggested that better health information systems were necessary to ultimately improve the health of Canadians.

The organization collects, analyzes and disseminates information on the health care system, primarily by gathering administrative data from provincial/territorial governments, hospitals, regional health authorities and medical practitioners. CIHI maintains pan-Canadian datasets of nationally comparable information necessary to determine the relative effectiveness of investments intended to address various health system issues.

CIHI is guided by a Board of Directors proportionately constituted to ensure a balance among health sectors and regions of Canada. The board provides leadership to CIHI on policy directions and advises the Conference of Deputy Ministers of Health on approaches to promote the effective use of health information in policy and decision making.

Since 1999, the Government of Canada has been CIHI's primary source of funding (78 % as of March 31, 2014). The HII, established in 2007, refers to the consolidation of these funding vehicles previously in place to support CIHI. As of 2012, the funding instrument is a contribution with Terms and Conditions rather than a grant. CIHI currently receives $77.7 million a year in contribution funding.

Conclusions - Relevance

There is a continued need for comparable health information pertaining to the full spectrum of care. CIHI is currently the only organisation in a position to collect such information at the pan-Canadian level although provincial level capacity increased significantly over the last decade. CIHI provides a mechanism that is respectful of jurisdictional roles and responsibilities in meeting Canadians' expectations for federal involvement in health and contributes to inform decisions geared toward system sustainability and responsiveness to population needs.

While the field of health information improved significantly in the last twenty years, the need for comparable health information to manage the health system, improve its performance and ensure its sustainability remains in a country where health expenditures consume on average 40% of provincial and territorial budgets.

The information necessary to improve health system performance needs to go beyond the hospital setting and to allow for the evaluation of outcomes of care, the planning of health services and for the appropriate allocation of resources. The provincial capacity in health information increased as the majority of provinces now have established health information organizations responsible for monitoring and evaluating health services. While some activities are similar to those of CIHI, only CIHI has the mandate to collect comparable pan-Canadian data, which appear more important to jurisdictions with less data management capability who continue to rely more heavily on CIHI support.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

CIHI has increased awareness, understanding and uptake of the standards and products related to data infrastructure which CIHI uses to engage stakeholders involved in the processing of health data. However, findings related to CIHI's ability to engage stakeholders more broadly are mixed.

Numerous uses of information produced by CIHI have been reported particularly, for the conduct of secondary analysis. While identifying examples where CIHI information contributed to concrete and broad policy or practice improvements proved to be challenging, the work of CIHI to adapt international tools to the Canadian context may facilitate the adoption of electronic health records, contribute to the expansion of activity-based funding and foster the culture of performance that is currently emerging.

While CIHI is best known for the collection and publishing of health care data, stakeholders' awareness of the full suite of products and services is not as high as could be expected considering CIHI is an established and well-known organisation. In this respect, CIHI engagement efforts should aim to move the culture toward one that values the use of evidence as part of a process of continuous performance improvement for the health system. CIHI technical support is highly valued and contributes to build stakeholders capacity to collect data in a reliable manner. This suggests that the ongoing interactions between CIHI and the provinces and territories facilitated a gradual cultural shift within health organisations where the importance of measurement and standardized data sets is now more broadly accepted.

While views vary as to whether CIHI should play a greater role in identifying clinical and/or administrative changes to realize greater efficiencies, CIHI may be well positioned to facilitate the uptake of best practices identified as a result of data analysis by provincial health information organisations.

Given the complexity of the environment in which CIHI operates, efforts to promote the use of output specifications for health data software, using specifications that were developed by CIHI, may contribute to data comparability as Electronic Health Records gain ground.

Demonstration of Economy and Efficiency

HII is delivered in an economical and efficient manner. Processes ensure the timely release of data and the transfer payment to CIHI is an appropriate funding mechanism to support CIHI activities.

The absence of output/outcome-specific financial data limited the ability to assess resource utilization as part of this evaluation. In terms of efficiency, the number of datasets releasing timely data increased significantly since 2001, although some provincial representatives still consider the time lag between data submission and report release to be too long.

The Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations observed that a transfer payment is an appropriate way to advance federal/provincial/territorial priorities for the healthcare system, although there could be increased efficiencies if CIHI and Infoway better integrated their activities as they work to ensure that Electronic Health Records consider all opportunities and risks for the use of data held by these systems.

Implications for the Future

Formal recommendations are not being proposed to Health Canada. Based on the findings and conclusions outlined in this evaluation report, the department is aware that changing health information circumstances may have future implications for the Canadian Institute for Health Information. Respecting the complex environment in which CIHI operates, CIHI may consider developing a vision which would guide a move beyond data management and bilateral interactions with stakeholders.

After 20 years of operations, a renewed vision should contribute to guide the organisation in becoming an agent of change by helping with:

Modernisation of CIHI mandate

Since its creation, CIHI's mandate has been focused on developing a pan-Canadian data infrastructure. Given the progress accomplished in this respect, it may be time for the organisation to reflect on what CIHI could do to take the next step and for them to facilitate health stakeholders' adoption of evidence-based processes and practices for improving the efficiency and effectiveness of the health system.
Refining the stakeholder engagement strategies to deliver on this mandate

CIHI stakeholder engagement strategies may benefit from being better balanced between consulting to identify priorities and the need to facilitate adoption of processes and practices for improving the health system. The stakeholder engagement strategies should be focussed on the stakeholders carrying the greatest potential to initiate the large-scale changes necessary to improve health system efficiency and effectiveness.

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Health Information Initiative for the period of 2012-2013 to July 2014. The evaluation follows as a supplement to an independent evaluation commissioned by the Canadian Institute for Health Information (CIHI) that was approved by the Deputy Minister, Health Canada in July 2013. The supplemental evaluation focussed primarily on relevance to comply with requirements of the Financial Administration Act.

A full evaluation covering all core issues required under the Treasury Board Policy on Evaluation (2009).

2.0 Program Description

2.1 Program Context

CIHI was created in 1994 in response to the recommendations of the National Task Force on Health Information¹ (Wilks Report) which were presented to the Conference of Deputy Ministers of Health in 1991. The Task Force suggested that better health information systems were necessary to support health care technologies and services; to improve management and policy processes; to help reconcile public expectations for expanded services and the need to control health care costs; and ultimately, to improve the health of Canadians.

The findings of the Task Force indicated that the development of a Canadian health information system could not be 'adequately discharged' by any government organization and that non-governmental operating capacity was required to be acceptable to all health constituencies to oversee, direct and coordinate the evolution of improved health information systems in Canada.

In Canada, the provinces and territories have the primary responsibility for the delivery of health care. The federal government, which provides transfer payments to support provincial and territorial health care systems, works in partnership with the provinces and territories to promote and protect the health of Canadians. CIHI operates in an increasingly complex environment where the respect for jurisdictional roles and responsibilities is essential, while providing good information to achieve a high performing health care system which is responsive to the needs and expectations of Canadians.

Since 1994, the Government of Canada has largely funded CIHI's work through a series of funding agreements referred to as the 'Roadmap Initiative' and the Health Information Initiative. The Health Information Initiative (HII), established in 2007, refers to the consolidation of these funding vehicles previously in place to support CIHI. As of 2012, the funding instrument is a contribution with Terms and Conditions. Since 2012, the HII provides CIHI with approximately $82 million annually.

CIHI has received approximately $650 million in federal funding since 1999. While resources will be presented in more detail in section 2.4, the HII provides $77.7 million a year in contribution funding. The latest contribution agreement, which was for a three year period (2012-13 to 2014-15), will expire March 31, 2015.

2.2 Program Profile

CIHI's mandate is to develop and maintain comprehensive and integrated health information that enables sound policy and effective management of Canada's health system, so that healthcare delivery is improved. The HII's goal is to modernize Canada's health information system and to improve accessibility and quality of health information for use by health care professionals and the public. CIHI maintains datasets of nationally comparable information necessary to determine the relative effectiveness of investments intended to address various health system issues, such as wait times and human resources management.

According to the Terms and Conditions, HII objectives are to:

Help provide the information necessary for health care providers and health system managers to measure and report on performance;
Help provide the evidence base necessary for health care providers and health system managers to make informed decisions about health system renewal;
Respond to emerging health information priority needs; and
Help create the information necessary for Canadians to make informed decisions about their health and the use of health care services.

While information generated through HII is expected to benefit all levels of government as well as other stakeholders such as regional health authorities and health professionals, current arrangements with CIHI clearly state that the provision of data to Health Canada is not a requirement of the contribution agreement. In this context, Health Canada's main activities related to HII are managing and monitoring CIHI's progress and compliance with the provisions of the contribution agreement.

2.2.1 CIHI Activities

CIHI collects, analyzes and disseminates information on the health care system, primarily by gathering administrative data from provincial/territorial (P/T) governments, hospitals, regional health authorities and medical practitioners. CIHI's data and reports focus primarily on health care services, health spending and health human resources.

CIHI provides national leadership for health system information through the development and maintenance of 28 broad range pan-Canadian health databases, measurements and standards. In order to provide comparable health system information, CIHI:

coordinates and promotes development and maintenance of national health information standards;
develops and manages health databases and registries; and
develops health indicators and conducts analysis.

Stakeholders access CIHI datasets through various mechanisms, some of which are publicly accessible while others are available on a fee-for-service basis as stipulated in data sharing agreements with provinces and territories.

Examples of applications available through restricted access include:

web-based applications for data submission;
the CIHI Portal which provides registered users with access to pan-Canadian comparative data and with the ability to create customized reports;
the coders' resource page,
custom data and documents,
the eQuery tools which are a repository of questions and answers about data-related topics such as data quality, coding, and grouping methodologies;
the health indicators interactive tool.

CIHI also provides education-related activities and technical support to their clients to undertand how to use and implement data standards (and other data management tools) during data collection. Over time, capacity building activities have contributed to the improvement of the quality and completeness of data submissions from jurisdictions.

2.2.2 CIHI Governance

CIHI is guided by a 15-member Board of Directors² proportionately constituted to create a balance among health sectors and regions of Canada. The Board is comprised of:

Ten regional directors, where each of the five regions is represented by two directors (five nominated by the P/T governments; and five associated with a region who are not associated with the government);
Two directors from the federal government (Deputy Minister - Health Canada, Assistant Chief Statistician - Statistics Canada);
Two directors at large not associated with the government; and
One director at large, independent of government or non-government organizations, who will act as the chair.

The Board is intended to provide strategic leadership and guidance necessary for CIHI to establish its policy directions while internal management is primarily the role of the President and CEO. The Board identifies the roles of CIHI and the Health Statistics Division of Statistics Canada in achieving the broad national vision for health information and advises the Conference of Deputy Ministers of Health on health information matters and on how to promote their effective use in policy and decision-making.

2.2.3 Collaborating with Partners

The intended reach of CIHI is national, engaging a broad number of diverse stakeholders including healthcare institutions, the federal, provincial and territorial governments, regional health authorities and agencies, and not for profit organizations and associations that are involved in developing health policy and/or managing the delivery of health services. Through consultation, CIHI regional and national offices identify emerging and/or changing needs of the health care sector as well as receiving input and feedback on the utility of CIHI products and services³.

CIHI is a member of Health Canada's Health Information Policy Committee (HIPC), which provides Health Canada, Public Health Agency of Canada (PHAC), Statistics Canada and CIHI the opportunity to share information on health data and its analysis. CIHI and Canada Health Infoway (Infoway) are the national co-leads on the Health System Use (HSU) of electronic health data. As co-leads, CIHI and Infoway work to ensure the use of health digital data is maximized in support of health system management, while being used appropriately from a privacy perspective⁴.

2.3 Program Logic Model

For the purpose of this evaluation, the Office of Evaluation revised the Logic Model developed by CIHI in 2012 to reflect increased focus on intermediate and long term outcomes, given that CIHI has been in existence for 20 years.

CIHI's objectives (Section 2.2), which include the provision of information to support accountability and decision making in the administration of the health system, suggest that CIHI's long term expected outcome is 'improved management and efficiency of the Canadian Health System'.

The activity areas, outputs, immediate and intermediate outcomes to achieve this final outcome are described in the following sub-sections.

Stakeholder Engagement

The Performance Audit (2014) conducted by KPMG reported that the achievement of CIHI's mandate and strategic priorities is, in large part, dependent upon understanding the needs of the healthcare sector and key stakeholder groups, and the alignment of CIHI funding and programming with these needs. Mechanisms are in place to engage stakeholders at the program, regional office, and/or senior management level to identify, analyze, and action their needs. These mechanisms include:

regular interaction between CIHI national and regional representatives and data providers/data analysts regarding the provision of data;
hosting conferences with partners to support various objectives such as coordinating progress towards e-health systems or the promotion of the use of available data;
convening forums which include a small number of key stakeholders from across the country to share information on national practices and concerns related to specific topics of interest (e.g., high-user forum) to inform CIHI's analytical approach to these topics.

KPMG further notes that interactions across various levels within CIHI allow the CIHI executive committee, with involvement of the Board, to identify emerging and/or changing needs of the healthcare sector to inform changes to products and services stemming from CIHI's analytical plan. CIHI has introduced a number of formal strategies to engage stakeholders in the analytical planning process, including a Strategic Analytical Advisory Committee (SAAC) and jurisdictional consultations.

Capacity Building

CIHI assists its stakeholders by building their capacity to use established data standards and analytical products. The purpose of this support is to increase the use of data and analytical products developed by CIHI. Capacity building activities refer to the delivery of education sessions and the provision of support to jurisdictions (e.g., activity-based funding) which are intended to increase the stakeholders' understanding of CIHI standards, data and analysis products. The enhanced awareness of the full potential of CIHI products and services is intended to result in the increased use of CIHI data and analysis by health departments and other health authorities.

Stakeholders are increasingly aware of CIHI products and expertise. The increased availability of reliable and comparable health system and population health analysis is expected to increase the use of CIHI products in the conduct of secondary analysis.

Database Development and Management

This activity is intended to enhance the scope, quality and timeliness of CIHI data holdings. It involves the development of data standards, the development and management of new and mature databases and registries and the production of data quality assessments.

Recent examples of CIHI undertakings related to this activity include:

The launch of an integrated eReporting system to support performance management at the regional level in Canada and to enhance the existing e-reports with new functionality;
The support to the Canadian Multiple Sclerosis Monitoring System (CMSMS) working with data providers to increase the amount of data submissions;
The exploration of the recommendations resulting from the readiness assessment of select CIHI databases to receive data from Electronic Health Records (EHR) and take the actions necessary to allow CIHI to receive EHR data across select databases; and
The expansion of the collection of data on a number of topics including: physician-level billing, physicians on alternate payment plans, patient-level cost information, and the Canadian Joint Replacement Registry.

These activities and outputs are expected to result in data submissions that comply with the newly established standards and in more pan-Canadian compatible and timely data which would also be of greater quality in terms of comparability, reliability and timeliness.

Data Processing and Analysis

Priority health information needs are responded to by developing and disseminating timely analysis and products associated with health indicators. Working with jurisdictions, CIHI implements strategies to ensure that data received can be linked and integrated.

All activities and immediate outcomes are expected to contribute to the increased use of data in analysis and decision making in relation to health policy and/or to the management of the health system.

The connections between CIHI activity areas and the expected outcomes are depicted in the logic model, which can be found in Appendix 1. The evaluation assessed the degree to which the defined outputs and outcomes were achieved during the period under review.

2.4 Program Alignment and Resources

Since 2012, CIHI receives contribution funding from Health Canada representing close to 80% of CIHI annual revenues. The remaining revenue is from provinces and territories based on a per capita equation and sales to other organizations and from conferences and forums (e.g., e-Health Conference). Since the original allocation, the funds were also subject to Government-wide budgetary reductions due to broader reviews, as well as to corporate levies and internal reallocation. Currently, CIHI receives $77,659,979 annually in contribution funding.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 1, 2012 to July 2014 and included all funding allocated under the Treasury Board Submission.

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) five core issues related to relevance and performance. Corresponding to each of the core issues, specific questions (Appendix 4) were developed based on program considerations and senior management information needs. Progress on the implementation of recommendations of previous audits and evaluations that are of particular interest for Health Canada (e.g., development of an overarching stakeholder management framework and a customer strategy) was also examined as part of this evaluation.

Given that CIHI is a mature organization, an outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes. The evaluation also examined unintended consequences and lessons learned. A non-experimental design was used based on the Evaluation Framework document which detailed the evaluation strategy for the program and provided consistency in the collection of data to support the evaluation.

The evaluation risk assigned to the HII evaluation in the Health Canada-Public Health Agency of Canada Five-year Evaluation Plan was 'Medium'. This evaluation risk, which is based on a number of factors including program complexity and maturity, materiality, target population size and vulnerability, influenced the evaluation design and the level of effort allocated for the completion of the evaluation. Given its 'Medium' level evaluation risk, the HII evaluation was subject to a 'targeted design' evaluation and relied on the following lines of evidence:

Literature Review
The literature review focused on grey literature released by provincial governments and health departments as well as from the federal government. The purpose of this exercise was to assess the extent to which the information generated by CIHI was used in the development of the various health-related documents released by provincial and territorial organisations.

Document Review
Federal government key documents (e.g., Speech from the Throne, Federal Budgets and reporting documents) as well as corporate documents published by CIHI (Strategic Plan, 2013 to 2015: Analytical Plan, annual reports) were reviewed and analysed.

Information obtained from CIHI
CIHI's 2014 Stakeholder Satisfaction and Impact Evaluation Surveys and the 2014 draft KPMG Performance Audit Report were key data sources. There was agreement with Health Canada (ARAD) and CIHI to use these reports to avoid duplication of effort.

Results of the Stakeholder Satisfaction and Impact Evaluation Surveys were received in July 2014. The response rate for the Stakeholder Satisfaction Survey was 18%, with 650 surveys completed represents a small decline since 2012 when the response rate was 21%. Approximately a third of survey respondents were from the province of Ontario (34%) while only 3% of respondents were from Quebec, the second largest population in Canada. Given the low response rate and the proportion of respondents originating from just one jurisdiction, survey results should be interpreted with caution.

The CIHI performance audit focussed on CIHI activities related to stakeholder engagement and elements of efficiency and economy.

Key Informant Interviews
Thirty-eight interviews were conducted with key informants from diverse backgrounds (i.e., CIHI headquarters and regional staff, representatives from provincial health departments, regional health authorities and provincial health information agencies and Health Canada, PHAC and Statistics Canada senior management), including policy and data management. Interviews were conducted to gather in-depth information on CIHI's opportunities and challenges related to the availability and reporting on health information and on the use of CIHI datasets to change health policy and practice.

Data was analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. Table 1 outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 1: Limitations and Mitigation Strategies
Limitation	Impact	Mitigation Strategy
Due to the large number of potential interviewees, interviews with P/Ts were not representative.	Findings may not provide an accurate or complete picture of the situation within a province or territory.	Attempted to vary the background of P/T interviewees.
2014 Stakeholder Surveys response rate 18%. Over 34% of respondents were from Ontario.	Low response rate could affect the reliability of data. Respondents reflect the population distribution of the country but offer limited insight on experiences and views of other provinces and territories.	Triangulation with interviews for which key informants were from a broader geographic background.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

There is a continued need for health information to improve the management of Canada's health care system for both the hospital and outside of hospital settings. CIHI is currently the only organisation in a position to fulfill the continued need for comparable pan-Canadian health information allowing for benchmarking, however, over the last ten years provincial level capacity in the areas of health information and analysis has increased significantly.

The 1991 Wilks Report qualified Canadian health information as an "unmapped forest with undefined boundaries". According to Wilks:

"...health constituencies have important information needs associated with their responsibilities. Typically, those information needs - for policy, management, research, health care provision, health promotion, ... outcomes monitoring, personnel planning and training, budgeting, resources allocation - are unmet or inadequately met.... And this situation is co-existent with a vast array of data collected and accumulated by many constituencies, public agencies and care-givers.

...no health constituency is in a position unilaterally to satisfy its information needs; and mechanisms for coordination and cooperation are insufficiently effective or do not exist."

While the field of health information has significantly improved in the last twenty years, there is still a need at all levels of government to make better use of data to improve performance and to increase the sustainability of the health system.⁵

Health expenditures continue to consume on average 40% of provincial and territorial budgets⁶ which make the containment of health costs one of the key priorities for all P/T governments. Total 2011 health expenditures in Canada were estimated to be $211.2 billion⁷. This supports approximately 75,000 doctors, 360,000 nurses as well as other health professionals and ancillary health-care workers delivering services. The latest figures reported by CIHI show that there were over 2.5 million procedures performed across the system in 2007-2008. Key informants reported that while the containment of health costs is one of the key priorities for provincial governments, it is challenged by multiple factors including the growth of an aging population, the increased incidence of chronic disease and expectations for improved access to care (e.g., diagnostic tests, treatment). The document review suggests that to improve health system performance including the containment of costs, health systems managers and health care providers require information from multiple sources (e.g., hospital and non-hospital settings, community programs, physicians) to evaluate outcomes of care, to plan services to meet future needs of the population and to allocate scarce resources⁸.

Responsiveness to Needs and Priorities

At the planning stage of the evaluation, it was agreed that the examination of CIHI stakeholder engagement activities would be primarily informed by the KPMG Performance Audit, which was supplemented by findings from the 2014 CIHI stakeholder survey conducted by CIHI and by interviews with CIHI, provincial/territorial and federal officials.

CIHI has established an engagement process to ensure the organization's responsiveness to stakeholder needs. Activities associated with stakeholder engagement are intended to ensure CIHI's priorities are aligned with these needs.

The findings from the stakeholder survey are generally positive. The proportion of respondents reporting that CIHI is stakeholder focused remained somewhat stable between 2009 and 2012, however, a more significant decline was observed in 2014 where only 64% of respondents considered CIHI as a stakeholder focused organisation as compared to 73% in 2009 and 75% in 2012.

The survey also reports slightly lower satisfaction levels compared to 2012 results, particularly on aspects related to CIHI's responsiveness to stakeholder needs. Fewer stakeholders agree that CIHI strives for service excellence (from 89% to 83%), is stakeholder focused (from 75% to 64%) and responsive to their needs (from 70% to 63%).

While the majority (75%) of stakeholders still consider that CIHI products are relevant to their needs, this has dropped from a high of 92% in 2009. As a quantitative tool, the survey does not provide the perspectives of respondents to explain the decrease. Although key informants were significantly involved with CIHI, they noted improvements in the last few years in shifting from a culture of setting priorities based on the perspectives of few provinces to being an organisation more engaged and more open to the views of all provinces and territories.

While some respondents report concerns related to CIHI's ability to respond to their needs, a small majority (from 58% in 2012 to 55% in 2014) consider that CIHI should increase its engagement with them. However respondents do not seem to have high expectations of CIHI's ability to include stakeholders in their decision-making processes (KPMG, 2014) with only 15% of respondents strongly agreeing with this statement.

Provincial Capacity in the Field of Health Information

According to the 2014 stakeholder survey, 57% of respondents agreed that CIHI has the ability to provide services/support that stakeholder organisations cannot provide. The large number of respondents not in agreement with this statement may reflect the increasing provincial capacity to process and publish their own health information, as the majority of provinces have established jurisdiction-specific health information organizations. The goals of the provincial health organisations are similar. They generally measure, monitor and evaluate health service quality to support continuous improvement. While some activities are similar to those of CIHI, only CIHI has the mandate to collect pan-Canadian data.

Interviews suggest that CIHI continues to be a recognized leader in the field of health information and the only organisation that provides comparable pan-Canadian health data. Larger provinces, with their own provincial level health information organisations, note that they appreciate the availability of comparable pan-Canadian data without considering it essential. One provincial representative emphasized that it was more useful to have access to one's own provincial longitudinal data in order to identify trends and to track performance. On the other hand, smaller provinces and all of the territories have less data management capability and continue to rely heavily on CIHI support.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

CIHI activities associated with the collection of pan-Canadian health data necessary to inform decisions related to the management and improvement of the health care system continue to be well aligned with Government of Canada and Health Canada priorities.

In its June 2011 Speech from the Throne the government expressed its interest in a sustainable and accountable health system. Such a system is deemed necessary in the long term to maintain Canadians' access to a health system responsive to their needs (Health Canada's PAA strategic outcome 1.0). Budget 2012 and 2013, as well as the 2013 Speech from the Throne reiterated that supporting families and communities by protecting health, and specifically patient safety, is a key priority for the current federal government." As recently as January 2014, the Minister of Health acknowledged the importance of data in advancing innovative and cost-effective health care practices.

In this context, the data infrastructure developed by CIHI, the organisation's technical expertise as well as the health system data collected with the support of federal funding appears to be necessary for the Government of Canada to foster a sustainable and accountable health system.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

CIHI activities align with federal roles and responsibilities by providing a mechanism for supporting and cooperating with provincial and territorial authorities that is respectful of federal roles and responsibilities and for meeting Canadians' expectations.

One of Health Canada's strategic outcomes is to support a health system that is responsive to the needs of Canadians. CIHI is a mechanism that helps P/Ts remain responsive, ensuring that data and information are available to monitor and measure health care system performance.

While the provinces and territories are primarily responsible for the delivery of health care, the federal government supports the provincial/territorial role by undertaking other health-related functions, such as the collection, analysis, interpretation, publication and distribution of information (Government of Canada, 1985; Health Canada, 2006, 2011b). These functions relate directly to CIHI's mandate, which is to lead the development and maintenance of comprehensive, accurate and integrated health information that enables sound health policy and effective health system management.

The role of CIHI includes the response to emerging health information priority needs. Health Canada has therefore looked to CIHI to provide data for detailed analyses of federal health policy interest which have included: cost analyses of non-drug technologies, Canadian physician distribution and compensation trends; patterns of use among drug classes; changes in health spending and revenue; as well as the growing role of the private health care sector.

CIHI has a role in leading the development of a number of data classifications and coding standards to collect, analyze and enhance the consistency and accuracy of information on health care in Canada. It represents Canada to international agencies working to enhance classification and terminology standards. CIHI is also responsible in the development and dissemination of pan-Canadian, comprehensive and comparable data, and to deliver on international data requirements with the World Health Organization (WHO) and the Organization for Economic Co-operation and Development (OECD).

CIHI's position as an arms-length organization is respectful of jurisdictional roles and responsibilities and meets the expectations of Canadians. It is a suitable mechanism to support the data needs of all of its provincial, territorial and federal stakeholders, without compromising quality of data and information.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

CIHI has increased awareness, understanding and uptake of the standards and products related to data infrastructure that it has developed. CIHI significantly contributes to building capacity and to providing technical expertise on data standards. Extensive research and data management capacity currently exists in larger provinces. Nonetheless all provinces and territories recognize CIHI as a leader in the field of health information and as the only organisation that provides pan-Canadian health data. The organisation's work to provide technical support, and to adapt international tools to the Canadian context, is being used by provinces and may facilitate the adoption of electronic health records. Findings related to CIHI's ability to broadly engage stakeholders are mixed and suggest that engagement practices may depend on the relationship holder within CIHI.

A hallmark product that CIHI publishes is the National Health Expenditure Trends, with the 18th edition of this annual publication released this year. The report provides an overview of how much is spent on health care annually, in what areas money is spent and on whom, and where the money comes from. It features comparative expenditure data at the provincial/territorial and international levels, as well as Canadian health spending trends from 1975 to the present. In addition, CIHI regularly releases reports on physicians, conveying demographic information about the supply of physicians in Canada and information about the payments made to them that are administered through provincial and territorial medical care plans. Data included in these publications calculates the average cost per clinical service for family medicine physicians and other specialists at the national and provincial levels. To complement these publications, CIHI also produces reports on trends among regulated nurses in Canada. These publications outline nursing practices across a variety of demographic, education, mobility and employment characteristics. This series highlights data from the three groups of regulated nursing professionals in Canada: registered nurses (RNs, including nurse practitioners or NPs), licensed practical nurses (LPNs) and registered psychiatric nurses (RPNs).

Increased Awareness and Understanding of CIHI Standards and Products

CIHI's maintains relationships with provincial and territorial health departments and health authorities, develops data standards, and releases analysis and other data products to help increase awareness of the products and expertise of the organisation.

An initial step in the awareness, understanding and uptake of standards and products is the engagement of stakeholders to identify their needs and priorities. In CIHI, the relationship established between regional staff and their counterparts within provincial and territorial administrations is seen as a key mechanism to understand provincial and territorial needs. However, key informant interviews and the 2014 KPMG Performance Audit note that current stakeholder engagement practices tend to depend on the role of the relationship holder within CIHI. This may result in inconsistencies in the nature and level to which stakeholders are engaged as well as differences in the consolidation of information gathered through stakeholder engagement activities.

According to KPMG, "The overall effectiveness of CIHI stakeholder engagement processes, including partnership development, could be strengthened through more clearly defined areas of strategic focus and greater central coordination of priority engagement activities". CIHI could take greater advantage of its resources, expertise and learning to identify areas of strategic focus based on specific criteria, which would consider the anticipated magnitude of the influence on the health system, the competing interests of the various stakeholders, as well as needs of the population, to identify priorities and to articulate them in a broad policy strategy.

The current CIHI Stakeholder Engagement Framework (Appendix 6), developed in response to the 2010 KPMG Performance Audit, establishes three 'tiers' of stakeholders which are identified in generic terms. The framework does not identify how each tier is engaged nor does it provide the objectives of this engagement nor an articulation of the mechanism to engage each category of stakeholders.

The document review suggests that consultation is not to be confused with broader, authentic and continuous engagement of health care providers, patients and family representatives. This level of engagement is most likely to raise stakeholders' awareness about issues dealt with by others and contribute to move to a culture beyond one of measurement toward one that values data sharing, data use and continuous performance improvements as referenced in Better Information for Improved Health: A Vision for Health System Use of Data in Canada, a document resulting from a number of consulting sessions that was prepared by CIHI in collaboration with Canada Health Infoway.

Uptake of CIHI Standards, Data and Analysis

Interviews with provincial/territorial and federal representatives confirmed the uptake of CIHI foundational pieces which describe procedures to be used for maintaining reliable datasets that are secure and conform with privacy legislation. CIHI data quality frameworks, health system performance measurement frameworks, and guidelines on privacy and data security, are used by a number of provinces. Interviews confirmed that the identification of standards for data collection is generally appreciated as it ensures that data is standardized and facilitates benchmarking. One province which developed a database to collect data on emergency departments' wait times and wait times for surgery resulted in voluntary adjustments to its system based on CIHI standards to reduce the amount of data processing necessary and permit faster turnaround times.

Development of Data Holdings and the Provision of Technical Support Contributing to a Change in Culture

In most provinces and territories, the processes for collecting health data are complex and cumbersome. Data is collected by health care providers or health facilities and often funnelled through health authorities or health departments before being transmitted to CIHI. Overall, comprehensive credible data is now available for the acute care sector in Canada. While CIHI has developed and continues to maintain databases/data holdings for hospitals and health facilities, there remains a need to further advance ambulatory care data. CIHI has made major inroads in recent years to building databases for the community care sector by supporting the implementation of clinical assessment tools known as interRAI assessments.

InterRAI is an international network of researchers promoting evidence-informed clinical practice and policy decisions through the collection and interpretation of high-quality data about the characteristics and outcomes of various populations. In collaboration with the interRAI network, CIHI has adapted interRAI instruments (i.e. the home and long-term care assessment instruments and the emergency department and intake from community and hospital assessment form, and the mental health assessment instruments) to the Canadian context.

Clinicians in inpatient and community mental health settings can use the new mental health tools to help them identify a wide range of mental health issues and to target modifiable outcomes for patients likely to experience improvements in a particular area, as well as for those patients at elevated risk of further decline if effective action is not taken. For example, the tool was piloted in Newfoundland where it was reported that it assisted staff to identify factors influencing patients' progress, adapt their treatment plans and track progress using standard and tested definitions.

Interviewees noted that CIHI technical support contributed to the development of their capacity to collect data in a reliable manner suggesting that the ongoing interactions between CIHI and the provinces/territories have facilitated a gradual cultural shift within health organisations where the importance of measurement and standardized data sets seems to have become a more generally accepted practice. As noted in document review and interviews, CIHI not only trained and provided ongoing technical support to Mental Health and Addiction Services staff in Newfoundland, but also facilitated interactions with the interRAI Canada research group (based at the University of Waterloo) while the interRAI database instrument was piloted in Central Newfoundland. Similar examples of capacity building and technical support were observed in Nova Scotia and in other provinces interested in the implementation of the Case-Mix Grouping methodology⁹.

CIHI's adaptation of interRAI contributes to the organization's first objective which is to help provide the information necessary for health care providers and health system managers to measure and report on performance.

CIHI is also developing technical specifications for interRAI's standardized outputs from interRAI Contact Assessment (CA). Should the uptake of interRAI tools continue, these specifications would assist software developers to integrate outputs into their interRAI CA data collection software. Given the complexity of the environment in which CIHI and Infoway operate, their joint efforts to promote the use of interRAI software outputs for electronic health records may facilitate implementation of electronic health records, which would in turn facilitate the collection of health information beyond the hospital setting.

4.4.2 To what extent have the intermediate outcomes been achieved?

Information produced by CIHI has been used on numerous occasions for the conduct of secondary analysis by provincial/territorial and federal levels of government and by other stakeholders (e.g., non-government organisations, academics, researchers). More recently, evidence suggests that CIHI information has been used to further discussions on changes in administrative and clinical practices at the provincial level.

Increased Use of CIHI Data for the Conduct of Secondary Analysis

To assess the use of CIHI data by provinces/territories, the evaluators reviewed documents released by various levels of government and data requests made to CIHI. Tracking the use of CIHI products proved challenging as a number of mechanisms, including direct requests to CIHI, eReports, QuickStats and analytical reports, are used to access data by data holding. For this reason the number of requests for CIHI data provides only a general sense of frequency.

Information provided by CIHI on the number of requests suggests that six databases¹⁰ out of 28 had more than 100 requests for each of them during the period comprised between April 1, 2012 to March 31, 2014) while 15 had less than 25 requests. Although this may suggest great variability in terms of database activity, fewer requests to a given database should not be interpreted as limited use as databases may be linked for the purpose of custom analysis or the development of meta-databases.

Interviews suggest CIHI data are primarily used for the conduct of internal benchmarking or other secondary analysis which are not released publicly. Benchmarking allowed for the identification of potential issues which are investigated in more depth by provinces or territories using their own data pertaining to administrative processes and/or clinical procedures. For example:

A regional health authority in Newfoundland compared the incidence of falls among patients using antipsychotic drugs while in long term care facilities to those in other provinces. The health authority worked with another jurisdiction to identify the cause of the increased number of falls, to learn from its experience and to consider changes to practices.
The Nova Scotia Department of Health and Wellness developed a plan to identify the number, mix and distribution of physicians needed by the province's population over the next decade. This model for physician recruitment and retention made extensive use of the CIHI physician dataset.

The Health Canada Applied Research and Analysis Directorate (ARAD) coordinates data requests to CIHI from the department's Strategic Policy Branch (SPB) and tracks data requests from other Health Canada branches and PHAC. Seven requests were made by Health Products and Foods Branch and one by Healthy Environments and Consumer Safety Branch during the period 2011-2012 to 2013-2014. ARAD itself used CIHI data for secondary analysis conducted in response to requests made by senior management within both Health Canada and PHAC or as information in departmental reports including:

2011-2012

Non-Insured Health Benefit (NIHB) micro-simulation model assessing the costs and benefits of various policy proposals related to the NIHB program
Physician compensation in Canada
Health spending efficiency across Canadian provinces
The impact of rising public health spending on other public program spending, and private spending in Canada

2012-2013

Disparities in health spending and revenue capacity among Canadian provinces
The impact of interprovincial migration on P/Ts' healthcare spending
Estimating public subsidies in private sector health spending

2013-2014

Non-Drug Health Technology Project: Analysis of non-drug health technology in Canada hospitals
Estimation and Forecasting Canadian Health Expenditures
Provincial Prescribed Drug Spending in Canada compared with Federal Direct Prescribed Drugs Expenditure Project
As an Accord commitment for the Federal Government to improve accountability and reporting to Canadians, Health Canada (SPB, ARAD) produced a series of reports entitled Healthy Canadians
National Survey of Selected Medical Imaging Equipment.

The Parliamentary Budget Office also regularly uses CIHI data as part of its Fiscal Sustainability Report to develop scenarios and projections related to the Canada Health Transfer and the Canada Social Transfer.

Despite variations in how health care is delivered across the country, there are important similarities in the challenges addressed by provincial/territorial health systems. A report released by the Commission on the Reform of Ontario's Public Services notes that "It is no longer sufficient to simply ask whether a practice or a pharmaceutical offers the prospect of improved health. A much more stringent test is to determine if it is an efficient way to achieve positive health outcomes¹¹.

Views among interviewees varied as to whether CIHI should play a greater role in identifying clinical and/or administrative changes to realize further efficiencies. Concern was expressed that CIHI may lack the expertise to provide this level of advice on clinical issues. Some interviewees suggested that identification of best practices maybe more the role of provincial health information and research organisations and, although CIHI's reports address best practices using data provided by provinces and territories, it would be difficult for the presentation of results to reflect political relevance and sensitivity.

The Commission's report further identifies the need to "explore the potential for a national Organization for Economic Co-operation and Development-type entity that collates and enhances evidence-based policy directions and provides enhanced collaboration on issues across jurisdictions". Interviews suggest that CIHI could be well positioned to play such a role by facilitating the uptake of best practices identified as a result of the work conducted by stakeholders such as provincial level health information organisations.

Use of CIHI Products in Decision-Making

As observed in section 4.1, the document review noted that health data is expected to be used for the management of health human resources, to monitor the quality of health care services and to track health care costs. While all levels of the health system acknowledge the value of a strong health information system¹², having more data available is only an initial step as this information needs to be used as part of decision-making processes to result in change.

Document review and interviews identified numerous examples of how CIHI products are being used by provinces in their review of current health policies and clinical practices. While assessing the impact of CIHI's work on system-level change is a challenging undertaking, evidence suggests that CIHI's contributions to the overall body of knowledge supporting decision making in the health care system is emerging.

For examples:

The Provincial End-of-Life Care Action Plan for British Columbia (Ministry of Health, 2013) referred to a study conducted by CIHI which examined the health and functional status of persons aged 65 and over with dementia. The study evaluated the factors that most contributed to the need for admission to residential care and the utilization of health care resources. The action plan suggests that CIHI's study findings informed community health services priorities by suggesting that the availability of able caregivers and their capacity to cope with the person's dementia needs may have even more influence than the status of cognitive impairment.
The CIHI Western Office, in collaboration with the Canadian Partnership Against Cancer, released a pan-Canadian report (2008) which presented data on the provincial rates of mastectomy versus lumpectomy and radiation, using CIHI's Hospital Morbidity Database, National Ambulatory Care Reporting System as well as the Alberta Ambulatory Care Reporting System. The data illustrated that rates of mastectomy varied widely among jurisdictions. The report offered some suggestions for the differences such as age, geography and/or an attempt to avoid the side effects of radiation. CIHI working collaboratively with the British Columbia cancer agency and breast cancer surgeons presented at surgical rounds to consider the variation of practice in mastectomies across the province.
CIHI briefed the Saskatchewan Ministry of Health before the pan-Canadian report on provincial rates of mastectomy was released, as well as physician groups that advise the Ministry on clinical appropriateness. The Saskatchewan Surgical Initiative established a mastectomy working group to determine reasons for their province's increased mastectomy rate. The University of Saskatchewan has undertaken a study to examine why cancer patients choose certain approaches. The document review and interviews suggest that information generated by CIHI contributes to the overall body of knowledge that supports decision making in the health care system.

4.4.3 To what extent has the longer term outcome been achieved?

CIHI has led the development of data infrastructure and paved the way for the introduction of activity-based funding and of other applications providing evidence supporting system improvements within and outside the hospital settings.

Hospital funding reforms introduced in provincial jurisdictions are based on principles and methods of "activity-based funding". This approach pays hospitals based on the services they provide for patients - that is, 'the money follows the patient'.¹³ In the course of the evaluation, a case study was conducted to provide an illustration of the extent to which the work of CIHI has been instrumental in the implementation of activity-based funding. While activity-based funding has grown in popularity as a new funding approach, it is not without controversy. The assessment of the benefits and constraints associated with this funding method were beyond the scope of the evaluation.

Currently, most hospitals in Canada are funded through a block funding method (also known as global budgeting) where hospitals receive a lump sum budget at the beginning of a year. Funding is largely based on historical payments, a method that ensures predictable budgets for hospitals and in part controls hospital spending growth. The document review illustrates that the use of block funding does not provide a strong incentive for the collection of operational data and therefore, data is unavailable in many facilities.

Activity-based funding is presented as a funding approach which provides solutions related to the issues of access to health services and the need to control the increase of health care costs. Since service volumes and pre-set payment rates drive hospital revenues, activity-based funding is viewed as having a built-in financial incentive for hospitals to increase activity levels and improve efficiencies in operations, such as shortening the length of patient stays and reducing unit costs of services which - at the system level - should translate into improved wait times and increased transparency in hospital funding¹⁴.

Although the principles of activity-based funding are straightforward, its implementation is complex and requires detailed data on all aspects of hospital operations to identify and classify the direct and indirect costs associated with each type of patient service. Initially developed in the U.S., CIHI has adapted the methodology to the Canadian context using its databases. In the CIHI-maintained Discharge Abstract Database (DAD) hospitals record administrative, clinical and demographic information on hospital discharges (including deaths, sign-outs and transfers). It was used to assign each patient case in the database to a unique group, i.e., a Case Mix Group (CMG+), which is clinically meaningful and homogeneous in resource usage. The CMG methodology identified nearly 600 of these groups¹⁵.

The document review further indicates that grouping design choices included various other applications such as Resource Intensive Weights, a resource-use measure developed by CIHI for inpatients in acute-care hospitals, and interRAI assessments.

At this point, British Columbia and Alberta have started introducing activity-based funding, and Ontario has made use of case mix-based funding (a broader category, which includes activity-based funding) for a number of years. Additional details on jurisdictional implementation of case mix-based funding can be found in Appendix 5.

Since the introduction of CMGs for inpatients in acute-care hospitals, CIHI has refined the CMG+ system to improve the comparability of national data and to provide a utilization management tool which is beginning to have applications outside hospital settings. By ensuring that grouping systems are available in other databases, such as Continuing Care Reporting System, Home Care Reporting System, National Rehabilitation Reporting System and National Ambulatory Care Reporting System, CIHI has led the development of data infrastructures paving the way for the healthcare funders and system managers to introduce activity-based funding and/or to initiate system improvements which could extend beyond current hospital settings.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

The HII is delivered in an economical and efficient manner. Processes ensure the timely release of data. The transfer payment to CIHI is an appropriate funding mechanism to support CIHI activities.

The assessment of economy and efficiency focuses on issues related to the timely release of data and the appropriateness of the funding delivery mechanism. Observations on economy and efficiency are based on findings from the literature and document review, key informant interviews and available financial data.

Based on an agreement between the Office of Evaluation and CIHI during the evaluation planning stage, other elements of economy and efficiency were reviewed in the KPMG 2014 Audit of Performance. The key findings of KPMG Audit of Performance note the need for CIHI to further clarify and formalize the process and guidelines supporting investment decisions and to explore cost-effective means for obtaining information on product costing. More detailed observations and recommendations of the KPMG audit, which assess the economy and efficiency of CIHI management practices and the effectiveness of CIHI's management structure, can be found on the CIHI web-site.

As CIHI does not provide output/outcome-specific costing data (it is not required in the contribution agreement), resource utilization in relation to outputs and outcomes was not assessed as part of this evaluation.

4.5.1 Are processes in place ensuring the timely release of data?

CIHI's strategic goals for 2012-2017 include: improved comprehensiveness, quality and availability of data; and the support of population health and health system decision making. To measure progress in achieving these strategic goals, CIHI has created a framework which includes two outcomes addressing timeliness, one of which being 'improved timeliness in the data used for health system management'.

While only 69% of respondents to CIHI's 2014 stakeholder survey agreed that CIHI's data, analyses and services were timely, 90% of CIHI data was less than a year old in 2014 (CIHI Corporate Performance Report (2013-2014)). This represents significant progress since 2001 when none of CIHI data was less than a year old.

According to CIHI, timeliness is not the only dimension of quality which must be respected. Some provincial interviewees considered that the time lag between the submission of provincial data, and the release of CIHI reports, was too long. They suggested that while all dimensions of quality are important, data loses its relevance and usability if it is not timely. However, it could also be argued that data usability is impacted by accuracy and comparability for stakeholders intending to use data for benchmarking purposes. Most federal and provincial interviewees acknowledged that expectations may be based on a society that expects information 'in real time', which may not be appropriate given the mandate of CIHI and the environment in which it operates. We also examined other pan-Canadian information gathering (e.g., Canada Census reports) and determined that the release of CIHI data was within a reasonable period of time.

4.5.2 Are there alternative delivery approaches that would be more efficient or economical?

In terms of assessing the appropriateness of the funding mechanism, this evaluation included the review of the results for the Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations (March 2014). These organizations are also funded through transfer payments. The report observed that a transfer payment was an appropriate way to advance federal health system priorities. 'Having an independent organization facilitate collaboration among many stakeholders, including provinces and territories, while respecting their jurisdiction in healthcare, was seen as a positive approach to create national system-level changes. It was also seen as more efficient having one national organization focusing on a particular issue than creating a program with multiple, potentially disjointed, time-limited projects, which would not have been able to leverage the resources required to achieve national-level changes.'

The 2012 Terms and Conditions for the HII allowed for increased accountability and reporting requirements. More frequent cash flow requests, progress reports and the inclusion of all future evaluations in the Health Canada Five-Year Evaluation Plan support the monitoring of progress on a more regular basis.

The 2014 Synthesis Report also examined whether funding to pan-Canadian organizations should be managed individually or together as a program. While the report observed that 'there were advantages to examining the suite of pan-Canadian organizations to share lessons learned and to identify potential integration and/or coordination of activities amongst the organizations', it concluded that 'the pan-Canadian organizations still require a degree of individual management given the differences in their policy objectives, business and governance models, and their size, scope and maturity'.

In terms of CIHI, a number of federal and provincial interviewees suggested that, given their joint responsibilities as national co-leads on the Health System Use (HSU) of electronic health data and the increasingly complex environment in which CIHI and Infoway operate, there is a need for the two organizations to better integrate their activities as they work to ensure that Electronic Health Records consider all opportunities and risks for the use of data held by these systems. While CIHI has also identified this need and plans to address it, Health Canada could be instrumental in the support of CIHI in these efforts.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

While the field of health information improved significantly in the last twenty years, the need for comparable health information to manage the health system, improve its performance, and ensure its sustainability, remains in a country where health expenditures consume on average 40% of provincial and territorial budgets.

The Department of Health Act delegates to the Minister of Health the duty to cooperate with provincial and territorial authorities with a view to coordinate efforts nationally. The information necessary to improve health system performance needs to go beyond the hospital setting and to allow for the evaluation of outcomes of care, the planning of health services and for the appropriate allocation of resources. The provincial capacity in health information has increased as the majority of provinces have now established health information organizations responsible for monitoring and evaluating health services. While some activities are similar to those of CIHI, only CIHI has the mandate to collect pan-Canadian data, which appear more important to jurisdictions with less data management capability who continue to rely more heavily on CIHI support.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes

Achievement of Expected Outcomes (Effectiveness)

5.2.2 Demonstration of Economy and Efficiency

The absence of output/outcome-specific financial data limited the ability to assess resource utilization as part of this evaluation. In terms of efficiency, the number of datasets releasing timely data increased significantly since 2001 although some provincial representatives still consider the time lag between data submission and report release to be too long.

The Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations observed that a transfer payment is an appropriate way to support the provinces and territories in the delivery of health care, although there could be increased efficiencies if CIHI and Infoway better integrated their activities as they work to ensure that Electronic Health Records consider all opportunities and risks for the use of data held by these systems.

6.0 Implications for the Future

Formal recommendations are not being proposed to Health Canada. Based on the findings and conclusions outlined in this evaluation report, the department is aware that changing health information circumstances may, have future implications for the Canadian Institute for Health Information. Respecting the complex environment in which CIHI operates, CIHI may consider developing a vision which would guide a move beyond data management and bilateral interactions with stakeholders.

After 20 years of operations, a renewed vision should contribute to guide the organisation in becoming an agent of change by helping with:

Modernisation of CIHI mandate

Since its creation, CIHI's mandate has been focused on developing a pan-Canadian data infrastructure. Given the considerable progress made in this area, it may be time for the organisation to reflect on what CIHI could do to take the next step and facilitate health stakeholders' adoption of evidence-based processes and practices improving the efficiency and effectiveness of the health system.
Refining the stakeholder engagement strategies to deliver on this mandate

CIHI stakeholder engagement strategies may benefit from being better balanced between consulting to identify priorities and the need to facilitate adoption of processes and practices improving the health system. The stakeholder engagement strategies should be focussed on the stakeholders possessing the greatest potential to initiate the large-scale changes necessary to improve health system efficiency and effectiveness.

Appendix 1 - Logic Model

Appendix 2 - Coverage of CIHI data holdings

Table 1: Coverage of CIHI data holdings
Healthcare Areas		No. of Data Holdings	Comprehensiveness of CIHI Data Holdings
Acute and Abulatory Care	Inpatient	2	Data collection is complete across all 13 jurisdictions.
	Day surgery	3	Data collection is complete across all 13 jurisdictions.
	Emergency Department	1	Data collection is complete in three jurisdictions, and in progress in 7 jurisdictions.
	Ambulatory Clinics	1	Data collection is not implemented in most jurisdictions. Data is complete in one jurisdiction, with two making progress.
Community Care		2	Data collection in continuing care is trailing behind that in acute and ambulatory care. Data is complete in a few jurisdictions (six jurisdictions for Continuing Care Reporting System; three for Home Care Reporting System), with most other jurisdictions making progress.
Specialized Care		8	Hospital Mental Health Database is complete in all 13 jurisdictions. National Trauma Registry Databases are complete where applicable. Data holdings in other specialized care areas, such as Canadian Joint Replacement Registry and National Rehabilitation Reporting System, are mostly not implemented.
Pharmaceuticals		1	Data collection for National Prescription Drug Utilization Information System Database is either complete or in progress in all applicable jurisdictions.
Patient Safety		1	National System for Incident Reporting is in development.
Workforce		9	Data collection is complete in most jurisdictions, with a few making progress.
Health Spending		3	National Health Expenditure Database and Canadian MIS Database are complete in all jurisdictions. Canadian Patient Cost Database is incomplete in most jurisdictions - it is complete in three jurisdictions, with one making progress.
Source: CIHI, Corporate Performance Report 2013-14; President's Quarter Report to the Board, 2011.

Appendix 3 - Summary of Findings

Rating of Findings: Ratings have been provided to indicate the degree to which each evaluation question and issue have been addressed.
Relevance Rating Symbols and Significance: A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Questions	Indicators	Overall Rating	Summary
Continued Need for the Program
Is there a need for a Canadian health information system, i.e., HII?	Description of Canadian health information systems Existence and role of other organizations involved in the same field Perception (e.g., CIHI users and Canadians) regarding the need for health care information and analytical products	High	The field of health information has significantly improved in the last twenty years, however the need to make better use of data to improve performance and to increase the sustainability of the health system remains. Health expenditures continue to consume on average 40% of provincial and territorial budgets and the containment of health costs is one of the key priorities for provincial and territorial governments. The information necessary to improve health system performance needs to go beyond the hospital setting and to allow for the evaluation of outcomes of care, the planning of health services and for the appropriate allocation of resources. CIHI has established an engagement process to ensure the organization is responsive to stakeholder needs. While stakeholders generally consider that CIHI products are relevant to their needs, satisfaction levels have declined since 2012. The provincial capacity to process and publish province specific health information increased as the majority of provinces have established health information organizations responsible for monitoring and evaluating health service quality. While some activities are similar to those of CIHI, only CIHI has the mandate to collect pan-Canadian data. The availability of pan-Canadian data is not considered essential by all jurisdictions, however those with less data management capability continue to rely heavily on CIHI support.
Alignment with Government Priorities
Does HII align with the priorities of the Government of Canada?	Initiative objectives align with federal priorities	High	The latest reference to the health system by the federal government was in the June 2011 Speech from the Throne where the government expressed its interest in a sustainable and accountable health system. At the end of 2011, the Government announced set increases for health care transfers which are associated with the growth of the economy. The data infrastructure developed by CIHI, the organisation's technical expertise as well as the health system data collected with the support of federal funding appears to be necessary to inform decisions fostering a sustainable and accountable health system.
Alignment with Federal Roles and Responsibilities
Is HII an appropriate federal responsibility?	Initiative objectives align with federal roles and responsibilities	High	In Canada, the federal government is responsible for administering the Canada Health Act and provides transfer payments to provinces and territories who are primarily responsible for the delivery of health care. Canadians expect Health Canada to play a leading role in the health system and the Department of Health Act delegates to the Minister of Health the duty to cooperate with provincial and territorial authorities with a view to coordinate efforts nationally. The differing contexts of national health information organisations from abroad make comparisons challenging, however it should be noted that the Australian Institute of Health and Welfare was founded by an act of Parliament establishing the core functions and accountability of the organisation.
Legend - Relevance Rating Symbols and Significance: High There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear. Partial There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear. Low There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Performance Rating Symbols and Significance:: A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Questions	Indicators	Overall Rating	Summary
Achievement of Expected Outcomes (Effectiveness)
To what extent are data and analysis generated through the HII used by health stakeholders involved in planning and management of health care services delivery?	Description of users capacity/challenges in accessing and using CIHI-generated data and analytical products Evidence of use in the public domain	Progress Made; Further Work Warranted	While CIHI is best known for the collection and publishing of health care data, stakeholders awareness of the full suite of product and services is not as high as it could be expected considering CIHI is an established and well-known organisation. While views vary as to whether CIHI should play a greater role in identifying clinical and/or administrative changes to realize greater efficiencies, CIHI may be well positioned to facilitate the uptake of best practices identified as a result of data analysis by provincial health information organisations. Numerous uses of information produced by CIHI have been reported, particularly for the conduct of secondary analysis. While identifying examples where CIHI information contributed to concrete and broad policy or practice improvements proved to be challenging, the work of CIHI to adapt international tools to the Canadian context may facilitate the adoption of electronic health records, contribute to the expansion of activity-based funding and foster the culture of performance that is currently emerging.
To what extent are data and analysis contributing to improvements in the Canadian health system?	Evidence of health care system improvements associated with CIHI data or analysis	Progress Made; Further Work Warranted
Demonstration of Economy and Efficiency
Are HII activities conducted in the most economical and efficient manner?	Are processes in place ensuring the timely release of data?	Achieved	Output/outcome-specific costing data is not required in the contribution agreement and therefore, resource utilization in relation to outputs and outcomes could not be assessed as part of this evaluation. In terms of efficiency, the number of datasets releasing timely data increased significantly since 2001. Some provincial representatives still consider the time lag between data submission and report release to be too long. The Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations observed that a transfer payment is an appropriate way to advance federal health system priorities although there could be increased efficiencies if CIHI and Infoway better integrated their activities as they work to ensure that Electronic Health Records consider all opportunities and risks for the use of data held by these systems.
Legend - Performance Rating Symbols and Significance: Achieved The intended outcomes or goals have been achieved or met. Progress Made; Further Work Warranted Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed. Little Progress; Priority for Attention Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.

Table 3: Summary of Relevance Ratings
Evaluation Issue	High	Partial	Low
Issue 1: Continued need for the program
Is there a need for a Canadian health information system, i.e., HII?	High	NA	NA
Issue 2: Aligned to federal government priorities
Does HII align with the priorities of the Government of Canada?	High	NA	NA
Issue 3: Program consistent with federal roles and responsibilities
Is HII an appropriate federal responsibility?	High	NA	NA
Legend - Performance Rating Symbols: High There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear. Partial There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear. Low There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Table 4: Summary of Performance Ratings
Evaluation Issue	Achieved	Progress Made; Further Work Warranted	Little Progress; Priority for Attention
Issue 4: Achievement of intended outcomes (effectiveness)
To what extent are data and analysis generated through the HII used by health stakeholders involved in planning and management of health care services delivery?	NA	Progress Made; Further Work Warranted	NA
To what extent are data and analysis contributing to improvements in the Canadian health system?	NA	Progress Made; Further Work Warranted	NA
Issue 5: Demonstrated economy and efficiency
Are HII activities conducted in the most economical and efficient manner?	Achieved	NA	NA
Legend - Performance Rating Symbols and Significance: Achieved The intended outcomes or goals have been achieved or met. Progress Made; Further Work Warranted Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed. Little Progress; Priority for Attention Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.

Appendix 4 - Evaluation Description

Evaluation Scope: The scope of the evaluation included ... (activities, organizations, timeframe, geography, etc.)
Evaluation Issues: The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Table 1: Core Evaluation Issues and Questions
Core Issues	Evaluation Questions
Relevance
Issue #1: Continued Need for Program	Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians Question
Issue #2: Alignment with Government Priorities	Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes Question
Issue #3: Alignment with Federal Roles and Responsibilities	Assessment of the role and responsibilities for the federal government in delivering the program Question
Performance (effectiveness, economy and efficiency)
Issue #4: Achievement of Expected Outcomes (Effectiveness)	Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes Question
Issue #5: Demonstration of Economy and Efficiency	Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes Question

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources.

Sources of information used in this evaluation included literature review, document review, surveys, interviews, case studies, etc. (briefly list data collection methods, sampling approach and size, etc.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included (briefly describe analysis methods used, e.g.,):

Systematic compilation, review and summarization of data to illustrate key findings.
Statistical analysis of quantitative data from databases.
Thematic analysis of qualitative data.
Trend analysis of comparable data over time.
Comparative analysis of data from disparate sources to validate summary findings.

Additional Detail on the Challenges and Mitigation Strategies

Use this section of the Appendix only if needed.

Appendix 5 - Jurisdictional adoption of Case Mix-based funding

British Columbia

In 2011/12, a portion of the funding (approximately 18%) of each regional health authority was converted from block funding to activity-based funding, representing a total of $23 million out of the $6 billion for all B.C. health authorities and hospitals.¹⁶

It was the first introduction of activity-based funding in the Canadian healthcare system, which since has been closely watched by other jurisdictions considering adoption of this approach.¹⁷ As of today, there are mixed reports on the implementation results. According to the B.C. Health Service Purchasing Organization, the shift to activity-based funding has achieved positive results. In its annual report for 2011/12, the Health Service Purchasing Organization indicated that its programs cut waiting lists for many operations by more than 10% and significantly increased the volume of emergency patients treated within a set time¹⁸. On the other hand, a study funded by the Canadian Institute of Health Research (CIHR) to examine the payment reform in B.C. hospitals did not find the definitive association between activity-based funding and a change (or increase) in volume of surgeries. With an examination of Resource Intensive Weights for day surgeries over time, the study indicated that there was a long-term increasing trend in day surgeries among B.C. hospitals that began before the introduction of activity-based funding, which could not be attributable to the impact of activity-based funding.¹⁹ The B.C. government remains committed to the continuation of the new funding policy.²⁰

Alberta

The creation of Alberta Health Services in 2008 set out to streamline multiple funding systems. In 2010, Alberta began the reform with its long-term care and introduced the activity-based funding approach. Some of the policy objectives Alberta sought through activity-based funding were: to manage prices, volumes and locations of services for equitable access and long-term sustainability; to meet standards and quality; to increase access through incentives to efficient providers; to meet regulatory requirements such as staffing ratios for long-term care; and to provide equitable funding across long-term care providers.²¹ Results associated with the implementation of activity-based funding remain to be documented.

Ontario

In 2012, Ontario announced a health system funding reform that moved away from block funding to a new model called the Health Based Allocation Model (HBAM). HBAM makes use of a more general case mix-based funding model, with an Ontario adaptation. Some of the policy objectives Ontario intended to achieve through the new model are: to allocate funding based on evidence; to recognize the costs of providing care; to ensure funding stability; and to encourage quality improvement in healthcare. Ontario highlights that the new model ensures funding is tied directly to the healthcare provided, encourages smarter use of resources, and so helps the province build a sustainable healthcare system. The new model is to be implemented for hospitals, Community Care Access Centres and long-term care facilities. In 2015-2016, Ontario plans to cover 70% of its funding for hospitals with the new model and the remaining 30% with global funding.²²

Other jurisdictions

Across Canada, other jurisdictions, i.e., Manitoba, Nova Scotia and Quebec, are contemplating to implement this funding model as well.

Appendix 6 - Stakeholder Engagement Framework

This framework has been developed in response to a 2010 KPMG performance audit that recommended an overarching framework to coordinate and integrate CIHI's stakeholder engagement activities as a whole. It is based on CIHI stakeholder tier groups 1, 2 & 3.

Stakeholder Engagement Framework
Stakeholder Tiers and Segments	Segment Description	Usual CIHI Interaction at Product Level	Usual CIHI Interaction at Regional Level	Usual CIHI Interaction at National Level
Tier 1 Segment Funders Policy makers Health system managers/program delivery managers	Tier 1 Segment Description Federal/Provincial/Territorial governments Government executives/Policy analysts Health care provider executives/Program managers/Physicians/Clinicians	Program Areas*	Regional Offices*	Senior Management*
Tier 2 Segment Data providers Data analysts Policy-related researchers Influencers	Tier 2 Segment Description Health information managers/coders/Clinicians Professional accrediting organizations/Vendors Data analysts/Decision support by type Policy-related researchers by type Media/NGOs/Public interest groups/Professional associations
Tier 3 Segment Other researchers	Tier 3 Segment Description Academic researchers by type/Clinical researchers

List of Acronyms

ADM: Assistant Deputy Minister
ARAD: Applied Research and Analysis Directorate
CA: InterRAI Contact Assessment
CIHI: Canadian Institute for Health Information
CMG: Case-Mix Groups
CPHO: Chief Public Health Officer
DAD: Discharge Abstract Database
DG: Director General
DM: Deputy Minister
DPR: Departmental Performance Report
EHR: Electronic Health Records
F/P/T: Federal/Provincial/Territorial
HII: Health Information Initiative
HIPC: Health Information Policy Committee
HQ: Headquarters
HSU: Health System Use
Infoway: Canadian Health Infoway
NGO: Non-Governmental Organizations
NIHB: Non-Insured Health Benefit Program
PAA: Program Alignment Architecture
PHAC: Public Health Agency of Canada
P/Ts: Provinces and Territories
RMAF: Results-based Management and Accountability Framework
RPP: Report on Plans and Priorities
SAAC: Strategic Analytic Advisory Committee
SC: Statistics Canada
SPB: Strategic Policy Branch
TBS: Treasury Board of Canada Secretariat

Page details

Date modified:: 2020-08-31

Evaluation of the Health Information Initiative 2012-2013 to 2014-2015

Table of Contents

List of Tables

Executive Summary

Evaluation Purpose and Scope

Program Description

Conclusions - Relevance

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

Demonstration of Economy and Efficiency

Implications for the Future

1.0 Evaluation Purpose

2.0 Program Description

2.1 Program Context

2.2 Program Profile

2.2.1 CIHI Activities

2.2.2 CIHI Governance

2.2.3 Collaborating with Partners

2.3 Program Logic Model

Stakeholder Engagement

Capacity Building

Database Development and Management

Data Processing and Analysis

2.4 Program Alignment and Resources

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

3.2 Limitations and Mitigation Strategies

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

Responsiveness to Needs and Priorities

Provincial Capacity in the Field of Health Information

4.2 Relevance: Issue #2 - Alignment with Government Priorities

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Increased Awareness and Understanding of CIHI Standards and Products

Uptake of CIHI Standards, Data and Analysis

Development of Data Holdings and the Provision of Technical Support Contributing to a Change in Culture

4.4.2 To what extent have the intermediate outcomes been achieved?

Increased Use of CIHI Data for the Conduct of Secondary Analysis

Use of CIHI Products in Decision-Making

4.4.3 To what extent has the longer term outcome been achieved?

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

4.5.1 Are processes in place ensuring the timely release of data?

4.5.2 Are there alternative delivery approaches that would be more efficient or economical?

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes

Achievement of Expected Outcomes (Effectiveness)

5.2.2 Demonstration of Economy and Efficiency

6.0 Implications for the Future

Appendix 1 - Logic Model

Appendix 2 - Coverage of CIHI data holdings

Appendix 3 - Summary of Findings

Appendix 4 - Evaluation Description

Data Collection and Analysis Methods

Additional Detail on the Challenges and Mitigation Strategies

Appendix 5 - Jurisdictional adoption of Case Mix-based funding

British Columbia

Alberta

Ontario

Other jurisdictions

Appendix 6 - Stakeholder Engagement Framework

List of Acronyms

Page details