Evaluation of the Health Information Initiative 2007-2008 to 2011-2012

June 2013
Prepared by Evaluation Directorate, Health Canada and the Public Health Agency of Canada

List of Tables

Table 1 - Annual HII Funding (2007-2008 to 2011-2012)
Table 2 - Evaluation Questions by Issue
Table 3 - Limitations and Mitigation Strategies
Table 4 - Rating of Awareness and Understanding of Stakeholders by HII Product
Table 5 - Attendance at Educational Events
Table 6 - Usage of HII Products (Data and Information) and Services
Table 7 - Extent of Use of HII Products in Health System Decision-Making
Table 8 - HII Annual Expenses (2007-2008 to 2011-2012)
Table 9 - Rating of the Value of HII Products to its Stakeholders
Table 10 - CIHI Operating Expenses in Relation to its Strategic Directions

List of Acronyms

CCRS: Continuing Care Reporting System
CIHI: Canadian Institute for Health Information
CORR: Canadian Organ Replacement Register
DM: Deputy Minister
FAA: Financial Administration Act
HC: Health Canada
HCRS: Home Care Reporting System
HICC: Health Information Coordinating Council
HII: Health Information Initiative
HMRI: Hospital Medical Records Institute
HSMR: Hospital Standardized Mortality Ratio
IHI: Institute for Health Information
LMS: Learning Management System
MAP: Management Action Plan
NACRS: National Ambulatory Care Reporting System
MIS: Management Information Services
NGO: Non-Government Organizations
OECD: Organization for Economic Cooperation and Development
OMHRS: Ontario Mental Health Reporting System
ORN: Ontario Renal Network
PHAC: Public Health Agency of Canada
RPP: Reports on Plans and Priorities
SARS: Severe Acute Respiratory Syndrome
SC: Statistics Canada
TB: Treasury Board
UK: United Kingdom
US: United States
WHO: World Health Organization

Executive Summary

Purpose

The evaluation of the Health Information Initiative (HII) was conducted by the Evaluation Directorate of Health Canada (HC) and the Public Health Agency of Canada (PHAC) to provide the Deputy Minister of Health and senior program managers with credible and neutral information on the ongoing relevance and performance of the HII. The evaluation also fulfills the requirement to evaluate all grant and contribution programs every five years under the Treasury Board (TB) Policy on Evaluation (2009) and ensure compliance with the Financial Administration Act (FAA). The evaluation covers activities and performance from 2007 to 2012.

Background

The HII was created in response to recommendations to improve health information systems in Canada. The HII aimed to develop standards and methodologies, to create and maintain databases that hold reliable pan-Canadian comparable data, and to share health information for the development of health information products and services. The Canadian Institute for Health Information (CIHI) manages the HII. CIHI is a non-government organization, with an annual budget for HII of $81.7M, dedicated to the coordination of health information with the ultimate objective of improving health system performance and, indirectly, the health of Canadians.

Methodology

The evaluation assessed the ongoing relevance and performance (effectiveness and efficiency) of the HII. The evaluation built on existing findings from the 2010 CIHI Independent Evaluation, the 2010 Performance Audit, the 2012 CIHI stakeholder survey, and was supplemented with a document and literature review.

Conclusions

The evaluation confirmed, with respect to relevance, that the HII objectives, mandate and activities are aligned and consistent with Health Canada strategic objectives, Government of Canada priorities, and the federal role with respect to the development of standardized and comparable information at the pan-Canadian level.

With respect to performance, the HII achieved successes in achieving its immediate and intermediate outcomes. The HII increased awareness and understanding of its data and information among intended clients and achieved increased awareness of health care system performance and factors affecting good health both across Canada and within jurisdictions. This evaluation determined that the HII provided relevant, timely, responsive and integrated data and information to its clients in reporting of health care system performance. There was also strong evidence that CIHI was a trusted partner and leader in safeguarding health information. The long term outcomes of improved decision-making regarding health policy, effective management of the Canadian health system, and improved quality of health were substantiated with anecdotal evidence. Given the many players involved in improving the health of Canadians through increased awareness and improved quality of health, it was difficult to determine the HII's contribution to the overall improvements in health.

With respect to economy and efficiency, the evaluation concluded that the HII could not be achieved more economically through another organization. The evaluation also demonstrated that CIHI's processes and financial practises have provided evidence of effective monitoring and use of resources for the HII. The 2010 performance audit indicated that CIHI minimized resources while maximizing outcomes.

Recommendation

Given the recent audit and evaluation that were conducted on the Health Information Initiative, each with their own recommendations for improvements, this evaluation focused on key recommendations that would be feasible to address prior to the commencement of the next HII evaluation process in 2014. This evaluation recommends that:

Performances measures for the HII should be further aligned to a more outcome-based approach and, where feasible, use mechanisms to systematically document this evidence, particularly for the intermediate and long term outcomes.

HC will work in collaboration with CIHI to ensure the implementation of the recommendation.

Management Response and Action Plan

CIHI: Health Information Initiative -- Evaluation May 2013
Recommendations	Management Response	Deliverables	Accountability	Anticipated Completion Date
Performances measures for the HII should be further aligned to a more outcome based approach and where feasible, use mechanisms to systematically document this evidence, particularly for the intermediate and long term outcomes.	We support the recommendation and have informed CIHI of the recommendation as well as the need to track performance measures specific to the HII.	A performance measurement framework, specifically for the HII, that is realigned to a more outcome based approach and identifies systematic mechanisms to support their documentation.	Director General Applied Research & Analysis Directorate Strategic Policy Branch Health Canada	October 2013
		An Evaluation Framework that examines the HII in the context of the achievement of Health Canada objectives. This framework will identify what indicators and data that Health Canada will be required to collect in order to assess performance.	Director General , Evaluation Directorate in collaboration with the Director General, Applied Research & Analysis Directorate Strategic Policy Branch Health Canada	October 2013

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Health Information Initiative (HII) under the Canadian Institute for Health Information (CIHI) for the period of April 2007 to March 2012.

The evaluation was conducted to fulfill the requirement to evaluate all grant and contribution programs every five years by the Treasury Board (TB) Policy on Evaluation (2009) and ensure compliance with the Financial Administration Act (FAA).

2.0 Program Description

This section of the report provides an overview of the program being evaluated.

2.1 Historical Context

In 1991, the National Task Force on Health Information of the Conference of Deputy Ministers (DMs) of Health presented recommendations based on a series of workshops and discussions with key partners in the area of health information. A main finding was described by the Chair of the Task Force as the "deplorable state" of health information in Canada (Government of Canada, 1991). The Task Force determined that Provinces and Territories are highly protective of their health information, and that they are resistant to share with other organizations, including the two organizations that were dedicated to gathering health information at that time: the Management Information Services (MIS) Group and Hospital Medical Records Institute (HMRI) (National Health Information Council, 1991).

The Task Force identified a number of users of health information including health care providers, health care facility administrators, policy makers and the broader public consumers of health services who required information on the quality of health care in order to make informed decisions. The Task Force identified four reasons for improving health information systems in Canada:

To improve the health of Canadians;
To support health care technologies and services;
To improve management and policy processes; and
To help resolve the contradiction of the public expectation of expanded health services and the perceived need to control health care costs.

The report of the Task Force concluded:

"... it is the predominant view of researchers, health care providers, policy analysts and managers that significant improvements can be achieved, both in the health of Canadians and in resource management, if health information could be made more reliable, comparable, extensive and available" (p.6).

The findings indicated that the development of a Canadian health information system required the work of both the Canadian Centre for Health Information within Statistics Canada (SC), as well as a proposed non-government organization (NGO) which, ultimately, became CIHI. As a result, the Task Force recommended that:

A Health Information Coordinating Council (HICC) be established, acceptably representative of all Canadian health constituencies;
A non-government, non-profit Institute for Health Information (IHI) be established to operate under the direction of the HICC, as a collaborating partner of the Canadian Centre for Health Information;
Staffing of IHI should draw heavily on secondments and rotational assignments from participating health constituents;
Resource support should include revenues (or the equivalent) from direct customers of IHI and from all constituents who are represented (directly and indirectly) in HICC.

Based on these recommendations, CIHI was created in 1994 to lead the development and maintenance of a comprehensive and integrated health information system that would be used for policy development and health system management to improve health and health care.

2.2 Program Profile

The Health Information Initiative (HII) of CIHI, first established as the Roadmap Initiative, was a national vision and action plan to modernize Canada's health information system. Through the HII, CIHI seeks to improve accessibility and quality of health information for use by health care professionals and the public. Working together to provide health information, CIHI collaborates with partners including Health Canada (HC) and Statistics Canada (SC) as follows:

HC contributes to the development of a national strategy for an integrated Canadian health info-structure by providing funding as well as recommendations and strategic advice to the Federal Minister of Health.
SC, authorized under the Statistics Act^{Footnote 1}, collects, compiles, analyzes, abstracts and publishes statistics related to the health and well-being of Canadians. SC's primary objective in this area is to provide statistical information and analyses about the health of the population, determinants of health, and the scope and utilization of Canada's health care sector.
CIHI's broader mandate is to use the HII as a national mechanism to coordinate the development and maintenance of a comprehensive, integrated health information system for Canada and to provide and coordinate the provision of accurate and timely information required for the establishment of sound health policy, the effective management of the Canadian health system and generating public awareness about factors affecting good health.

CIHI defined its intended clients in its 2008-2012 Strategic Plan under "Who We Serve" as follows:

"CIHI produces information and analyses that are publicly available for stakeholders in the health system and beyond including ... senior government leaders and others involved in developing health policy; those who lead, plan and manage the delivery of health services at the front line; and members of the general public. CIHI ... supports health and health services research, health professionals and health records staff who are involved in the collection of data and academics ... as well as ... funders, policy-makers and health system managers--as well as other important customers, including data providers, analysts, clinicians and researchers."

Other groups contributing to health information include other national/provincial/territorial health organizations. The funding was designed to support the development of quality and timely health data and information, including making available a variety of pan-Canadian health indicators as well as other health reports and analyses. This funding also provided the HII with funds to enhance the comprehensiveness of health data systems as well as its reporting on wait times, and emerging health issues.

2.3 Program Logic Model and Narrative

The program logic model (Annex A) was founded in three strategic directions: More and Better Data, Relevant and Actionable Analyses, and Improved Understanding and Use.

Strategic Direction # 1: More and Better Data - was intended to enhance the scope, quality and timeliness of its data holdings. This strategic direction involved the development and management of databases and registries, the identification of gaps in data and methods to fill these gaps, the promotion of data standards and the assurance of data quality. The outputs were data sets, reporting systems, data/technical standards and quality reports. These outputs were aligned to generate immediate outcomes that focussed on increased comparability of data and improvement of quality of data for its clients.

Strategic Direction # 2: Relevant and Actionable Analyses - was intended to produce quality information and analyses that are relevant and actionable. This strategic direction involved the development and promotion of analysis of health system and population health issues, population health information systems and infrastructure, and health indicators while ensuring actionable and quality data. The outputs were reports and special studies that addressed priority health issues, health indicators and hospital balanced scorecard reports. These products were developed to realize immediate outcomes of increased awareness of health system performance and factors affecting good health.

Strategic Direction # 3: Improved Understanding and Use - was intended to help CIHI clients better understand and use its data and analyses. The outputs were education products and services including e-reporting applications, sites/portals to present health data and fulfillment of data requests while supporting privacy of data and client information. HII focussed on the use of these education products in support of immediate outcomes to increase awareness of data and information, increase attendance at educational events, and continue trust in CIHI's ability to safeguard health information.

The integration of immediate outcomes and strategic directions was to provide an increased understanding and use of health information and products as an intermediate outcome. The ultimate outcomes of the HII initiative were improved decision-making regarding health policy and effective management of the Canadian Health system and improved health of Canadians.

2.4 Resources

The majority of CIHI revenues during its initial years between 1994 and 1997 came from fees-for-service. This changed when the 1999 federal budget provided additional funding through the Health Information Roadmap Initiative (Roadmap I) which focussed CIHI activities on answering: "How healthy is Canada's health care system?" while Statistics Canada focused on "How healthy are Canadians?".

The 2003 Health Accord provided additional funds for Roadmap II and III, which:

included a continuation of Roadmap I objectives;
supported federal/provincial/territorial reporting of comparable performance indicators;
addressed new priority health information needs identified in the 2003 Health Accord; and
aimed to improve data collection and reporting on health system performance.

In 2007, HC consolidated the various Roadmap funding under the HII and also provided up to $24.7 million additional funding per year (Budget 2007) to CIHI for enhancement of data holdings and the continued development of new comparable health indicators for patient safety, quality and access to care, health outcomes, primary care, mental health and Aboriginal health.

Table 1 provides a breakdown of total HII funding for the five years covering 2007-2008 to 2011-2012, including core funding and grant funding for the HII.

Table 1 - Annual HII Funding (2007-2008 to 2011-2012)*
	Health Information Initiative / Roadmap	Core> Funding	Total Funding	Total Expensed	Surplus^{Table _t1b1 footnote _t1b1}
2007-2008	51,063,918^{Table 1 footnote _t1b2}	2,230,000	53,293,918	49,807,979	3,485,939
2008-2009	81,746,294	-	81,746,294	81,714,872	31,422
2009-2010	81,746,294	-	81,746,294	79,215,917	2,530,377
2010-2011	81,746,294	-	81,746,294	81,746,294	-
2011-2012	81,746,294	-	81,746,294	81,551,503	194,791
Total $	378,049,094	2,230,000	380,279,094	374,036,565	6,242,529
Table 1 footnotes Table 1 footnote 1 Surplus was returned to HC at end of fiscal year. Return to table 1 footnote 1 referrer Table 1 footnote 2 CIHI did not anticipate, and was not able to plan for, the new and expanded funding level in 2007-2008. This increased funding was received late in the fiscal year and as a result, CIHI was not able to spend the whole $81.7 allocation and returned the remaining amount back to HC. Return to table 1 footnote 2 referrer * Reflects funding received from HC.

3.0 Evaluation Methodology

This section of the report provides a description of the evaluation methodology including:

Evaluation scope;
Core evaluation issues and questions;
Overall evaluation design;
Data collection methods and analysis approach; and
Limitations to the methodology with mitigation strategies.

3.1 Evaluation Scope

The scope of the evaluation included all HII activities from fiscal year 2007-2008 to 2011-2012.

3.2 Evaluation Issues and Questions

This section of the report provides a description of the evaluation questions addressed in this evaluation, in relation to the core issues to determine whether the HII met its objectives (relevance and performance) as specified in the Treasury Board Policy on Evaluation (2009) (see Table 2).

Table 2 - Evaluation Questions by Issue
Issues	Report Section	Questions
Relevance
Issue 1 - Continued need for program: Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.	4.1.3	Does the HII continue to address a demonstrable need?
Issue 2 - Alignment with government priorities: Assessment of the linkages between program objectives and (i) Federal Government priorities and (ii) departmental strategic outcomes.	4.1.1	Is the HII aligned to Government of Canada and Health Canada strategic priorities?
Issue 3- Alignment with federal roles and responsibilities: Assessment of the role and responsibilities for the federal government in delivering the program.	4.1.2	To what extent are HII objectives consistent with Government of Canada roles and responsibilities regarding health information?
	4.1.4	Are there other organizations that provide or could provide similar information?
Performance (effectiveness, efficiency and economy)
Issue 4 - Achievement of expected outcomes: Assessment of progress towards immediate, intermediate and ultimate outcomes with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes.	4.2.1	Is the Program achieving the outcomes expected as outlined in the Logic Model? Immediate Outcomes a)To what extent has the HII contributed to increased comparability of data and improved quality of data in participating facilities and jurisdictions? b) To what extent has the HII contributed to increased awareness among its intended clients of health care system performance and factors affecting good health? c) To what extent are the intended clients aware of the HII's data and information? d) To what extent has CIHI increased attendance at educational events? e) Has CIHI developed trust in safeguarding health information? Intermediate Outcome f) To what extent do intended clients understand and use the HII's data and information? Long term Outcomes g) To what extent has the HII met its objective of improved decision-making regarding health policy and effective management of the Canadian Health System and improved quality of health? h) To what extent has the HII met its objective of improved health of Canadians through increased awareness and improved quality of health?
Issue 5 - Demonstration of efficiency and economy: Assessment of resource utilization in relation to the production of outputs and progress towards expected outcomes.	4.3.1	Does CIHI funding minimize resources while maximizing outcomes for the HII?
	4.3.2	Is the HII managed in the least costly way?
	4.3.3	How have the activities and outputs demonstrated value in terms of expenditures to achieve results?
	4.3.4	Are there more economical and / or effective alternatives to HII funding?

3.3 Evaluation Design

A mixed method design was used for this evaluation that primarily focussed on a review of evidence from existing documents and literature, recent surveys, evaluations and performance audits in order to reduce duplication of effort, reduce unnecessary recipient reporting burden and ensure good stewardship of government resources. Additionally, this cost-effective evaluation approach was selected given a full evaluation of the Health Information Initiative is scheduled in the near future (2014-2015).

3.4 Data Collection Methods

The evaluation used a number of methods to collect data. An evaluation framework was developed to guide the collection of data in support of the evaluation. Gathering multiple pieces of corroborating evidence helped improve the quality of the evaluation. Most evaluation questions were addressed through multiple lines of evidence as determined through a data collection template.

The methods used in this evaluation included:

a document review;
a literature review; and
a review of previous evaluations, audits and survey findings completed during the evaluation period.

Each of the methods is described in detail below.

3.4.1 Document Review

The document review provided important historical and contextual information for the HII. The documents included: Speeches from the Throne, departmental plans and priorities, CIHI strategic plans and annual reports, and information related to the health status of Canadians, the performance of the health care system and the availability of health information, which the HII objectives were intended to address. Information from these documents addressed relevance issues as well as performance issues related to the achievement of outcomes. Additionally, HII's financial documentation was reviewed to inform the evaluation on economy and efficiency.

3.4.2 Literature Review

The literature review focussed on examining the general need for health information for practitioners, decision-makers and the public. It also examined the approach used by other countries to address health information requirements and whether there were other organizations that could provide similar information.

The literature review gathered information from both peer-reviewed scholarly journals and grey literature such as industry journals, articles and websites. Data relevant to the logic model were extracted. The review also included internet-based literature searches for the purpose of supporting the evaluation questions related to alignment with government and departmental priorities and the federal roles and responsibilities.

An on-line review of a selection of Organization for Economic Cooperation and Development (OECD) nations resulted in a number of web-based documents and web pages that indicated whether similar organizations exist in these nations. The documents and web pages included descriptions of the scope and mandate of these organizations as benchmarks for international comparisons and addressed the question as whether or not other organizations do or could provide a similar function to CIHI. All documents are listed in Annex B - Bibliography.

3.4.3 Evaluations, Performance Audits and Surveys

The evaluation used data from previous evaluations, audits and surveys to address questions of performance (effectiveness, efficiency and economy). Specifically, survey results provided information on the evolution of HII, awareness and use of HII data, and satisfaction among stakeholders. Evidence from case studies found in previous evaluations provided additional validation of the achievement of expected outcomes.

The lines of evidence included two separate reports and a comparative survey: the 2010 Independent Evaluation of CIHI (which included results of a 2010 stakeholder survey), the 2010 Performance Audit of CIHI, and the 2012 CIHI Stakeholder Survey. The two stakeholder surveys were conducted to evaluate stakeholders' satisfaction, understanding and use of HII products, services and performance. These stakeholders included senior leaders and policy makers at all levels of government, leaders, manager and planners of front-line health care delivery, professional and health record staff, researchers and academics, and partner groups, such as, Statistics Canada. The 2012 survey was compared against 2010 survey data in order to provide evidence of trends/changes over time.

3.5 Data Analysis

The data were analyzed using qualitative and quantitative methods. Data collected by each method were systematically analyzed by question. Data from all lines of evidence were integrated and verified in order to arrive at the overall evaluation findings.

Preliminary findings were reviewed for accuracy and validity by internal Health Canada Applied Research and Analysis Directorate personnel who were familiar with CIHI and the HII. This was an important aspect of the data analysis as it added nuance to the interpretation of the data and also helped to validate the evaluation findings.

3.6 Limitations and Mitigation Strategies

Most evaluations have data limitations that may have implications on the validity and reliability of the evaluation findings and conclusions. The mitigation strategies put in place are to ensure that the evaluation findings can be used with confidence. Table 3 illustrates the data limitations and the mitigation strategies used for this evaluation.

Table 3 - Limitations and Mitigation Strategies
Limitation	Explanation	Mitigation Strategies
Selection bias	As part of the previous audit and evaluation, only a limited number of external stakeholders were included in the interviews (compared to the number of internal CIHI stakeholders), and these external stakeholders were selected by CIHI, thereby introducing a potential selection bias.	To mitigate the risk of self-selection bias and heavily weighted internal interviewees, the 2012 stakeholder survey was conducted with a broad selection of external stakeholders, representing regional, provincial and local health clients' perspectives.
Reliance on secondary data	Since this evaluation relied solely on secondary data (i.e., no primary data collection took place), there was a risk that the data was not always relevant or compatible with the scope of this evaluation.	To mitigate this, data from the previous evaluation and audit findings were triangulated and verified with additional information from the literature and document review to the extent possible.

4.0 Findings

This section of the report presents the findings of the evaluation organized according to the main evaluation issues related to the relevance (Section 4.1) and the performance of the program (Section 4.2), including efficiency and economy (Section 4.3). For ease of reference, key findings are profiled in text boxes at the start of each section with important summary text highlighted in bold.

4.1 Relevance

4.1.1 Alignment with Government of Canada and Health Canada strategic priorities

The activities, outputs and outcomes for the HII are clearly aligned with federal government and Health Canada strategic priorities and objectives, given that the HII aims to provide credible health information to be used by nation-wide stakeholders to improve the health care system.

The document review found evidence that the federal government continued to emphasize its commitment to protecting the health of Canadians and improving the health care system in two recent Speeches from the Throne (Government of Canada, 2010a, 2011a). Although the HII is not directly mentioned in these speeches, these two speeches indicate federal support for the continued investment for improvements in the performance of the health care system, collaboration on health information, better decision-making for health care, establishment of sound health policy, and effective management of the Canadian health system. Specifically, the 2011 Throne Speech reiterated the federal government's commitment to invest in health care "while respecting provincial jurisdiction and working with the provinces and territories to ensure that the health care system is sustainable and that there is accountability for results". This statement is in line with CIHI's mandate to work with provinces and territories to coordinate the development and maintenance of a comprehensive and integrated approach to health information in Canada. CIHI, using the HII, is accountable to collect, analyze and provide health information data that is pan-Canadian, reliable, and comparable (Government of Canada, 2011a).

In a review of successive 2008 to2012 federal budget documents and ongoing funding plans, it was clear that the federal government has maintained its commitment to address health information issues as evidenced by the continuation of an annual funding allocations to CIHI of $81.7M, $79.3M and $77.6M for 2012-13, 2013-14 and 2014-15 respectively(Government of Canada, 2008, 2009, 2010b, 2011b, 2012a).

In a review of the past five years of HC Reports on Plans and Priorities (RPP), the department expressed its ongoing commitment to the "generation and sharing of knowledge and information on which personal decision-making, regulations and standards, and innovation in health rely", which is the essence of the HII. Specifically, the RPP stated that HC was committed to its role as information provider, which aligns with the HII strategic directions and the HII logic model. To achieve health system innovation, an identified key initiative was to "support development and dissemination of knowledge and information" through funding for the HII. The 2011-2012 HC Departmental Performance Report further stressed the department's continued and important role "to be an information provider and to work closely with provincial and territorial governments to develop national approaches to health system issues, and promote the pan-Canadian adoption of best practices". As CIHI's primary role is that of a pan-Canadian health information provider, and the HII is an information sharing enabler, it is clearly aligned to HC strategic priorities.

4.1.2 Consistent with Government of Canada roles and responsibilities

HII objectives were consistent with Government of Canada roles and responsibilities regarding health information including support for carrying out the principles of the Canada Health Act.

Jurisdictional authority for health care in Canada is largely determined by the Constitution Act, 1867. While the provinces and territories are primarily responsible for the delivery of health care, the federal government supports the provincial/territorial role by providing transfer payments, establishing national standards, and undertaking other health-related functions outlined in the Department of Health Act and the Canada Health Act, such as collection, analysis, interpretation, publication and distribution of information as well as cooperating with provinces and territories on efforts to improve public health (Government of Canada, 1985; Health Canada, 2006, 2011b). These functions relate directly to HII's objectives to improve accessibility and quality of health information in Canada.

4.1.3 Continue to address a demonstrable need

There is a demonstrable need for a centralized, standardized pan-Canadian source of health information.

The Report of the National Task Force on Health Information (1991) formally recognized the need for a centralized, standardized pan-Canadian source of health information. As highlighted in the Program Activity Architecture, HC recognized the need for the health information to support the successful achievement of departmental goals for the Canadian Health System. Additionally, CIHI's strategic goals are to link quality pan-Canadian health information data to decisions and health outcomes that lead to healthier Canadians. The need for health information was further substantiated in the 1998 report on "Health Information Needs in Canada", which summarized national consultations with approximately 500 participants, representing six key health system groups and consumers from all provinces and territories. Success stories from users of health information such as those listed in Section 4.2.1g also verified the ongoing need.

CIHI clients also confirmed that HII serves a demonstrable need. In the 2010 evaluation^{Footnote 2}, 90% of respondents (senior decision-makers/executives, managers/directors, health care professionals, NGOs and researchers/educators) indicated that the HII fulfills an important need. The 2012 survey^{Footnote 3} confirmed this finding with 88% of respondents (senior leaders/policy makers at all levels of government, leaders, managers and planners of front line health care delivery, professional and record staff involved in the collection and use of health information, researchers and academics who analyze data and teach the next generation of health care providers, and partner groups such as SC) who stated that CIHI provided value to its clients.

Canada is not the only nation or health care organization that requires health and health care-related information to monitor and manage its health care system. The World Health Organization (WHO) advocates that "reliable, complete and timely information is essential for public health decision-making and action, including policy making, planning, programming and monitoring" (WHO, 2011). The National Institutes of Health in the United States (US) has similarly identified a growing need among health care delivery professionals for adequate health information based on the growing numbers of chronic conditions, and the burden on the health care system.

The information gathered by HII also allows CIHI to be a knowledge broker for the provinces and territories, where CIHI identifies best practices and learnings and shares them with the provinces and territories. Furthermore, without the HII, there would be a lack of collated pan-Canadian health information, analyses, databases and reports that could be used to inform decisions on how to improve the performance of the health care system. Decision-makers (including senior leaders/policy makers, managers and planners of front line health care delivery, researchers, academics, and partner groups such as Statistics Canada) rely on this information for planning and decision-making, and it is likely that the performance of the health care system would be negatively impacted through decisions made without reliable and complete health information.

The evaluation also confirmed a recognized need for a health information program through a review of literature, web-based documents and web pages of 11 of the current 34 OECD nations (Australia, Austria, Belgium, Demark, United Kingdom (UK) and US to name a few). These nations identified the need for national coordination of health information and created or engaged organizations with mandates similar to CIHI. Some countries established organizations outside of government, while others maintained the function within government. The specific activities of these organizations vary, but there is a commonality with respect to health system performance and the need for standardization of health information at a national level for decision-making and public awareness.

4.1.4 Other organizations that provide or could provide similar information

No other organization has been able to provide comprehensive, standardized and validated pan-Canadian data at the same breadth and to the same standard as CIHI.

The evaluation reviewed both government and private sector organizations that produced products that were similar to HII products. For example, from the government sector, Statistics Canada (SC) and HC produced some similar information/products often in collaboration with CIHI. HC published detailed reports titled: "The Health of Canadians" based on SC data and the document review also found that SC and CIHI jointly produced the Health Indicators Report. These reports include standardized and validated pan-Canadian data in specific areas of health.

In the not-for-profit sector, there are organizations like the Cancer Society of Canada and Wait Time Alliance that produced detailed and specific reports related to a particular aspect of health. For example, Wait Time Alliance^{Footnote 4} released an annual report focusing on changes in wait times for medical services. Although CIHI has provided information to organizations like Wait Times Alliance to help develop these types of reports, many reports include internet and other data that cannot be validated.

Hence, some organizations have provided similar health information to that produced by the HII, but the information was limited to specific issues and did not cover the same breadth as HII products, which address a wide spectrum of topics from the National Ambulatory Care Reporting System to the Canadian Joint Replacement Registry. Reports and analyses produced by other organizations may not use validated data and therefore may not be as accurate or reliable as the products produced by the HII.

The evaluation used survey data to confirm the usefulness of HII products. In the 2010 stakeholder survey, 77% of CIHI clients answered that many of products and services are not available elsewhere "to a very great/significant extent". The 2012 stakeholder survey confirmed this evidence, where 90% of the respondents indicated that data, analyses and services were not available anywhere else in Canada.

The evaluation noted, through multiple lines of evidence, that stakeholders believe that CIHI has fulfilled a unique role through an established trust relationship between federal/provincial/territorial stakeholders. CIHI maintains a neutral and independent position as a non-government organisation which allows the sharing of data by provinces and territories as well as the collection of data against well-defined standards to ensure security of health information as well as compatibility and comparability across the country. The evaluation concluded that this trusted relationship has taken a long time to establish and mature. It requires leadership and neutrality, not to mention the reliance on security and privacy of data, as its basis to encourage the sharing and collecting of provincial health information and could not be easily duplicated by another organisation as evidenced by the lack of success of CIHI's predecessors, the Management Information Services (MIS) Group and the Hospital Medical Records Institute (HMRI) where regional and provincial health care organizations were reticent to share data due to lack of national standards, lack of a pan-Canadian reach and the narrow focus on medical audit and other specialized areas.

4.1.5 Relevance - Conclusions

Overall, the evaluation concluded that activities of the HII are aligned with federal and departmental priorities, roles and responsibilities. The evaluation determined that the HII is aligned with the priorities of the Government of Canada and HC as it supports the improvement of health information for sustainability of the health care system. HII objectives were aligned with the federal role regarding health information, and stakeholders agree there is a continued need for standardized pan-Canadian health information. CIHI was perceived as a trusted partner by federal/provincial/territorial stakeholders and this enables the sharing and collecting of information by partners. No other organization has been able to provide such comprehensive, standardized and validated pan-Canadian data, at the same breadth and to the same standard, as CIHI.

4.2 Performance - Achievement of Outcomes

As indicated in the logic model (Annex A), the program funds activities that contribute to the achievement of the following immediate outcomes (listed in order of presentation in subsequent sections under 4.2.1):

a) Increased comparability of data and improvement of quality of data;
b) Increased awareness of health system performance and factors affecting good health;
c) Increased awareness of HII data and information;
d) Increased attendance at educational events; and
e) Continued trust in CIHI's ability to safeguard health information.

As outlined in the HII logic model, these immediate outcomes are anticipated to contribute to the achievement of the following intermediate outcome:

f) Increased understanding and use of health information/products.

Ultimately, these outcomes are anticipated to contribute to the following long term outcomes:

g) Improved decision-making regarding health policy and effective management of the Canadian Health System and improved quality of health; and
h) Improved health of Canadians through increased awareness and improved quality of health.

This section presents findings pertaining to the performance of the program organized according to the expected outcomes over the evaluation period (2007-2008 to 2011-2012).

4.2.1a) Immediate Outcome: Increased comparability of data and improved quality of data

The number of jurisdictions that are collecting data according to recognised standards has increased over the evaluation period, which lead to increased comparability of data as well as improved quality of data.

Increased comparability and quality of data supports the HII's strategic direction of "More and Better Data". Since 2011, CIHI has worked extensively with provinces and territories to improve comparability of data by continuing to develop common indicators that are acceptable to the stakeholders and aligning the databases in relation to the available information to increase both the number of jurisdictions that participate in the data collection process but also the comparability of data (collection of the data to recognised standards). The impact has been that more provinces and territories are participating and sharing health information and thus, there is higher quality and pan-Canadian coverage of data.

This work with the provinces and territories was conducted partially to respond to the 2011 CIHI annual report which stated that the comparability of the data had some limitations. Specifically, the 2011 Canadian Standards for Management Information Systems in Health Service Organizations database (MIS Standards) published by CIHI indicated that major data limitations included that not all hospitals report complete data and not all hospitals adhere to the MIS reporting standards which impacted the consistency and comparability of the data. This report indicated that many organizations chose not to submit operating statistics, and so data should be viewed with care.

The 2012 stakeholder survey provided some evidence of the extent to which the HII has been successful in its efforts to increase comparability and improve quality of data in participating facilities and jurisdictions. With respect to comparability, the 2012 stakeholder survey results indicated that 89% of respondents rated CIHI as "excellent/good" in that the HII data, analyses and services that are provided enable pan-Canadian comparisons. This result was in line with the 2010 survey where 87% of respondents rated compatibility of HII data as "excellent/good". Although a 2% increase may not be seen as significant, there was only a short timeframe between 2011 (when the increased sharing occurred) and the 2012 survey. The 2012 stakeholder survey also addressed the quality of the HII data, analyses and services. In both the 2010 and 2012 surveys, CIHI achieved a consistent 91% "excellent/good" rating from stakeholders representing all jurisdictions.

In the report Measuring and Reporting on Health System Performance in Canada, the Health Council of Canada (2012) recognized CIHI as "producing regular comprehensive public reports on the health system as well as on the health outcomes and health status of Canadians, using indicators derived from comparable data." Documented evidence identified two annual pan-Canadian reports produced under the HII, Health Indicators (2011) (co-produced with SC) and Health Care in Canada (2011) as reports that are comparable across the country with data at the provincial and regional levels. The Council also recognized that CIHI continued to work towards developing new data infrastructure and comparable health indicators that report on the health care system performance.

4.2.1 b) Immediate Outcome: Increased awareness among its intended clients of health care system performance and factors affecting good health

The HII had an impact on increasing awareness of health care system performance among its stakeholders, but was less successful at increasing the awareness of factors affecting good health.

A survey conducted as part of the 2010 evaluation confirmed that HII had contributed to increased awareness of health care system performance among health care providers, policy analysts and decision-makers Survey respondents believed that the HII had a major impact on increasing overall awareness of health care system performance (94% rated "to a very great/moderate extent"). Government clients were the most positive regarding HII's impact on increased awareness, but the differences among the respondent groups were not large. This confirmation was independently substantiated with evidence from the 2012 stakeholder survey. These surveys measured increased awareness of the health system performance factors across Canada and within jurisdictions where 96% of respondents stated that data, analyses and services helped contribute to increased awareness of health system performance factors across Canada. The survey also found that 90% of respondents indicated that the data, analyses and services helped to contribute to increased awareness of health system performance factors within jurisdictions.

The 2010 evaluation indicated that the public awareness of products identifying factors affecting good health was less comprehensive (but still satisfactory), citing 77% of those surveyed as being aware of products providing this information. Some of these products were analytical reports outlining various factors affecting good health, including depression rates in seniors, discharge destinations following strokes, hormone replacement theory, and drug use in seniors. Additionally, HII produced a report on Health Care in Canada (2010) that collated information on factors affecting good health such as the impact of smoking on heart attacks. Although these products were available, approximately a quarter of the Canadian public was not always aware of them. Also, the evaluation noted that the activities of the HII were more focussed on priority areas related to health care system performance instead of health issues in general. Overall, the evaluation concluded that there is less of an awareness of products related to factors affecting good health than those related to health care system performance.

4.2.1 c) Immediate Outcome: Increased awareness of the HII's data and information

The extent of awareness of HII data and information including the portal and e-reports among intended clients was consistently high.

The extent of awareness of HII products other than those dealing with health care system performance and factors affecting good health additionally supported the strategic direction of "Relevant and Actionable Analysis". The 2010 evaluation assessed stakeholders' perceptions of awareness and understanding by its identified intended clients, including senior decision-makers/executives, managers/directors, health care professionals, NGOs, and researchers/educators. HII used educational events, outreach through regional office and extended contact via email and notices to its client base to increase awareness. The evaluation confirmed that the awareness and understanding of the HII's data and information among HII's intended clients was consistently high across all products and services as outlined in Table 4. This awareness was validated by the 2010 and 2012 stakeholder surveys that both showed an "excellent" rating of the awareness and understanding of HII products by 91% of the respondents. These results also directly support the intermediate outcome of understanding of the HII data and products in Section 4.2.1 f). The table below demonstrates the level of awareness of HII products in the form of the percentage of the respondents who were aware of the HII products.

Table 4 - Rating of Awareness and Understanding of Stakeholders by HII Product
		Percentage of stakeholders who rated their awareness and understanding of HII products as "excellent"
Year	N	Data Collection Resources & Standards	Health Indicators	Analytical Products	Education Services	Subscriber Reports	Portal	Website Apps	Custom Data Requests & Analyses
2010	624	93%	93%	92%	91%	89%	89%	86%	84%
2012	593	93%	91%	94%	93%	89%	89%	87%	89%

4.2.1 d) Immediate Outcome: Increased attendance at educational events

HII maintained and slightly increased attendance at educational events which were perceived by clients as having "excellent/good" value.

The strategic direction "Improved Understanding and Use" focussed on ensuring that the products are understood and used appropriately by HII stakeholders. To achieve this strategic direction as part of the HII, CIHI offered events including workshops, e-learning events and conferences to communicate the use of their products. Stakeholder surveys measured the interest in these events by clients, satisfaction levels and ultimately, the perceived value of these events. In the 2010 survey, 93% of the respondents rated the clients' value of these services as "excellent/good", and 91% did so in 2012.

Statistics for attendance at educational events indicated an increase in attendance at educational events over the period of the evaluation as outlined in the table below.

Table 5 - Attendance at Educational Events*
Year	2007-2008	2008-2009	2009-2010*	2010-2011*	2011-2012	Totals
# of Educational Events^{Table _t2b1 footnote _t2b1}	311	335	477	292	257	1672
Attendees	10,180	12,223	9,992	24,376	22,733	81,176
Average attendees per event	33	36	21	83	88	48
Table 2 footnotes Table 2 footnote 1 Educational events included web conferences, video conferences, self-study packages, and workshops. Return to table 2 footnote 1 referrer * 2009-2010 registrations were lower due to a change in registration systems and data that were not recorded in the Learning Management System (LMS). 2010-2011 and 2011-2012 included additional learning events such as webcasts and portal events.

4.2.1 e) Immediate Outcome: Trust in safeguarding health information

CIHI had strong policies and programs in place to ensure safeguarding and protection of privacy of health information and was recognized by their stakeholders as a trusted agent.

CIHI developed and managed twenty six HII data holdings that are accessible to authorized internal personnel and authorized external parties. Due to the sensitivity and confidentiality of much of the data that was collected and held as part of the HII, the security and integrity of data holdings was central to CIHI's reputation and credibility with its stakeholders. As evidenced in successive annual reports, CIHI has formally documented information security policies, procedures and standards in place to safeguard health information and achieved acknowledgement as a prescribed entity, thereby recognized as meeting provincial safeguarding standards, by the Ontario Privacy Commission. Based on its policies on privacy and security CIHI has successfully achieved its renewal of prescribed entity status through the Office of the Information and Privacy Commissioner (OIPC) of Ontario. CIHI is subject to these standards in the province where its headquarters is situated and meeting the provincial government standards is required for its clients.

The 2010 performance audit indicated that CIHI had a strong formal information security program and that formal processes were in place to ensure that its privacy policies and programs were consistent with federal and provincial privacy legislation. As well, the success of these efforts to ensure privacy and safeguarding of health information under HII were reflected in the 2012 stakeholder survey. In this survey, 93% of the respondents indicated that they strongly agree that CIHI ensured privacy of sensitive matter and was recognized by them as a trusted partner.

These security measures to safeguard information contributed to provincial/territorial/regional stakeholders' increased comfort in sharing data with CIHI.

4.2.1 f) Intermediate Outcome: Intended clients understand and use HII's data and information

There has been an increase in the understanding and use of HII's data and health information among CIHI's intended clients.

In addition to the details outlined in Table 4 where awareness and understanding of HII's data and information among CIHI's intended clients was consistently high across all products and services, there were two lines of evidence that provided information on the extent of clients' use of HII's data and health information: the 2010 evaluation and the 2012 stakeholder survey. The 2010 evaluation addressed usage in five categories of HII products as shown in Table 6. As noted in this table, a comparison with the 2012 stakeholder survey data showed that all of the HII products experienced an increase in usage between 2010 and 2012.

The table below demonstrates the level of use of HII products in the form of percentages of the respondents who used its product and services.

Table 6 - Usage of HII Products (Data and Information) and Services
Products and Services	2010 Evaluation/survey	2012 Survey
Products	(N= 624)	(N= 593)
Data collection resources/standards	65%	71%
Data from data holdings	59%	63%
Analytical products	38%	56%
Health indicators	46%	63%
Subscriber reports	28%	44%
Services
Formal education services (e.g., workshops, e-learning, conferences)	54%	52%
Customized data requests/analyses	19%	14%
Client support services (e.g., e-query, direct contact and support from staff, including regional offices)	40%	5%

As shown in Table 6 above, all the products produced by HII experienced an increase in the percentage usage from the 2010 survey to the 2012 survey. On the other hand, the usage of services provided by HII decreased from 2010 to 2012. This trend is not unexpected. As HII clients become more aware and understand how to use HII data and analyses, client support services and formal education needs will reduce. Also, as HII increases the number (and use) of its products to meet the needs of its clients, the demand for specialised or customised data analyses will also reduce as identified in the table above.

4.2.1 g) Long Term Outcome: Improved decision-making regarding health policy and effective management of the Canadian Health System and improved quality of health

There is evidence to suggest that decision-makers are using HII information for health policy and effective management of the Canadian health system.

Through the achievement of increased understanding and use of HII information and products, the HII contributed to the long term outcome of improved decision-making regarding health policy, effective management of the health care system and improved quality of health. The 2010 evaluation indicated that the HII helped provide the evidence base necessary for health care providers and managers to make informed decisions about health system renewal. Specifically, the evaluation provided evidence that four of the eight categories of products and services (data collection resources/standards, data from data holdings, formal education services and health indictors) had been used by about half to two-thirds of CIHI's intended clients to help provide the evidence base necessary for health care providers and managers to make informed decisions about health system renewal in the past 12 months. From the 2010 evaluation, about two-thirds of CIHI's intended clients who used HII products and services in the past 12 months found them to be either "extremely useful" or "very useful" in this regard.

In the 2012 stakeholder survey, 91% of the decision-makers, including senior leaders, policy makers, professional and records staff, rated the HII contribution to the improvements to health policy and effective management of health services as either significant or moderate. Eighty-two percent of respondents also noted that the HII's data, analyses and services "supported the efficient and effective delivery of health services across Canada".

Table 7 - Extent of Use of HII Products in Health System Decision-Making
2012 Survey Results	Locations	To a very great/significant extent	To a moderate extent	Totals
Help inform policies	Across Canada	61% (n=362)	30% (n=178)	91% (n= 540)
Help inform policies	Within jurisdictions	46% (n=273)	39% (n=231)	85% (n= 504)
Supporting the efficient and effective delivery of health services	Across Canada	53% (n=314)	37% (n=219)	90% (n= 533)
	Within jurisdictions	44% (n=261)	38% (n= 225)	82% (n= 486)

As illustrated in Table 7 above, CIHI clients indicated (in the 2012 stakeholder survey) that HII products were used to provide evidence for decisions regarding both health policies as well as decisions regarding the efficient and effective delivery of health services not only within jurisdictions but also across Canada.

The document review of the CIHI annual reports also provided some anecdotal evidence indicating improved decision-making supporting improved quality of health, such as the Home Care Reporting System (HCRS) designed to help screen clients to identify their need for, and urgency of, home care services. An additional project, the Canadian Hospital Reporting Project, allowed 600 hospitals to make peer comparisons and identify best practices for implementation.

The document review also identified almost 60 anecdotal success stories/case studies as examples of how the use of HII products had improved health system decision-making leading to improved quality of health. The following success stories are samples from pan-Canadian sources:

National Rehabilitation Reporting System (NRRS) Data is being used and disseminated (2009) - Albert Health Services reported that the data allows program staff to plan for the challenges with stroke patients and adjust services to meet the needs of their patients. Specifically, follow-up data was recently used to evaluate system performance in which a Maintenance of Total Function Score from discharge to follow-up was implemented as a performance indicator in a regional evaluation of services for stroke patients. This valuable follow-up information tracks how clients are functioning as they continue their recovery in their home and community settings and allowed staff an opportunity to ensure that clients have access to the services and support to meet their needs.
Data Improves Rehabilitation Services in Canada (2009) - Toronto University Health Network used comparable national data to develop a Patient Flow Initiative to reduce the length of stay for clients in the program. As a result of this initiative, in just one year, the average duration of hospital stays for clients in the program decreased by 15 days, from 86.2 to 71.5 days. The shorter hospital stays permitted staff to provide care to 12% more clients. Importantly, this increase in efficiency and access to care was coupled with an improvement in functional outcomes. Clients and program staff alike were pleased with the benefits reaped by promoting standardization in practice.
Ontario Mental Health Reporting System (OMHRS) (2008) - The Providence Mental Health Site in Kingston used Mental Health Assessment Protocol summaries for patients who are being discharged to assertive community treatment teams to facilitate care planning. These summaries resulted in a collaborative review of the clients' needs and facilitated up-to-date care planning by the community teams, rather than focusing solely on clients' prior experiences in the community.
Hospitals using the Hospital Standardized Mortality Ratio (HSMR) to track improvement efforts (2010) - To track its patient safety efforts, the Brantford General Hospital used the HSMR. It allowed hospitals to assess their mortality rates and identify areas for improvement. As a result, Brantford created a decision-support team to mine data, research trends and help teams identify good practices and where they need to improve. Throughout this process, CIHI's HSMR was used as their baseline for their corporate scorecard of quality and safety indicators.
Canadian Organ Replacement Register (CORR) (2011) - This CIHI data allowed a team of clinical and program experts to classify Chronic Kidney Disease patients into seven clinically meaningful categories with distinct care and resource requirements. Once the patients were categorized within CORR, the data helped identify what services should be provided for patients and estimated what the cost would be for the medical staff at the Ontario Renal Network (ORN). This model resulted in better patient care, promoted a focus on quality and provided hard evidence on which patterns of care result in better outcomes.
At Willow Lodge Home for Special Care in Nova Scotia, staff used CIHI tools to assess the well-being of residents to assess mental wellness. The tools allowed staff to monitor not just how individual patients are doing (both physically and mentally) but how the nursing home is doing as a whole, compared with other long-term care facilities, using evidence-based quality of care indicators. The changes to residents' medications, activities and approach to patient care resulted in depression rates at the home dropping by half.

These success stories, along with many others, show HII's influence in the process of decision-making through the provision of evidence to improve health policy and effective management of the Canadian Health System with hospitals, health information management organizations, government policy makers and other targeted stakeholders. However, there is no formal process currently in place to capture and track these successes, making it difficult to assess the extent to which HII is achieving this long term outcome.

4.2.1 h) Long Term Outcome: Improved health of Canadians through increased awareness and improved quality of health

Given the many players involved in improving the health of Canadians through increased awareness and information, it was difficult to measure the HII-specific contribution to improvements to the health of Canadians. However, there was anecdotal evidence available as noted above in the success stories. A number of these success stories indicated an increase in awareness of staff within health organizations regarding areas of improvement. This increased awareness led to the implementation of new policies and/or practices in these organizations and improvements in the quality of health services being delivered, not to mention improvements in patient health outcomes. For example, the Willow Lodge Home for Special Care and the Toronto University Health Network identified above.

Other success stories led to improved quality of care and ultimately to improved patient outcomes through better tracking of patients' needs and progress as indicated in the examples concerning stroke patients and the National Rehabilitation Reporting System, and the patients of the Providence Mental Health Site and the Ontario Mental Health Reporting System.

As indicated in section 4.2.1 g), the 2012 stakeholder survey demonstrated that provincial/territorial ministries, regional health authorities, hospitals and other health care facilities used the HII products to make evidence-based decisions that have led to improvements to policies and practices within the health care system, which would indirectly improve the health of Canadians. CIHI's intent was to contribute to the improvement of the health of Canadians and improved quality of health through more and better data, relevant and actionable analyses, and improved understanding and use.

4.2.2 Performance - Conclusions

Overall, the HII increased awareness and understanding of its data and information among its intended clients. As well, the HII achieved, as noted by its clients, an excellent/good awareness of health care system performance and factors affecting good health across Canada and within jurisdictions. This awareness led to an increase in the use of the HII's data and information and increased attendance at educational events. Perception of client users was that the HII produced and maintained high quality comparable data. Clients also rated the comparability of data as excellent or good by 91% of stakeholders representing all jurisdictions. Stakeholders noted that CIHI was a timely and responsive contributor to the increased reporting of health care system performance. There was also strong evidence that CIHI was a trusted partner in safeguarding health information. The perceptions of stakeholders and anecdotal evidence also suggested that the HII has contributed to its long term outcomes; however, ideally there would be a routine process in place to capture and track these successes more systematically.

4.3 Performance - Efficiency and Economy

This section presents data on the efficiency and economy of the program and more specifically, focuses on optimization of outcomes, management of program resources, demonstrating value and whether available alternatives to CIHI exist.

4.3.1 Efficiency and Economy Outcome: Minimize resources while maximizing outcomes

CIHI had processes in place to minimize resources while maximizing outcomes.

A document review was conducted including annual reports, financial statistics and previous evaluations/audits to determine if CIHI minimized resources while maximizing outcomes. CIHI had a number of processes in place to support the efficient use of resources to maximize outcomes for the HII. For example, the 2010 performance audit indicated that CIHI had designed and implemented practices and processes that promoted economy and efficiency in the use of resources and were effective in supporting the achievement of HII objectives and expected results including:

the active engagement of stakeholders at all levels to provide input into new and modified products and services;
the implementation of new and improved processes to enable CIHI to respond to client and stakeholder needs in a more timely manner, such as the emerging issues team that is designed to respond to new and/or high priority issues (e.g., Severe Acute Respiratory Syndrome (SARS)) within the health care system;
a strong data quality management framework to safeguard the integrity of client data (efficiency);
a strong privacy program to protect the privacy of personal health information in accordance with relevant federal and provincial legislation;
a strong information security policy and related procedures to guide the safeguarding of information contained in data holdings; and
strong communication mechanisms and protocols, such as the publishing of CIHI Updates, which provides regular communications to targeted stakeholder groups; and CIHI Directions (now called LAND), which promotes success stories.

The 2010 performance audit further identified certain areas for improvement that addressed increased monitoring of practices and processes to ensure enhanced resource utilization to maximize outcomes. These areas for improvement included:

the addition of an overarching umbrella framework in place to coordinate and integrate CIHI's stakeholder engagement activities as a whole;
a revision of its performance measurement framework and approach to include more outcomes-based measures and to reduce the overall number of performance measures being formally monitored and reported on at the senior management and Board levels to only critical performance measures; and
some improvements to the policies on privacy and security involving strengthening passwords and authentication mechanisms.

Work has already begun by CIHI to address the three areas of improvement identified above. However, it is too early to tell whether these new processes and documents will have the expected results.

4.3.2 Efficiency and Economy Outcome: Managed in the least costly way

CIHI had the necessary processes in place to ensure appropriate spending, tracking and oversight of its allocations.

The 2010 performance audit indicated that CIHI had the necessary processes and measures in place to ensure all allocations are tracked, monitored and reviewed by CIHI management. In addition, CIHI was subject to performance audits and evaluations to ensure oversight of their activities and fiscal responsibility were practised.

As the HII was a knowledge-based activity, the allocations of its funds were directed in salaries and benefits, professional services, travel and advisory/committees along with expenses related to the supporting infrastructure including IT support to website and associated products. The breakdown of costs for HII is outlined below for 2007-08 to 2011-12. The breakdown showed percentage allocations for each of the key areas and activities associated with the operation of the HII.

As shown in Table 8, the only expenditure that has increased over the past five years is salaries and benefits, which is due to the increased number of products that CIHI is producing. More employees are required to support CIHI's increased product offering. However, expenses related to 'travel and advisory' and 'committee expenses' decreased due to a concerted effort to use technology instead of travel. Also, the 2010 performance audit focussed on value in hiring and concluded that "CIHI conducted cost-benefit or other relevant analysis to support the creation of new positions within the organization" and there is "quality and consistency in process to ensure value for money" in procurement.

Table 8 - HII Annual Expenses (2007-2008 to 2011-2012)
Nature of Expenses	2007-2008	2008-2009	2009-2010	2010-2011	2011-2012
Compensation and Pension Contributions	63%	52%	62%	71%	73%
Professional Services, Travel and Advisory Committee Expenses	16%	17%	18%	14%	11%
Occupancy, IT and Other	15%	15%	16%	14%	13%
Capital Expenditures*	6%	17%	3%	2%	2%
TOTAL	100%	100%	100%	100%	100%
* Per CIHI quarterly cashflow forecast and record of expenditures submitted to Health Canada.

4.3.3 Efficiency and Economy Outcome: Demonstrated value in terms of expenditures to achieve results

The HII's activities and outputs demonstrated value to clients and formal processes were in place to measure and report on performance.

The expenditures had a positive impact on the progress made toward the achievement of expected outcomes given that the achievement of immediate outcomes was generally successful. Value in terms of expenditures was also evaluated against the perceived value to the client. The 2010 performance audit established that the results of the 2010 survey proved the value of the HII where "CIHI is clearly believed to have filled an important and relevant need" to its stakeholders, and its "products are extremely useful". The value of the HII activities and outputs was demonstrated by HII clients in terms of their "excellent/good" rating for relevant, actionable, timely and integrated response as identified in Table 9 below. This table indicates the percentage of stakeholders that were surveyed who believed that the HII products were valuable in terms of their relevance, usefulness, timeliness and the degree to which they were integrated.

Table 9 - Rating of the Value of HII Products to its Stakeholders
Year of Survey	Percentage of stakeholders who rated the value of HII products as "excellent/good"
Year of Survey	N	Relevant to needs	Actionable	Timely	Integrated
2010	624	92%	76%	69%	N/A
2012	593	82%	78%	70%	69%

In general, CIHI's resource allocation was linked to its activities through alignment of funding with its three strategic directions. The table below shows the percentage of total funding in relation to each of the strategic directions.

Table 10 - CIHI Operating Expenses in Relation to its Strategic Directions*
Fiscal Year	Strategic Directions
Fiscal Year	More and Better Data	Relevant and Actionable Analysis	Improved Understanding and Use
2007-2008**	-	-	-
2008-2009**	-	-	-
2009-2010	39%	26%	35%
2010-2011	41%	26%	33%
2011-2012	42%	26%	31%
* CIHI operating expenses exclude capital assets acquisitions and contributions to the CIHI Pension Plan, but include amortization of capital assets and pension contributions. ** No data available as strategic directions not established.

The document review indicated that, over time, CIHI will purposefully shift its activity focus to the second and third strategic directives once the data infrastructure has been developed (more and better data). At this time, this trend is not visible. The tracking of the financial information for each strategic direction will facilitate this shift in focus by increasing awareness of where the dollars are being spent.

CIHI had a number of formal processes in place to track, measure and report on its performance. These processes included quarterly progress reports where CIHI formally reported on its progress against the HII strategic directions and priorities. These reports were circulated to the management team and also sent to the Board and made publicly available on CIHI's website. The reports included financial statements highlights, a description of key risks and opportunities identified through environmental scanning activities, financial and activity based progress by program and project, and actual results against targets for key performance indicators.

Performance against stated objectives was further reported on through CIHI's annual reports and periodic evaluations. Nonetheless, the evaluation concluded, using stakeholder perceptions and evidence presented in the 2010 performance audit, that HII demonstrated value to clients and had some formal processes in place to measure and report on performance, particularly for immediate outcomes. However, CIHI used many activity-based metrics (for example, number of educational events conducted and number of participants) instead of focussing on the results or outcomes of these activities, which limited the evaluation in determining overall results achieved for some outcomes, particularly in relation to the costs incurred.

4.3.4 Efficiency and Economy Outcome: More economical and/or effective alternatives

No other organization would be able to provide more economical and/or effective alternatives to CIHI.

This evaluation considered whether HII funding could be allocated to another organization, including another federal/provincial/territorial government or a private sector organization, to achieve results more efficiently or effectively.

The document review identified three major federal government organizations including Health Canada, Statistics Canada, and Public Health Agency of Canada (PHAC) who contribute to providing Canadians with health information (Health Council of Canada, 2012). There are other national non-governmental organizations similar to CIHI, such as the Canadian Partnership Against Cancer, Canadian Diabetes Association and the Canadian Stroke Network, that also contribute to health information. However, many of these organizations have specific mandates to provide information related to specific health conditions. CIHI was the only organization who reported on pan-Canadian health care system performance, developed and implemented data standards, and developed comparable national data structures for specific health information needs through the HII.

In order to assess whether CIHI is best positioned to provide national health information, the evaluation examined what similar countries have in place. Therefore, a literature review of 11 similar nations (Australia, Austria, Belgium, Demark, UK and US to name a few) was conducted and showed that these countries also have a specific organization(s) responsible for coordinating health information at the federal/national level in order to ensure commonality with respect to health system performance and standardization. The literature review also indicated that the risk of removing the federal/national involvement could potentially risk the standardization of data collected.

Due to the relationship that CIHI has developed with its partners (which has taken a long time to establish and nurture), provincial/territorial counterparts are now willing to share their information. This is partially due to the fact that CIHI is neither a federal nor a provincial/territorial organization. Therefore, to try to establish a new organization to take over this role of collecting standardized pan-Canadian health information would not be economical as it would take a significant amount of time to re-establish the trust that provinces and territories currently experience with CIHI.

4.3.5 Performance - Economy and Efficiency Conclusions

Overall, HII's activities and outputs demonstrated value to its clients. The evaluation concluded that CIHI's processes and financial practises provided evidence of effective monitoring and use of resources for the HII. The 2010 performance audit indicated that CIHI minimized resources while maximizing outcomes since "CIHI had designed and implemented practices and processes that promote economy and efficiency in the use of resources and were effective in supporting the achievement". The evaluation also concluded that the HII could not be achieved more economically through other means.

5.0 Conclusions

With respect to relevance, the evaluation found that the HII filled an acknowledged need for standardised, pan-Canadian coordinated health information to support decision-making and national reporting on health care system performance. CIHI has been able to encourage provinces/territories and other stakeholders to collect and share data in a standardized manner, which was not achieved prior to the inception of CIHI. With CIHI perceived as a reliable partner for sharing health information and a trustworthy agent for privacy protection with its stakeholders, it would not be economical to pursue alternative delivery models. Additionally, CIHI produces a significant breadth of information on various health issues which is not duplicated by any other single organization.

The evaluation confirmed that the HII objectives, mandate and activities are aligned and consistent with strategic objectives, mandates, roles and responsibilities of the Government of Canada with respect to the development of standardized and comparable information at the pan-Canadian level.

With respect to performance, the HII achieved successes in achieving its immediate and intermediate outcomes. The HII increased awareness and understanding of its data and information among intended clients, and achieved increased awareness of health care system performance and factors affecting good health both across Canada and within jurisdictions. This evaluation determined that the HII provided relevant, timely, responsive and integrated data and information to its clients in reporting of health care system performance. There was also strong evidence that CIHI was a trusted partner and leader in safeguarding health information. The long term outcomes of improved decision-making regarding health policy, effective management of the Canadian health care system and improved quality of health were substantiated with anecdotal evidence. Given the many players involved in improving the health of Canadians through increased awareness and improved quality of health, it was difficult to determine the HII contribution to the overall improvements in health.

6.0 Recommendation

Performances measures for the HII should be further aligned to a more outcome-based approach and, where feasible, use mechanisms to systematically document this evidence, particularly for the intermediate and long term outcomes.

HC will work in collaboration with CIHI to ensure the implementation of the recommendation.

Annex A -- Logic Model

The logic model for the HII (and, for all practical purposes, for CIHI as a whole) is shown below
	Stream 1	Stream 2	Stream 3
Strategic Directions	More and better data	Relevant and actionable analyses	Improved understanding and use
Activities	Develop and manage health databases and registries Identify and fill information gaps in targeted areas Develop and promote data standards (incl. EHR standards Ensure quality of data (incl. timeliness)	Develop new areas of analysis Promote analysis of health system and population health issues Contribute to the development of population health information systems and infrastructure Integrate data for actionable analysis Develop/promote health indicators Ensure quality of information	Provide appropriate access to health data and develop and extend access tools Enhance/expand education offerings and conferences Ensure privacy and security
Outputs	New/enhanced/maintained data sets/reporting systems New/enhanced/maintained data/technical standards Jurisdictional Data Quality Report	Reports/products and special studies that address priority themes New health indicators Hospital balanced scorecard report	Education products and services E-reporting applications Sites and registrants using the Portal Data requests filled and requests answered Privacy Impact Assessment Privacy audits completed
Outcomes (Immediate)	Increased comparability of data through implementation/adoption of data sets, reporting systems and standards Improved quality of data in participating facilities/jurisdictions	Increased awareness of health system performance Increased awareness about factors affecting good health	Increased awareness of CIHI data/information: Portal E-Reports Requests for data/information Increased attendance at CIHI education events Continued trust in CIHI's ability to safeguard health information
Outcomes (Intermediate)	Increased understanding and usage of health information/products
Outcomes (Long-term)	Improved decision-making regarding health policy and effective management of the Canadian health system and improved quality of health Improved health of Canadians through increased awareness and improved quality of health

Annex B -- Bibliography

CIHI. (1998). Health Information Needs in Canada. Canadian Institute for Health Information.Retrieved from http://www.cihi.ca/CIHI-ext-portal/pdf/internet/PROFILE_ROADMAP_HINC_EN
CIHI. (2010). Health Care in Canada. Canadian Institute for Health Information.
CIHI. (2011). Canadian MIS Database - Hospital Financial Performance Indicators, 1999-2000 to 2009-2010, Methodological Notes, Revised August 2011.
CIHI. (2012). 2012/2013 Operational Plan and Budget. Canadian Institute for Health Information.
CIHI. (2008). 2008 to 2012 Strategic Plan. Canadian Institute for Health Information.
Government of Canada. (1991).Canada Health Action: Building on the Legacy - Volume I - The Final Report.
Government of Canada. (1985). Canada Health Act, R.S.C., 1985, c. C-6. Retrieved from http://laws-lois.justice.gc.ca/PDF/C-6.pdf
Government of Canada (2009) Canada's Performance Report 2008-9 and 2010-11: The Government of Canada's Contribution. Treasury Board Secretariat.
Government of Canada. (2008). Responsible leadership: The Budget Speech 2008. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2008/pdf/speech-discours-eng.pdf
Government of Canada. (2009). Canada's economic action plan: The Budget Speech 2009. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2009/pdf/speech-discours-eng.pdf
Government of Canada. (2010a). Speech from the Throne to open the Third Session Fortieth Parliament of Canada. Retrieved April 10, 2012, from http://www.parl.gc.ca/Parlinfo/Documents/ThroneSpeech/40-3-e.html
Government of Canada. (2010b). Canada's economic action plan, year 2, the Budget Speech 2010: leading the way on jobs and growth. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2010/pdf/speech-eng.pdf
Government of Canada. (2011a). Here for all Canadians, Stability, Prosperity, Security: Speech from the Throne to Open the First Session of the Forty-First Parliament of Canada. Retrieved from http://www.speech.gc.ca/local_grfx/docs/sft-ddt-2011_e.pdf
Government of Canada. (2011b). The next phase of Canada's Economic Action Plan, a low-tax plan for jobs and growth - The Budget Speech. Retrieved from http://www.budget.canada.ca/2011/speech-discours/speech-discours-eng.pdf
Government of Canada. (2012a). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2012 - The Budget Speech. Retrieved from http://www.budget.canada.ca/2012/rd-dc/speech-eng.pdf
Harris - Decima. (2012). CIHI Stakeholder survey.
Health Canada. (2006, May 9). Government of Canada's role in Aboriginal health care - Fact Sheet - First Ministers' Meeting on Health Care September 2004. Retrieved January 20, 2012, from http://www.hc-sc.gc.ca/hcs-sss/delivery-prestation/fptcollab/2004-fmm-rpm/fs-if_02-eng.php
Health Canada. (2009). 2008-09 Annual Report on Plans and Priorities. Health Canada.
Health Canada. (2010). 2009-10 Annual Report on Plans and Priorities. Health Canada.
Health Canada. (2011). Healthy Canadians-A Federal Report on Comparable Health Indicators 2010. Health Canada.
Health Canada. (2011b). Canada's Health Care System. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/pubs/system-regime/2011-hcs-sss/index-eng.php
Health Canada. (2012). 2011-12 Departmental Performance Report. Retrieved from http://www.hc-sc.gc.ca/ahc-asc/performance/estim-previs/dpr-rmr/2011-2012/report-rapport-eng.php#a14
Health Council of Canada. (2012). Measuring and reporting on health system performance in Canada - Opportunities for Improvement". Retrieved from http://www.healthcouncilcanada.ca/tree/HCC_Health_Indicators_WP_EN_WEB.PDF
KPMG. (2010, November). The Canadian Institute for Health Information (CIHI) Performance Audit - Final Report.
KPMG. (2010, November). Evaluation of the Canadian Institute for Health Information.
National Health Information Council. (1991). Health Information for Canada: Report of the National Task Force on Health Information. National Council on Health Information. Retrieved from http://www.statcan.gc.ca/pub/4220352-eng.pdf
National Institutes of Health. (2011). Grant Guidelines: Department of Health Understanding Clinical Information Needs and Health Care Decision Making Processes in the Context of Health Information Technology. Department of Health and Human Services. Retrieved from http://grants.nih.gov/grants/guide/pa-files/PA-11-198.html
Office of Disease Prevention and Health Promotion. (2008). America's Health Literacy: Why We Need Accessible Health Information. An Issue Brief From the U.S. Department of Health and Human Services.
Senate Subcommittee on Population Health. (2009). A Healthy, Productive Canada: A Determinant of Health Approach. The Standing Senate Committee on Social Affairs, Science and Technology Final Report of Senate Subcommittee on Population Health.
Standing Senate Committee on Social Affairs, Science and Technology. (2002, October). The Health of Canadians - The Federal Role - Final Report. Volume Six: Recommendations for Reform. The Honourable Michael J.L. Kirby.
Statistics Canada. (2011). Canadian Community Health Survey, 2003-11.
Wait Time Alliance. (2012). Shedding Light on Canadians' Total Wait for Care: Report Card on Wait Time in Canada. Wait Times Alliance Canada.
World Health Organization. (2011). The need for strong health information systems: Rationale for the Health Metrics Network Framework.

Footnotes

Footnote 1

Return to footnote1 Referrer

Statistics Act, R.S.C. 1985, c S-19

Footnote 2

Return to footnote2 Referrer

The evaluation of CIHI was carried out over the period mid-January, 2010, through August, 2010, concurrently with a performance audit of CIHI. Both engagements were unique in their focus, with the performance audit independently assessing CIHI's operations for economy, efficiency, and effectiveness of Health Information Initiative ("HII") funds used within the scope of 7 key audit objectives. The evaluation on the other hand focused on evaluating CIHI as a whole, with a particular focus on analyzing the outcomes of CIHI's activities, irrespective of whether the outcomes resulted from the HII funding sources, given that distinguishing the outcomes of HII funding from CIHI as a whole is not possible, since the funding has been received through a series of grants since 1999.

Footnote 3

Return to footnote3 Referrer

The 2012 survey, which was completed in June 2012, was to evaluate satisfaction with products, services and performance.

Footnote 4

Return to footnote4 Referrer

The Wait Time Alliance (WTA) was formed out of concern among Canada's doctors over delayed access to care for their patients, and an interest in working collaboratively with stakeholders to improve wait times. The WTA is comprised of several national medical specialty societies whose members are directly involved in providing care to patients.

Page details

Date modified:: 2020-08-31

Evaluation of the Health Information Initiative 2007-2008 to 2011-2012

Table of Contents

List of Tables

List of Acronyms

Executive Summary

Purpose

Background

Methodology

Conclusions

Recommendation

Management Response and Action Plan

1.0 Evaluation Purpose

2.0 Program Description

2.1 Historical Context

2.2 Program Profile

2.3 Program Logic Model and Narrative

2.4 Resources

3.0 Evaluation Methodology

3.1 Evaluation Scope

3.2 Evaluation Issues and Questions

3.3 Evaluation Design

3.4 Data Collection Methods

3.4.1 Document Review

3.4.2 Literature Review

3.4.3 Evaluations, Performance Audits and Surveys

3.5 Data Analysis

3.6 Limitations and Mitigation Strategies

4.0 Findings

4.1 Relevance

4.1.1 Alignment with Government of Canada and Health Canada strategic priorities

4.1.2 Consistent with Government of Canada roles and responsibilities

4.1.3 Continue to address a demonstrable need

4.1.4 Other organizations that provide or could provide similar information

4.1.5 Relevance - Conclusions

4.2 Performance - Achievement of Outcomes

4.2.1a) Immediate Outcome: Increased comparability of data and improved quality of data

4.2.1 b) Immediate Outcome: Increased awareness among its intended clients of health care system performance and factors affecting good health

4.2.1 c) Immediate Outcome: Increased awareness of the HII's data and information

4.2.1 d) Immediate Outcome: Increased attendance at educational events

4.2.1 e) Immediate Outcome: Trust in safeguarding health information

4.2.1 f) Intermediate Outcome: Intended clients understand and use HII's data and information

4.2.1 g) Long Term Outcome: Improved decision-making regarding health policy and effective management of the Canadian Health System and improved quality of health

4.2.1 h) Long Term Outcome: Improved health of Canadians through increased awareness and improved quality of health

4.2.2 Performance - Conclusions

4.3 Performance - Efficiency and Economy

4.3.1 Efficiency and Economy Outcome: Minimize resources while maximizing outcomes

4.3.2 Efficiency and Economy Outcome: Managed in the least costly way

4.3.3 Efficiency and Economy Outcome: Demonstrated value in terms of expenditures to achieve results

4.3.4 Efficiency and Economy Outcome: More economical and/or effective alternatives

4.3.5 Performance - Economy and Efficiency Conclusions

5.0 Conclusions

6.0 Recommendation

Annex A -- Logic Model

Annex B -- Bibliography

Page details