Evaluation of the First Nations and Inuit Health Branch Communicable Disease Control and Management Programs 2008-2009 to 2013-2014

Prepared by
Office of Evaluation
Health Canada and the Public Health Agency of Canada

March 2015

Table of Contents

Executive Summary

Evaluation Purpose, Scope and Design

The purpose of the evaluation was to assess the relevance and performance of the First Nations Health Branch Communicable Disease Control and Management Programs (CDCM) for the period of April 2008 to March 2014. The evaluation fulfills the requirement of the Financial Administration Act (for Grants & Contributions) and the Treasury Board of Canada’s Policy on Evaluation (2009), as per Health Canada’s Five-Year Departmental Evaluation Plan.

Evaluation Purpose, Scope and Design

The evaluation assessed the relevance and performance (effectiveness, economy and efficiency) of the CDCM programs. Evaluation findings will support decision making for policy and program improvements.

The CDCM evaluation includes all the First Nations and Inuit Health Branch (FNIHB) CDCM programs and services as defined by the renewed 2011 Authorities (i.e., Immunization, Tuberculosis, HIV/AIDS and Communicable Disease Emergencies including Infection Prevention and Control). For the five years ending March 31, 2013, expenditures for the CDCM programs totalled $139.2 million, including $72.6 million in O&M and $66.6 million in contributions.

The methodologies used in the evaluation included an extensive document and administrative data review, literature review, 41 key interviews with program staff, an online survey with 105 community health representatives and a telephone survey of 15 regional program coordinators. Limitations of the evaluation, specifically those related to the reliability of surveillance data, were mitigated with various lines of evidence and the triangulation of data collected through various sources. 

Program Description  

The First Nations and Inuit Health Branch (FNIHB) supports the delivery of public health and health promotion services to on-reserve First Nations communities and one Inuit region (Nunatsiavut). The CDCM programs fund, through contribution agreements, initiatives that address immunization, including Vaccine Preventable Diseases (VPD), Blood Borne Diseases and Sexually Transmitted Infections (BBSTIs) focuses mainly on HIV/AIDS, respiratory infections focusing mainly on tuberculosis (TB), and the Communicable Disease Emergencies (CDE) initiative (e.g., pandemic influenza). CDCM also includes the Infection Prevention Control (IPC) Unit initiative (added in 2011/12).

Specific activities include public health measures for identification of risks (e.g., surveillance, reporting); prevention, treatment and control of cases and outbreaks of communicable diseases (e.g. immunization, screening, directly observed therapy); promotion of public education and awareness to encourage healthy practices; and strengthening community capacity (e.g., pandemic plans) to respond to and manage communicable diseases.

Evaluation Key Findings, Conclusions and Recommendations

KEY FINDINGS — RELEVANCE

Continued Need for the Program

Communicable Disease Control (including immunization and pandemic planning) is one of the mandatory programs designed to meet legislated standards such as provincial and territorial public health legislation and health acts that ensures public health and safety. The continuing need for the CDCM program activities is also attributed to the disparities in health outcomes experienced by First Nations living on-reserve in comparison to the general population, and an ongoing need for communicable disease emergency preparedness. The flexible design of the CDCM programs is largely responsive to the varied needs of communities, however, challenges still exist for both regional partners and individual communities to address competing health priorities and the prevalence of other contributing factors such as chronic health conditions and variations in community capacity and ability to plan and manage CDCM programs.

Alignment with Government Priorities

The CDCM objectives and priorities align with the Government of Canada and departmental current priorities and strategic outcomes by increasing access to health care and addressing the health status inequalities affecting First Nations and Inuit communities..

Alignment with Federal Roles and Responsibilities

The CDCM activities align with federal government jurisdictional, mandated and/or legislated roles and responsibilities through fulfilling various policies and authorities enabling the provision of health services for First Nations and Inuit.

KEY FINDINGS — PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

Immediate Outcomes

CDCM programs have made progress in improving public awareness and knowledge of communicable diseases. Alignment of regional initiatives allowed leveraging with provincial activities and resources. However, the reach and impact of such activities was uneven across the regions and communities and constrained by factors such as competing health priorities with limited human and financial resources.

Overall, most community health staff were satisfied with training, information, data and support provided to build knowledge and skills. Although training opportunities were provided, access varied considerably across regions and many regions were reliant on provincial activities. Investment in community health staff was perceived to be the most important factor contributing to the overall success and continuity of the CDCM programming.

FNIHB’s access to surveillance data on health outcomes has improved and was utilized to varying degrees to support policy development and program planning.  However, differences in key data sources as well as the quality and reliability of surveillance data across programs and regions constrains the utility of the information.

Significant progress has been made in terms of building collaboration and partnerships at the national, regional and community levels to share information and achieve integrated service delivery.  However, engagement and collaboration at the community level was more challenging due to staff turnover in some communities, changes in leadership, competing community priorities, and the political environment.

Intermediate Outcomes

CDCM programs contributed to increased uptake of communicable disease prevention and control measures by implementing various prevention and disease control strategies that enabled increased staff knowledge and skills for improved health care delivery within First Nations communities.

There are some indications that community capacity to manage CDCM programs has begun to increase, however, there are challenges at the community level including governance, accountability and human resource capacity.  Key informants indicated that they perceived community capacity building was critical for ensuring successful future transfer of communicable disease control and management to communities.

Long-term Outcomes

Although it was difficult to measure trends in incidence and prevalence rates, and to assess CDCM contributions to those changes, preliminary data indicated that the incidence of communicable diseases remained fairly stable in most regions and was reduced in some other regions over the period covered under this evaluation.

Demonstration of Economy and Efficiency

A review of the literature demonstrates the inherent economic benefit of programs designed to prevent communicable diseases. FNIHB introduced structural and other program changes that were perceived to have increased efficiency of program operations and introduced more strategic and coordinated approaches in the design and implementation of CDCM programming.

Appropriate Performance Measurement

While access to surveillance information has improved and was utilized to varying degrees to support policy development and program planning, further improvements are needed to increase the usefulness of the surveillance data for ongoing program monitoring, making policy and program decisions and setting priorities.

CONCLUSIONS

The CDCM program continues to fulfill the department’s obligation to delivery health services and meet the continued need to address health disparities.

The evaluation demonstrates that there is a continued need for CDCM program activities due to ongoing health disparities among First Nations related to communicable diseases. The activities within the CDCM program are priorities of the federal government including increasing access to health care and addressing the health status inequalities of First Nations.

While the increased program flexibility and more targeted approach is believed to be more effective in reacting to communicable disease issues, the absence of comparable and reliable information and ongoing surveillance of First Nations health outcomes makes it very challenging to assess the responsiveness to the needs of communities over time.   

Although it was difficult to measure trends in incidence and prevalence rates and to assess CDCM contributions to those changes, preliminary data indicated that the incidence of communicable diseases remained fairly stable in most regions and was reduced in some other regions over the period covered under this evaluation.

The CDCM is making progress in addressing the health effects of communicable diseases as evidenced by continued investments in community health resources, ongoing training and collaborative efforts between regional offices and the provinces, and efforts focused on improving community capacity to manage CDCM programs in terms of overall governance, accountability and human resource capacity.

Further, significant progress has been made in terms of building collaboration and partnerships at the national, regional and community levels to share information and achieve integrated service delivery leading to increased uptake of various prevention and disease control strategies and measures.

Although FNIHB’s access to surveillance data on health outcomes has improved and was utilized to varying degrees to support policy development and program planning, the complexity of surveillance data remains a challenge given differences in key surveillance data sources. The access to quality surveillance data and its reliability across programs and regions further constrains the utility of the information.

Recommendations

Recommendation #1

Continue efforts in building community capacity by  ensuring consistency in training and support for community health staff.

Development of community capacity to manage CDCM programs includes various supports which help communities to identify community needs, plan and organize services (including the short and long-term planning of financial and human resources), and maintain and/or improve the quality of services through relevant training for staff and the implementation of evidence-based policy, standards, guidelines, etc. The evaluation found that developing capacity of the community health staff and communities overall is crucial to success and continuity of CDCM programming. While some communities had high level of capacity for planning and management of the CDCM programming, other continued to face challenges with competing priorities and limited human resource capacity, and opportunities to engage with FNHIB regional offices.  

Recommendation #2

Strengthen program ability to address diverse ongoing as well as emerging community needs related to communicable diseases.

Program flexibility and targeted programming and investment were considered to be successful in meeting wide range of needs and priorities of communities across and within FNIHB regions. However, challenges remain with respect to Program’s ability to monitor and respond to wide range of needs and factors affecting communicable diseases such as further increasing collaboration and engagement with partners, integration and alignment of programming with other service providers and ensuring program resources are allocated appropriately based on priorities and need within communities.

Recommendation #3

Building on past successes, continue efforts to collect, access and use reliable surveillance data.

Numerous data limitations including issues with quality and reliability, timeliness and the ability to share data among partners and stakeholders remain a significant challenge for the Program and its efforts to increase community capacity to manage CDCM programs. Improved collection, reporting and overall access to surveillance data and health outcomes for First Nations living on-reserve will assist programs to continue to identify needs, planning and managing of programs, develop targeted responses and produce evidence based policies.

Management Response and Action Plan

First Nations and Inuit Health Branch Communicable Disease Control and Management (CDCM) Programs
Recommendations Action Plan Deliverables Expected Completion Date Accountability Resources
1. Continue efforts in building community capacity by ensuring consistency in training and support for community health staff.

Management is in agreement with the recommendation.

Supports FNIHB Strategic Goal #2 - Collaborative Planning and Partnerships and #4 - Supportive Environment in which employees can excel.
Based on the CDCM program standards, each region will identify top 2 priority core competencies for building community capacity. Annual regional progress report on top 2 priority core competencies training uptake by community health staff. March 2017 FNIHB ADM - Regional Operations

FNIHB Senior ADM

FNIHB Regional Executives

Executive Director, PPH

Executive Director, PHC
Resources from existing budget.
2. Strengthen program ability to address diverse ongoing as well as emerging community needs related to communicable diseases.

Management agrees with the recommendation.

Supports FNIHB Strategic Goal #1 - High Quality Health Services.
The Communicable Disease Emergencies (CDE) national team to revise, in collaboration with the FNIHB regions, the Communicable Disease Emergencies’ Planning Guidelines for On-Reserve First Nation Communities (Guidelines) to reflect lessons learned from preparedness and response activities to CDE events and to build on infection prevention and control principles (e.g., H7N9, EVD, seasonal influenza outbreaks, etc.). Revised Communicable Disease Emergencies’ Planning Guidelines for On-Reserve First Nation Communities and distributed to FNIHB Regions. December 2016 Dr. Tom Wong (ED)
Sony Perron (SADM)
Resources from existing budget.

In the event of a CDE, contingency funds are accessible to support national, regional and local preparedness and response activities.
The CDE national team to develop, in collaboration with the FNIHB regions, a process to review, strengthen, and test CDE plans for appropriateness periodically. This process will include the roles and actions of the national office, the regional offices, and the communities. Developed CDE Plan Cycle document and distributed to FNIHB regions. July 2016 Dr. Tom Wong (ED)
Sony Perron (SADM)
Resources from existing budget.
CDC National Office to lead the development of an STBBI (sexually transmitted and blood-borne infections) Framework STBBI Framework Document December 2016 Dr. Tom Wong (ED)
Sony Perron (SADM)
Resources from existing budget.
3. Building on past successes, continue efforts to collect, access and use reliable surveillance data.

Management is in agreement with the recommendation.

Supports FNIHB Strategic Goal #3 - Effective and Efficient Performance
SHIPCU to build on the existing relationship with PHAC to improve FNIHB access to, and usage of, PHAC CD data. Data sharing agreement between FNIHB(CDCD/SHIPCU) and PHAC for STBBI March 2017 Mary-Luisa Kapelus (DG)
Dr. Tom Wong (ED)
Sony Perron (SADM)
Resources from existing CDCD O&M budget to hire a contractor.
The CDC national team to continue to produce summary data tables based on Program Surveillance Data Annual Communicable Disease Program Data Tables document March 2016 Mary-Luisa Kapelus (DG)
Dr. Tom Wong (ED)
Sony Perron (SADM)
Resources from existing budget.

1.0 Evaluation Purpose

The evaluation assessed the relevance and performance of the First Nations Health Branch Communicable Disease Control and Management Programs (CDCM) for the period of April 2008 to March 2014.

The evaluation was required by the Financial Administration Act (for Grants & Contributions), and the Treasury Board of Canada’s Policy on Evaluation (2009), as per Health Canada’s Five-Year Departmental Evaluation Plan. The CDCM evaluation includes all the First Nations and Inuit Health Branch (FNIHB) CDCM programs and services as defined by the renewed 2011 Authorities (i.e., Immunization, Tuberculosis, HIV/AIDS and Communicable Disease Emergencies including Infection Prevention and Control).

2.0 Program Description  

2.1 Program Profile 

The First Nations and Inuit Health Branch (FNIHB) supports the delivery of public health and health promotion services to First Nations on-reserve and and one Inuit region (Nunatsiavut).

The CDCM Program was not administered in the same manner for Inuit as it was for First Nations. First Nations communities receive funds through contribution agreements signed with the First Nations and Inuit Health Branch (FNIHB). For Inuit living in Inuit communities (with the exception of those in Nunatsiavut, Labrador and Nunavik, Quebec), the program is managed directly by the Territorial governments. As a result, when referring to the CDCM in this document, the information largely refers to First Nations and not Inuit, unless specifically mentioned.

CDCM service delivery is intended to occur at the community level through community governance structures, consistent with the principles of health transfer and self-government. Although First Nations communities are responsible for the delivery of CDCM and client service results, the territorial governments of the Northwest Territories and Nunavut are responsible for CDCM service delivery for the First Nations and Inuit living in those territories. Within the CDCM Program, contribution funding is allocated from the FNIHB Regional Office to communities/tribal groups/First Nations.

In Inuit Nunangat (Inuit homeland), which includes all four Inuit regions: Inuvialuit Settlement Region, Nunavut, Nunavik, and Nunatsiavut, programs are administered in a slightly different manner. The Governments of Nunavut and the Northwest Territories deliver their programs to all residents regardless of ethnicity. Health Canada’s Northern Region acts as a coordinating body between Health Canada and the territorial governments. The Nunavik Regional Board of Health and Social Services delivers the program to Inuit in Quebec only. The government of Nunatsiavut delivers the program directly to Inuit residing in that region. Nunavut’s program is administrated through three regional coordination centres located in Cambridge Bay, Rankin Inlet, and Iqaluit with service delivery variations between the centres.

FNIHB delivers CDCM programming to reduce the incidence, spread and human health effects of communicable diseases. CDCM programs focus on vaccine preventable diseases (VPD), immunization, blood borne and sexually transmitted infections (BBSTI including HIV/AIDS), tuberculosis (TB), and communicable disease emergencies (CDE) which includes the Infection Prevention and Control Initiative (IPC). CDCM programs consist of the following four programming areas:

  • The Vaccine Preventable Diseases (VPD) - Immunization Program aims to improve vaccine coverage rates for routine immunizations and reduce VPD incidence and outbreaks in on-reserve First Nations communities. Through the VPD and Immunization Program, FNIHB works with First Nations partners, provinces/territories and other federal departments/agencies, such as the Public Health Agency of Canada, to support culturally appropriate immunization programming, improve data collection mechanisms, and increase immunization coverage rates on reserves. FNIHB regions work closely with their provincial counterparts to align immunization programs with the goal of ensuring basic immunization services on-reserve are comparable to those delivered by the provincial government to the general population.
  • The Communicable Disease Emergencies (CDE) Program (including IPC) works closely with partners such as other government departments, national and regional Aboriginal organizations and provinces to develop a comprehensive and coordinated response to public health emergencies in First Nations on-reserve communities. The CDE Program works closely with the Public Health Agency of Canada (lead organization for public health emergency planning in Canada) as well as Aboriginal Affairs and Northern Development Canada (AANDC). ANNDC, the lead organization for all hazards emergency planning in Aboriginal communities, is mandated to ensure the needs and considerations of First Nations communities are reflected in overall federal/provincial/territorial planning and responses to public health emergencies such as influenza pandemics. The CDE Program is also responsible for the development and implementation of Infection Prevention and Control (IPC) activities as outlined in the FNIHB IPC Action Plan. Further, the CDE program provides IPC expertise, guidance, and advice to national and regional staff within FNIHB.
  • The Blood Borne Diseases and Sexually Transmitted Infections (BBSTI) and HIV/AIDS Program focuses on prevention, education, awareness and community capacity (training) building, and facilitates access to quality diagnosis, care, treatment, and social support. Working closely with partners, FNIHB and the Assembly of First Nations are working towards the creation of a framework to guide and/or complement the delivery of Health Canada’s program components that address BBSTIs in First Nations on-reserve. Given that significant funding for this program originates from the Federal Initiative to Address HIV/AIDS in Canada, led by the Public Health Agency of Canada, the primary focus is on HIV/AIDS. However, work on other BBSTIs does occur at the national and regional level based on needs and priorities identified by communities and stakeholders.
  • The Tuberculosis (TB) Program works to assure that appropriate and effective TB prevention and control services are available to First Nations on-reserve and in one Inuit region (Nunatsiavut) by either providing services directly or funding communities, provinces and/or regional and First Nations health authorities to provide services. The program promotes access to equitable and culturally appropriate care, including timely diagnosis, treatment, follow-up care, health promotion and TB awareness activities.

2.2 Program Resources 

For the five year period ending March 31, 2013Footnote 1, expenditures coded to the CDCM programs were disclosed as totalling $139.2 million, of which about $72.6 million was spent on operations and management, including salaries, and $66.6 million in contributions to communities.

2.3 Program Logic Model and Narrative

The long term expected outcome for the CDCM Programs is to reduce the incidence, spread and health effects of communicable diseases. To achieve this final outcome, CDCM activities focus on five major themes:

  • The provision of services, focusing on CDCM prevention (health promotion and disease prevention) and control measures (disease management/treatment). Pandemic plans can also be considered as a prevention and control measure.
  • Capacity building through the training of staff and community health workers, and public education and awareness as well as community health planning and management support.
  • Stakeholder engagement and collaboration with provincial and territorial governments, First Nations, other federal organizations, and other stakeholders is critical given the complex and inter-jurisdictional nature of health care in Canada and the influence of multiple factors, such as social determinants of health, which are beyond the sole control of Health Canada. 
  • Data collection, research and surveillance, which informs the development and refinement of policies, standards, guidelines and frameworks, and supports service delivery.
  • Policy development and knowledge sharing involving the use ofsurveillance and evidence generated through relevant research and used to inform the delivery of communicable disease-related services on-reserve. 

The intended reach for the Program is First Nations reserves, nurses and support workers, provincial governments, Regional Health Authorities, FNIHB regions, and the Public Health Agency of Canada.

In the short term, the outputs associated with these five themes were expected to increase workforce skills, knowledge and expertise in CDCM, increase public awareness and knowledge of prevention and control measures as well as communicable disease health effects, inform service delivery, and increase community and stakeholder engagement and collaboration in policy/program development and service delivery.  In the intermediate-term, it was expected that there will be increased use/uptake of CDCM prevention and control measures as well as increased community capacity to manage CDCM programs. In the long term, the CDCM programs were expected to contribute to reduced communicable disease incidence and prevalence rates.

The connection between these activity areas and the expected outcomes is depicted in the logic model (provided in Appendix 3). The evaluation assessed the degree to which the defined outcomes were being achieved over the evaluation timeframe.

2.4 Program Alignment

 The activities of FNIHB contribute to Strategic Outcome #3 of Health Canada: First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status. Towards that end, FNIHB delivers a suite of programs, services and strategies provided primarily to First Nations and Inuit individuals, families, and communities living on-reserve or in some Inuit communities. CDCM is an activity of Public Health and Promotion, which forms part of the Primary Health Care activity.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design 

The evaluation covered CDCM program activities for the period from April 2008 to March 2014. CDCM program activities are provided to First Nations living on-reserve, with some support provided to address TB in one Inuit region (Nunatsiavut).  The evaluation focused primarily on First Nations communities as FNIHB has very limited involvement in direct service provision to Inuit communities. 

The evaluation addressed a series of evaluation questions aligned with the five core issues defined under the Treasury Board of Canada’s Policy on Evaluation (2009) including: the continued need for the program, alignment with government priorities, alignment with Federal roles and responsibilities, achievement of expected outcomes and demonstration of efficiency and economy. Specific evaluation questions addressed in this evaluation are presented in Appendix 4. A more detailed description of the evaluation methodology is provided in Appendix 4.

In summary, data for the evaluation was collected using various methods including: document and data reviews; interviews with a cross section of 41 key program staff including 10 from the national office, 24 regional staff members, two representatives of the Public Health Agency of Canada and five national program partners and experts; an online survey with 105 community representatives; a telephone survey of 15 FNIHB regional program coordinators; a literature review including a review of similar programs/initiatives operating in jurisdictions outside of Canada; and, a review of differences in models and practices across regions.  The use and triangulation of multiple lines of evidence was implemented to increase the reliability and credibility of the evaluation findings and conclusions.

The report is structured according to evaluation issues and questions (see Appendix 4 for a full list of evaluation questions). Key summaries of findings are provided for each evaluation question(s), followed by key findings (in bold) and detailed evidence.

For the ease of reporting survey and interview results, the following quantifiers are occasionally used in the report: Most, which refers to over 75% of participants who responded within the particular group; Majority, which refers to between 50% and 75% of  respondents; Nearly half, which refers to between 45% and 50% of respondents; Some, which refers to between 25% and 45% of respondents; and A few, which refers to less than 25% of, or between 1 and 3, respondents.

3.2 Limitations and Mitigation Strategies 

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 1: Limitations and Mitigation Strategies
Limitation Impact Mitigation Strategy
There were significant challenges associated with the availability, quality and reliability of the surveillance data.

Publicly available surveillance data typically does not report on First Nations on-reserve health outcomes (TB data is the exception).
While much of the surveillance data available to FNIHB was compiled and analysed by the Program, the limited reliability of the information did not allow for a comprehensive assessment of the trends and achievement of intermediate outcomes (increased prevention and uptake of health measures) and long term outcomes (reduced incidence and prevalence).  To assess the achievement of the Program with respect to these outcomes, the evaluation relied on the survey and interview findings as well as literature reviewed and data available on Aboriginal populations.  Information on Aboriginal populations was used as a ‘proxy’ to comment on the potential issues faced by First Nations communities and, only when such information was not available for First Nations.  
Due to the nature of the programming, most performance data collected by the Program is focused on program outputs including descriptions of activities, which was very difficult to quantify and use.  Much of the information collected through reporting templates was focused on the production of outputs and provided limited information about the effectiveness of these outputs as they related to outcomes.   Surveys and interviews were utilized to assess the extent to which program outcomes were perceived to be achieved including effectiveness. This information was used to identify perceived effective strategies.
The use of key informant interviews and surveys with the community health representatives and regional coordinators, respondents who have a vested interest in the Program, can create the potential for respondent biases. There is also the potential for non-response error among the survey of community health representatives. Respondent biases may affect reliability and validity of the findings. Given the self-selected nature of the survey, the characteristics of community health representatives who completed the survey may be different from those who did not. Several measures were implemented to reduce the effect of respondent biases: (i) the purpose of the evaluation, its design and methodology, and strict confidentiality of responses were clearly communicated to respondents; (ii) the interviews were conducted by skilled interviewers; (iii) answers from each sample of respondents were cross-checked with those of other groups for consistency and validation; and (iv) the findings of the key informant interviews and surveys were validated with the results from other lines of evidence.
Wide variations in program structure, design and delivery across regions and operating environments created limitations in assessing the effectiveness of some Program components across the entire CDCM program. Not all findings of the evaluation apply equally to all regions. Whenever possible, the differences across regions and programs were highlighted in the report.

4.0 Findings

4.1 Relevance: Issue #1 – Continued Need for the Program

4.1.1    Does CDCM continue to address a health need? Has the need changed over time? Has the program adapted to changing needs?

Communicable Disease Control (including immunization and pandemic planning) is one of the mandatory programs designed to meet legislated standards such as provincial and territorial public health legislation and health acts that ensures public health and safety. The continuing need for the CDCM program activities is also attributed to the disparities in health outcomes experienced by First Nations living on-reserve in comparison to the general population, and an ongoing need for communicable disease emergency preparedness. The flexible design of the CDCM programs is largely responsive to the varied needs of communities, however, challenges still exist for both regional partners and individual communities to address competing health priorities and the prevalence of other contributing factors such as chronic health conditions and variations in community capacity and ability to plan and manage CDCM programs.

The Need for CDCM Program Activities

The CDCM Programs are mandatory health programs designed to meet legislated standards (such as provincial and territorial public health legislation) and health acts that ensure public health and safety. The CDCM Programs address the CD-related health needs of First Nations on-reserve communities under federal jurisdiction. According to the literature and documents reviewed, Canada’s Aboriginal population (including First Nations) continue to experience disparities with respect to the incidence rates of communicable diseases.Footnote 2 For example:

  • TB rates continue to be higher among First Nations on-reserve communities when compared to the non-Aboriginal population. In 2012, the reported incidence rate of new active and re-treatment TB cases was 23.3 per 100,000 population of First Nations living on-reserve, while the incidence rate reported for non-Aboriginal populations born in Canada was 0.7 per 100,000.Footnote 3 From 2000 to 2008, a total of 27 TB outbreaks were reported among First Nations on-reserve communities across Canada.
  • Although public information on the incidence of BBSTIs including HIV/AIDS among First Nations people living on-reserve is very limited, the available data estimates that Chlamydia is almost seven times higher among First Nations adults than among the population overall.Footnote 4 Rates of acute Hepatitis C Virus were 5.5 times higher for Aboriginal persons than for non-Aboriginal persons between 2004 and 2008. Public Health Agency of Canada data indicates that Hepatitis C rates among First Nations (2012) were 81.5/100,000.Footnote 5 According to the 2011 National Household Survey, 4.3% of the Canadian population self-identified as ‘Aboriginal’; however, in 2011, Aboriginal people accounted for an estimated 8.9% of all people living with HIV in CanadaFootnote 6 and 12.2% of all new HIV infections (as compared to 10.5% in 2005).Footnote 7 The estimated overall new HIV infection rate among Aboriginal peoples was 3.5 times higher than among the non-Aboriginal population in 2011.
  • Although VPD incidence rates among on-reserve First Nations communities are low, with the exception of pertussis and invasive pneumococcal diseaseFootnote 8, the need for continuing immunization programs remains high. The Public Health Agency of Canada warns that, as the incidence of a VPD decreases following successful immunization programs, there is the potential for Canadians to become complacent and to question the role of vaccines in preventive healthcare. This can lead to lower immunization coverage and a resurgence of the diseases. There is a limited reliable data available on immunization rates among on-reserve First Nations communitiesFootnote 9. The Canadian Supplement to The State of the World’s Children reported, in 2009, that the immunization rates among First Nations children in Canada have been well below the acceptable target of 95% to 97% set by the National Advisory Committee on Immunizations.

There is a strong continuing need for FNIHB’s targeted CDCM Program activities, given that First Nations people continue to experience much greater incidence and prevalence of communicable diseases relative to the general population.

While other developed countries with large indigenous populations also report discrepancies in health outcomes between the Aboriginal and non-Aboriginal population, Canada appears to perform worse on some communicable diseases. For example, the TB incidence rate among the Indigenous Australian population was 5.7 cases per 100,000 populations in 2007, and the rate among American Indians and Alaska Natives was 6.5 per 100,000 population in 2010, compared to 23.3 per 100,000 population of First Nations living on-reserve in 2012. The rates of new HIV diagnoses among the Aboriginal and Torres Strait Islander population in Australia is very similar to rates among the non-Indigenous population (5.5 vs. 5.1 per 100,000 in 2012).Footnote 10 As stated previously, the estimated overall new HIV infection rate among Aboriginal peoples was 3.5 times higher in Canada than among the non-Aboriginal population in 2011.

Variations of Needs across Communities

The need for, and priority placed on, particular types of CDCM programming can vary widely across regions and communities. 

Communities served by CDCM programming vary widely in terms of geography, culture, language, and political structures. According to program documents, 35% of the First Nations communities are geographically isolated (i.e. have limited or no road access); the majority (56%) are small communities with a population of fewer than 500; the communities speak over 50 distinct languages; and over 100 First Nations communities across Canada can be under a Drinking Water Advisory at any point in time.Footnote 11 In addition, socio-economic status, the frequency and severity of environmental emergencies, and health status, including issues directly related to communicable diseases, vary significantly across communities. Some of the differences across regions and communities that impact the need for, and priority placed on, CDCM programming include:

  • Variations in community capacity and ability to plan and manage programs. The level of engagement of community leaders and other stakeholders, the accessibility of existing infrastructure (e.g., health centres or nursing stations, HIV/AIDS testing units, etc.), and the ability to plan and manage communicable disease programming varied significantly across communities. Some regions noted that the ability to manage CDCM programs and, deal with turnover of health staff and changes in community leadership differed even among communities that had transfer agreements and full authority over CDCM delivery.
  • Experience with recent outbreaks of communicable diseases. According to FNIHB Performance Measurement Systems reports and key informant interviews, over the last five years, regions in Eastern Canada had very low levels of TB. As such, CDCM programming activities in these regions continued to maintain a high level of TB awareness; however, TB related activities were not a high priority in most communities. The majority of TB outbreaks were reported among First Nations living in Manitoba and Saskatchewan, partially due to large or ongoing outbreaks in certain communities and the remoteness and isolation of a large proportion of the First Nations living in these two regions.Footnote 12  
  • The level of involvement of other stakeholders and partners in communicable disease programming. The number of First Nations on-reserve communities for which CDCM programming is under direct management of FNIHB varied across regions (e.g., in Quebec about 80% of communities directly delivered CDCM programming, while that was the case in 10% of communities in Alberta). In some regions, provincial partners were more involved in developing resources or strategies specifically for First Nations. This means that the level of engagement and support available for communicable disease programming (e.g., provincially developed awareness materials) varied not only across regions, but also across communities, which in turn required highly flexible and adaptable CDCM programming.
  • The prevalence of other contributing factors. Communicable disease rates are often linked to socio-economic factors such as low levels of income, poverty, and inconsistent employment which affect the quality of care First Nations can receive, the existence of other chronic health conditions, individuals’ tendency towards drug use etc.Footnote 13 According to most regional staff interviewed, communities dealing with a significant number of health priorities required a higher level of collaboration, a more integrative approach to health services and, additional supports for community capacity (i.e., training) building. 
Responding to Communicable Disease Emergencies (CDE)

The need for CDCM programming, and specifically for a CDE response that reflects the special considerations and needs of First Nations on-reserve communities, is particularly evident in times of outbreaks and during events such as H1N1.

Concerns around public health often come to the forefront in times of crisis, such as during outbreaks of Severe Acute Respiratory Syndrome, avian influenza, H1N1, or the occurrence of natural or accidental disasters. The outbreak of H1N1 in 2009 clearly demonstrated a need for on-going planning, collaboration and implementation of pandemic responses that address the specific needs of First Nations and the special characteristics of First Nations on-reserve communities. In response to the H1N1 influenza pandemic, Health Canada worked with First Nations on-reserve communities to complete pandemic plans, reallocate health human resources as needed, and collaborate with all levels of government.

The Trilateral First Nations Pandemic Committee and working group, established in 2008, was extended through fiscal year 2010-2011. This committee had the mandate to develop and monitor the implementation of a trilateral work plan on pandemic influenza preparedness and response for First Nations on-reserve communities. A joint work plan with Indian and Northern Affairs Canada, ( now AANDC) was developed, which outlined the activities, expected outcomes, leads, timelines and status with respect to information sharing, First Nations pandemic planning and emergency management, web-based tools, surge capacity and business continuity, and collaboration with partners.

In 2009, the Influenza Pandemic Planning Considerations in On-reserve First Nations Communities, Annex B of the Canadian Pandemic Influenza Plan for the Health Sector was updated to clarify the roles and responsibilities. In collaboration with the regional CDE Coordinators, CDE guidelines were developed to support the development, strengthening, and implementation of CDE plans, formerly referred to as pandemic plans, at the community level. Health Canada estimates communities have a pandemic plan (91.7%), emergency management plan integrated with a pandemic plan (81%) or have an all hazards emergency plan (66.9%)Footnote 14.

CDCM Programs Success in Responding to Needs

The flexible design of the CDCM Programs allowed for targeted programming and investments as well as integrated services and enabled the programming to be responsive to differing needs and operating environments.   

In recognition of the vast differences in the needs and priorities of communities across and within FNIHB regions, the CDCM programming increased funding flexibility to allow regions to take a more targeted approach in responding to ongoing as well as emerging needs of First Nations communities. The ability of the programs to be able to shift resources quickly in 2009 toward preventing and managing H1N1 was a good example of the flexibility and responsiveness of the programming to emerging needs. The flexibility of the programming was reflected in regional differences in approaches including: 

  • The level of integration in programming at the regional level. Some regions adopted a more cluster-based approach to the delivery of FNIHB programs and services, allowing more integration between the programs. For example, in the Alberta region, the CDE coordinator position was filled by an environmental health officer to achieve a more strategic approach in responding to environmental and communicable disease emergencies. Some regions had coordinators who were the single point of contact for more than one CDCM program for a number of communities. Establishment of interdisciplinary teams among various health groups (e.g., mental health, environmental health and public health) in some regions was identified as an effective approach to service integration beyond CDCM programs.
  • The strategies used to respond to isolated and remote communities. In order to respond to the unique needs and challenges of isolated and remote communities, some FNIHB regions worked to adapt their programming and develop specific strategies to serve these communities. For example, the Saskatchewan, Atlantic and Ontario regions identified in their Transition Plans that they were adapting the Health Service Delivery Models in Remote and Isolated First Nations Communities to the needs and context of their region.
  • The alignment of CDCM programming with provincial strategies. Some regions had more strongly aligned their programming with provincial structures and priorities in terms of materials used, policies, data collection tools and sharing of information.
  • FNIHB involvement in direct service delivery and management of CDCM programming. The portion of communities that had assumed greater responsibility for the programming through flexible transfer funding varied greatly across the regions. Saskatchewan had one of the highest proportions of First Nations communities that had taken on greater control and responsibility for management and delivery of primary health services and other community health programs. Transfer of responsibility to communities allowed for greater flexibility of resources.

The ability to tailor CDCM programming to meet regional and local needs was one of the strengths of the programs. For example, FNIHB regional offices often worked with provincial partners to develop or tailor provincially developed resources, policies, guidelines and tools, particularly with respect to TB and the provincial immunization schedules and, where available, the BBSTI & HIV/AIDS protocols and guidelines.

Challenges

Significant challenges remain with respect to FNIHB’s  ability to monitor and respond to changes in needs for CDCM programming and supports required by First Nations on-reserve communities over time.  

While the flexibility of the CDCM program design was perceived to be a key strength of the CDCM programs, it is not without challenges. Variations in design and focus of the programming across the regions and communities create challenges in monitoring and planning for the Program on a national level and over time. For example, particularly challenging are the differences in data collection systems and quality of data collected in gaining overall understanding of trends, risks and factors contributing or constraining success of CDCM programs over time. Some key informants suggested that the programs were more successful in reacting to specific communicable disease related needs (e.g., responding to outbreaks), rather than proactively addressing health outcomes of First Nations living on-reserves. This may be a function of several factors, one of which is the paucity of data that would help the program better understand changes in needs over the longer term. Thus, the ongoing surveillance and reliability of data collected across the regions and communities is important in ensuring Program’s ability to act and proactively respond to the needs of communities.  

The challenges in measuring communicable disease health related outcomes for First Nations communities were not unique to this program. Literature illustrated that Canada has a serious deficit in the availability of accurate, complete and up-to-date statistical information about the health of Aboriginal populations in general, and limited data available by sub-population and geographical location.Footnote 15 According to the National Collaborating Centre for Aboriginal Health, a key challenge is inconsistencies in First Nations, Inuit, and Métis ethnic identifiers in provincial health data collected through vital registration systems, primary care and hospital administrative datasets, and acute and chronic disease surveillance systems.Footnote 16 As a result, First Nations, Inuit and Métis populations are often ‘invisible’ in health statistics. In addition, national health surveys such as the Canadian Community Health Survey commonly do not provide First Nations, Inuit and Métis specific information at the regional level and often exclude on-reserve populations.Footnote 17, Footnote 18

About one-third of key informants noted that it was difficult to comment whether the programs have had sufficient reach and impact to support behavioural change, improved vaccination rates, and reduced the incidence and prevalence of the targeted communicable diseases over time because of the limitations of the data and the lack of pre and post assessments of preventative interventions.

4.2 Relevance: Issue #2 – Alignment with Government Priorities 

4.2.1    Do the objectives and priorities of CDCM align with current Government of Canada and Health Canada priorities and strategic outcomes?

The CDCM objectives and priorities align with the Government of Canada and departmental current priorities and strategic outcomes by increasing access to health care and addressing the health status inequalities affecting First Nations and Inuit communities.

Alignment with Government Priorities

The objectives of the CDCM Program are aligned with the priorities of the Government of Canada.

CDCM programming aims to reduce the incidence, spread, and human health effects of communicable diseases, as well as improve health through prevention and health promotion activities of First Nations on-reserve communities and one Inuit region (Nunatsiavut).. These objectives are aligned with current Government of Canada priorities, namely to protect the health and safety of Canadians and their families, to work in partnership with Aboriginal peoples to create healthy, prosperous, self-sufficient communities, and to ensure better health outcomes for First Nations and Inuit. For example, the Budget 2011 included on-going “investments that seek to improve health outcomes for First Nations people and Inuit.” The CDCM Programs are also well aligned with the 2014 Speech from the Throne, which emphasized the government’s commitment to create healthy First Nations communities by stating: “…our Government will continue to work in partnership with Aboriginal peoples to create healthy, prosperous, self-sufficient communities.”

Alignment with Strategic Outcomes

The CDCM programs are aligned with Health Canada strategic outcomes. The objectives of the CDCM Programs are also aligned with Health Canada’s stated departmental priority: “Strengthen First Nations and Inuit Health Programming”. They are also aligned with the strategic outcomes: “First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status” and “Better Health outcomes and reduction of health inequalities between First Nations and Inuit and other Canadians”.Footnote 19 Health Canada’s Report on Plans and Priorities 2014-2015 (as well as the previous 2013-2014 version of the report) indicates the federal government’s commitment to address human health risks for First Nations and Inuit communities associated with communicable diseases by increasing community capacity (i.e., ability to deliver programs) to respond to these risks. FNIHB fulfills its mandate, in this context, by providing and/or funding a range of programs and services to prevent and control communicable diseases.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities  

4.3.1    Do Health Canada activities related to CDCM align with federal government jurisdictional, mandated and/or legislated roles and responsibilities?

The CDCM activities align with federal government jurisdictional, mandated and/or legislated roles and responsibilities through fulfilling various policies and authorities enabling the provision of health services for First Nations and Inuit.

Federal Mandate

In Canada, health care is a shared responsibility between federal and provincial governments. Provinces and territories provide universal insured health services (physician and hospital services) to all residents, including First Nations, Inuit, and other Aboriginal people. The federal government’s role with respect to the provision of health care services for First Nations and Inuit is based on government policy and involves supplementing programs and services provided by provincial and territorial governments.

The Constitution Act, 1867 does not explicitly include “health” as a legislative power assigned either to Parliament (in section 91) or to the provincial legislatures (in section 92).  Nonetheless, the Courts have confirmed that most aspects of the regulation of health care are within provincial jurisdiction.  For example, provinces have extensive authority over public health as a local or private matter under s. 92(16) of the Constitution Act, 1867; over the regulation of medical professions as matters of property and civil rights under s. 92(13); and over hospitals under s. 92(7).

The federal government has the power to enact legislation in relation to certain health related matters which are ancillary to other federal powers including the federal spending power (e.g., Canada Health Act) and the criminal law power (e.g., drugs, tobacco and hazardous product laws).  In addition, the federal government has power to enact health legislation based on its “peace, order and good government” powers under s. 91 of the Constitution Act, 1867 (e.g., laws pertaining to quarantine and national emergencies).  These laws would apply to First Nations on or off reserve and Inuit.  Additionally, the federal government may legislate in relation to First Nations and Inuit as a result of its jurisdiction over “Indians, and Land reserved for the Indians” in s. 91(24) of the Constitution Act, 1867.  Currently, there is no federal health legislation enacted.

Federal policies, such as the Indian Health Policy and the Indian Health Transfer Policy, outline the goals of the federal government with respect to Aboriginal health (National Collaborating Centre for Aboriginal Health, 2011). In addition, the Canada Health Act, outlines the intent of Canadian health care policy in general and describes the conditions under which provincial and territorial governments qualify for the Canada Health Transfer.

Further, the Canada Health Act (Section 3) states that Canadian health care policies are intended to ensure the physical and mental well-being of Canada’s residents, and to facilitate reasonable access to health care (Government of Canada, 1984).

The Indian Act, describes various authorities and jurisdictions related to Aboriginal health (Government of Canada, 1876). The Act states that the Governor in Council can regulate the provision of health services and medical treatment for Indians (Section 73(1)[g]). However, such regulations do not exist.  Band Councils are given the authority to create bylaws regarding the provision of health services for people who live on-reserve (Section 81(1)[a]). The Indian Act does not apply to Inuit.

Finally, the federal government also has a mandate to address emergency issues related to public health in First Nations communities. Currently, there are two statutes that grant the federal government authority to prepare for, and act, in the context of a public health emergency: the Emergencies Act and the Emergency Management Act. Nevertheless, the Auditor General’s 2013 report noted that responsibility among stakeholders for emergency management on reserves is not clear, and recommended that the federal roles and responsibilities be further clarified. A lack of formal agreements that clearly outline roles and responsibilities for emergency management on reserves was noted as a major issue.

Federal Role

The 1979 Indian Health Policy is based on the three pillars of community development; the traditional relationship of the “Indian people” to the federal government; and a single interrelated Canadian health system, consisting of federal, provincial and community-based elements.

The Indian Health Policy was adopted on September 19, 1979, during a period of transition for FNIHB. Previously, FNIHB had more involvement in providing direct health care services to First Nations and Inuit communities. However, FNIHB began shifting toward helping communities gain more direct control over community-based health services. The Policy aims to improve the health status in First Nations and Inuit communities “through mechanisms generated and maintained by the communities themselves” (National Collaborating Centre for Aboriginal Health, 2011, p. 23). The Indian Health Policy (1979) also recognizes the interdependent nature of the Canadian Health system and identifies that two of the most significant federal roles are in public health activities on reserves and health promotion (Health Canada, 2014).

The Policy describes the importance of reinforcing relationships between multiple levels of government and increasing the capacity of Aboriginal communities to “play a positive and active role within the Canadian health care system” (National Collaborating Centre for Aboriginal Health, 2011, p. 23). As a result of the Policy, the federal government no longer directly provides services to First Nations people in the Northwest Territories, nor to the four Inuit regions:  in Nunavut, the Inuvialuit Regional Settlement in the Northwest Territories, Nunavik in Quebec, and Nunatsiavut (with the exception of TB programming) in Newfoundland and Labrador. Instead, federal government funding flows through transfer agreements with territorial governments and self-governments/land claim agreements.

The federal roles in the area of public health protection are in the domain of public health activities on reserves, health promotion, and the detection and mitigation of hazards to health in the environment. Several government departments such as Health Canada, the Public Health Agency of Canada and AANDC are involved in the delivery of public health and emergency management for on-reserve First Nations and Inuit communities. FNIHB is the main body within Health Canada responsible for the delivery of public health and health promotion services on-reserve and in Inuit communities. The Minister of Health has no specific statutory obligation to provide health programs and services to First Nations and Inuit. Nonetheless, the Minister of Health has general authority over matters relating to the promotion or preservation of health of the people of Canada under section 4 of the Department of Health Act, votes under the Appropriation Act authorizing FNIHB spending and authority from Treasury Board for specific program activities.

The primary mandate of FNIHB is toFootnote 20:   

  • ensure the availability of, or access to, health services for First Nations and Inuit communities;
  • assist First Nations and Inuit communities address health barriers, disease threats, and attain health levels comparable to other Canadians living in similar locations; and
  • build strong partnerships with First Nations and Inuit to improve the health system.

The activities of FNIHB are divided into three categories: Primary Health Care; Supplementary Health Benefits; and Health Infrastructure Support. CDCM is an activity of Public Health and Promotion, which forms part of the Primary Health Care activity.  Primary health care programming encompasses health promotion and disease prevention programs which work to improve health outcomes and reduce health risks. They also include public health protection, including surveillance, to prevent and/or mitigate human health risks associated with communicable diseases and exposure to environmental hazards; and, primary care, where individuals are provided diagnostic, curative, rehabilitative, supportive, palliative/end-of-life care, and referral services.

4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)

4.4.1    To what extent have the immediate outcomes been achieved?

Immediate outcome #1: Improved public awareness and knowledge of prevention and control measures as well as communicable disease health effects among First Nations on-reserve.

CDCM programs have made progress in improving public awareness and knowledge of communicable diseases. Alignment of regional initiatives allowed leveraging with provincial activities and resources. However, the reach and impact of such activities was uneven across the regions and communities and constrained by factors such as competing health priorities with limited human and financial resources.

Increased Knowledge and Awareness

Some progress has been made in increasing knowledge and awareness of the risks, prevention and treatment of communicable diseases among First Nations living on-reserve. Community representatives and regional coordinators somewhat agreed that the CDCM programs made progress in increasing awareness of the benefits of immunization  and agreed that they were increasing knowledge of BBSTIs and HIV/AIDS.  They also agreed that programs were increasing awareness and promoting better understanding of TB, and most agreed they were educating community members about CDEs.  

During the interviews, most of Health Canada headquarters representatives and the majority of FNIHB regional staff  confirmed that CDCM awareness and public education activities had some success with respect to improving awareness and knowledge of communicable diseases in the communities, attributing their views mostly to anecdotal evidence and the stability or decrease of incidence rates in some communities. Some FNIHB national and regional representativesFootnote 21 noted that very few comprehensive studies were done to provide tangible evidence of the level and change in knowledge and understanding of communicable diseases among First Nations on-reserve communities.  

The literature review found two studies completed in recent years which directly measured the perceptions and attitudes of First Nations on-reserve communities. In 2011, Health Canada commissioned public opinion research which involved telephone interviews with a representative sample of 659 First Nations individuals living on-reserve and 162 Inuit in one Inuit region (Nunatsiavut)., who are parents or caregivers of children up to six years of age, or expectant parents.Footnote 22 The results of this research revealed a reasonably good understanding of the importance and benefits of childhood vaccination. Almost all caregivers reported having had their child vaccinated, and were confident that their vaccination status was up-to-date. However, concerns about overall safety and side effects of vaccines were fairly widespread, as were perceptions about the potential for vaccines to cause disease and the potential for traditional practices to eliminate the need for vaccination altogether.

In 2012, public opinion research was conducted to measure awareness, knowledge, attitudes and behaviours related to HIV/AIDS.  The report concluded that overall perception of the seriousness of HIV/AIDS was relatively the same for Aboriginal Canadians, including First Nations, Inuit and Métis as it was for the Canadians population overall; however, some differences in opinions were noted. A comparison of survey results between Aboriginal groups (First Nations living on- and off-reserve, Métis, and Inuit) found that First Nations living on-reserve and Inuit were more likely to be unaware of some facts about HIV/AIDS. Both First Nations living on-reserve and Inuit were more likely to hold stigmatizing views toward people living with HIV/AIDS, and were generally less likely to be comfortable in many of the situations tested where they might encounter a person with HIV/AIDS.Footnote 23 When asked, as part of the evaluation survey, to rate the progress made by BBSTIs and HIV/AIDS program activities in reducing stigma and facilitating acceptance and support for First Nations people living with diseases, community health representatives and regional coordinators provided reported significant progress. Both groups noted that stigma remained a challenge in some communities.

Development and Implementation of Effective Prevention Measures

The CDCM programs supported the development and implementation of large numbers of prevention awareness activities and educational products. Over the period covered under this evaluation, all CDCM programs developed and disseminated a large number of prevention marketing activities, promotional materials and products to educate and create awareness of communicable disease prevention strategies. Educational products and activities implemented across each of the program areas included: 

  • Tuberculosis. FNIHB implemented the TB Social Marketing campaign, which incorporated different media to reach communities as well as health care providers. In 2009-2010, as part of the campaign, 53,000 copies of an illustrated storyline were distributed to target First Nations communities. The storyline utilized illustrated cartoons to depict scenarios where First Nations and Inuit individuals receive TB testing, treatment and are cured of TB.Footnote 24 The products and activities that the community health representatives reported most commonly using included brochures, pamphlets, books and posters (77%), one-on-one education and counselling support (54%), educational resource materials and teaching resources (48%), and workshops (38%).  TB awareness efforts were more likely to be delivered in communities with high incidence rates.
  • BBSTI and HIV/AIDS. According to the FNIHB  Performance Measurement Systems reporting templates, FNIHB disseminated 46 different BBSTI and HIV/AIDS related information products and materials. The community health representatives surveyed most commonly used brochures, pamphlets or booklets (87%), workshops on HIV/AIDS (76%), educational resource materials (69%) and distributed condoms, needles, sharp kits and straws (68%). Various materials were distributed to an estimated 28,700 individuals from the intended target populations and risk groups and a total of 2,798 community and educational events were hosted across the country, which were estimated to have reached over 150,000 First Nations individuals. The information and education campaigns were most successful in reaching out to youth, health care workers, and women.
  • Immunization. Regions reported developing and disseminating a total of over 80 awareness campaigns, promotional activities, or culturally-sensitive training materials. The First Nations and Inuit National Immunization Social Marketing Campaign was developed in 2010-2011, and, over the course of three years, distributed a large number of products (more than 40,000 guides, 2,500 posters, 1,000 flip charts and 19,000 calendars were distributed to communities) and received over 100,000 clicks and 69,000 web views and collected 17,000 fans on social mediaMost community health representatives surveyed reported using strategies which included in-person and one-on-one services for parents and community members, brochures, pamphlets or posters, follow-up phone calls and reminders.
  • Communicable Disease Emergencies. Most CDE awareness activities included developing, updating and testing community pandemic plans in First Nations communities. The plans were developed through regular meetings and in collaboration with community leaders and other stakeholders such as Regional Health Authorities, bands and Aboriginal organizations such as the Assembly of First Nations, and tested through mock scenarios, table-top exercises, and meetings with stakeholders and community leaders. Community health representatives reported that, to promote awareness on CDEs and pandemic planning, the Program used public education and awareness materials (e.g., hand washing decals), workshops and conferences and other resources and templates.

The most effective awareness and educational campaigns were those that were culturally-sensitive and used appropriate materials, were targeted to address specific community needs,  focused on early engagement of community leaders and other partners, were aligned with provincial strategies, where available, and were community driven. The national campaigns provided information to community health staff and nurses and ensured consistent messaging. 

Reach and Impact

The reach and impact of educational activities varied across regions, communities and target groups.

The number of communities that reported delivering at least one educational or awareness related activity varied across years, regions and program components. This was in part due to the varying needs to respond to communicable disease outbreaks or priorities that were community-specific and so the following information should be interpreted within that context. CBRT reports showed that the percent of the communities delivering TB related educational activities varied from 13% to 54% across regions in 2011-2012 and from 18% to 56% in 2012-2013. The communities reporting delivering BBSTI and HIV/AIDS related activities varied from 56% to 75% across regions in 2011-2012 and from 64% to 86% in 2012-2013; and immunization related activities varied from 50% to 71% across regions in 2011-2012 and from 54% to 61% in 2012-2013. Finally, the percentage of communities that reported delivering at least one education or awareness raising activity related to pandemic planning varied from 21% to 50% across regions in 2011-2012 and from 17% to 59% in 2012-2013.

The majority of regional staff interviewed reported that, while educational initiatives and activities had some impact, they were limited in reaching the most vulnerable groups (those in remote communities, high risk groups). However, they were successful in directly debunking misinformation about vaccines, and reducing stigma, for example. Of those who said that activities had little impact, most noted that broad and nationally-developed educational materials reached those who were highly functioning and had access to the internet and health information. According to these regional staff and some community health representatives, what was needed were more personal (one-on-one) and comprehensive health strategies and campaigns. Limited time (due to competing priorities) and insufficient financial and human resources for community health nurses to engage in educational activities, addressing unique circumstances in the communities, were other factors identified by regional staff as limiting the reach of programming.

Immediate Outcome #2: Increased workforce skills, knowledge and/or expertise in CDCM.

Overall, most community health staff were satisfied with training, information, data and support provided to build knowledge and skills. Although training opportunities were provided, access varied considerably across regions and many regions were reliant on provincial activities. Investment in community health staff was perceived to be the most important factor contributing to the overall success and continuity of the CDCM programming.

Building Knowledge and Skills of Community Health Staff

FNIHB regional offices worked to build the capacity of community health staff, sometimes in collaboration with the provincial government.  These efforts were perceived to be effective in building knowledge and expertise related to CDCM.

Training needs were addressed regionally. FNIHB regions commonly assessed and delivered various training and capacity building activities periodically based on identified needs. Regional coordinators reported identifying and addressing training needs through phone and email communications and during various face-to-face meetings with community health nurses. Identification of particular gaps in knowledge occurred during working groups or steering committees related to provincial or national initiatives, or during outbreaks or other emergencies. In some regions, FNIHB worked with provincial partners to identify, design and deliver capacity building activities in on-reserve communities.

According to regional staff, most FNIHB regions did not have a formal training program related to CDCM although they typically provided orientation for new community health staff. Three-quarters of community health representatives reported having received training provided by FNIHB or other partners, and the majority reported having participated in conferences, presentations, symposia or workshops focused on specific issues or subjects relevant to CDCM programming.  Representatives from two regions indicated that community health staff regularly participated in training provided by the province. Overall, community health representatives were satisfied with the training, information, data and research received (an average rating of 3.6, where 5 is very satisfied). The knowledge and skills gained through capacity building activities were most commonly used to keep community health practices up-to-date, share and implement effective approaches and techniques, more effectively use surveillance data and health studies to plan and manage programs, and build partnerships. Most coordinators agreed that the training and other supports enabled community health staff to improve their knowledge, skills, or ability to perform their job, particularly in communities with high turnover. 

Impacting Program Success through Increased Training and Support

Many regional and community representatives indicated a strong need to increase the level of training and other support provided.

Most Health Canada and regional representatives agreed that the successful delivery of CDCM programming and continuity in services was ultimately dependent upon the competencies and capacities of community health staff.  They suggested that additional effort and investments were needed to better enable community health nurses to deal with growing workloads, competing health priorities, and increased requirements and, focus monitoring and reporting on health outcomes. Reporting requirements on CDCM related health outcomes and activities had increased in recent years (e.g., introduction of BBSTI and HIV/AIDS surveillance data, revisions to Performance Measurement Systems indicators). These requirements had increased the need to build capacity by improving systems (providing technology and implementing more effective data collection systems) and expanding training related to performance and surveillance data collection and reporting.  A few community health representatives noted that more support is needed to increase the use of technology to collect and record information.

Most regional staff indicated that limited resources and recent travel restrictions in the last year (2013-2014) in some regions, had weakened training and capacity building efforts for some communities.  Teleconferences and webinars were now the most common avenues of training provision; however, the number of community health nurses participating in these activities had been low in some regions. The majority of regional coordinators and some community representatives suggested that consistent hands-on, face-to-face training and support is necessary to ensure greater numbers of community health staff across the regions were able to participate. Some regional offices reported that they were restricted from hiring external experts to deliver the training and capacity building, which resulted in reduced training quality.

Immediate Outcome #3: Service delivery is increasingly informed by surveillance data, evidence based policy, standards, guidelines and frameworks.

FNIHB’s access to surveillance data on health outcomes has improved and was utilized to varying degrees to support policy development and program planning.  However, differences in key data sources as well as the quality and reliability of surveillance data across programs and regions constrained the utility of the information.

Surveillance Data Collection and Use

Some improvements were made in the collection, reporting and accessibility of surveillance data and health outcomes for First Nations peoples living on-reserve for all program areas. However, differences in the key data sources as well as the quality and reliability of surveillance data across programs and regions constrained the utility of the information.

The CDCM Programs made some progress in improving the collection and reporting of surveillance data across all programming. The progress, as well as the challenges and limitations associated with each program include:

  • Tuberculosis. Program coordinators reported significant progress in improving the collection, reporting and use of TB surveillance data.  This was largely the result of a 2012 Memorandum of Understanding (MOU) signed between the Public Health Agency of Canada, FNIHB’s Communicable Disease Control Division and Surveillance Health Information and Policy Coordination Unit, which resulted in departmental access to more timely and detailed data on TB for on-reserve First Nations. However, FNIHB was not able to share this surveillance information with external partners (i.e., via inclusion in policy documents) due to the restriction of the use of data (data is for FNIHB internal use only) as per the data sharing agreement with PHAC.
    Other improvements in this area included revisions to FNIHB TB reporting templates which now collect and report information on Latent TB Infections. As a result, in 2012 an estimate for Latent TB Infections prevalence among contacts in First Nations on-reserve communities was produced for the first time.
  • BBSTI and HIV/AIDS.  All BBSTI and HIV/AIDS data for FNIHB was collected through the program monitoring reporting templates submitted annually by the FNIHB regions. The reporting template was piloted in 2012 for the 2011 data. Prior to 2012, FNIHB relied on Public Health Agency of Canada published reports for any estimates of HIV among Aboriginal populations (the data was not broken down by First Nations population). Although FNIHB had gained access to some national level Public Health Agency of Canada HIV data, this data cannot be shared or cited in policy documents. Availability of this type of data was dependent on the inclusion and completion of an on and off-reserve First Nations identifier on the case reporting forms, and the accessibility of these data sources to FNIHB regional offices.
    As FNIHB was not part of the formal notifiable disease reporting system, there was no obligation for provinces to share this data with FNIHB regional offices. Consequently, access to and availability of data varied by region and by disease (e.g., many regions did not have access to HIV test report data). Some regions continued to face barriers to accessing provincial or Regional Health Authorities data due to concerns about privacy, particularly in small communities. Community health representatives and regional coordinators reported progress with respect to improving the collection, reporting and use of BBSTIs and HIV surveillance. Regional coordinators attributed the progress made to the development of data sharing agreements with provinces and Regional Health Authorities, but suggested that the lack of First Nations identifiers in data collected by partners, and limited access to data, continued to be the most significant barriers.
  • Immunization. The main source of immunization surveillance data for on-reserve First Nations communities was community reporting templates prepared by FNIHB regional offices. The data was consolidated by the national office and shared with the regions in the form of summary reports. Over the period covered under this evaluation, progress was made in improving the quality and amount of immunization surveillance data reportedby both community health representatives and coordinators. Regional coordinators and community health representatives attributed the progress to the use of electronic data systems to collect and report on community health records. Some key informants and regional representatives identified Panorama, the electronic public health record system, as a system which had been increasingly used in some communities and could, if well integrated across First Nations communities, significantly improve surveillance collection and reporting on immunization. However, coordinators and community health representatives identified insufficient staffing, high staff turnover, paper based data collection in some communities, and an inability to share data with provincial partners for individuals that receive health services off-reserve, as factors that limited further progress.
Increased Use of Data Supporting Policy and Programs

Overall, improved community access to data and availability of health reports led to increased use for policy development, planning and management purposes (e.g., developing of targeted responses). However, the use of data and data products continues to be constrained by a number of factors such as the quality, timeliness and completeness of the data and the capacities of First Nations communities to interpret and undertake actions based on the reliable evidence. 

As described in the following paragraphs, surveillance reports, health status reports, and other research was produced and used by national, regional and community representatives to develop policies and guidelines, plan and manage programs, and deliver services.

In 2013, for the first time, the FNIHB national office created and provided regional offices with a set of Communicable Disease Summary Surveillance Tables containing detailed regional data for each program area that was used to inform program priorities and planning. At the national level, performance measurement information, data, and evaluation reports were used for policy development, reporting on program outcomes, setting or revising program priorities, and reporting to Treasury Board and the Federal Initiative to Address HIV/AIDS. Most Health Canada representatives interviewed reported using performance data to understand the context, issues, gaps and weaknesses, and determine priorities.

During the period covered under the evaluation, the CDCM Programs developed or initiated the development of a number of policy standards, guidelines and frameworks at the national level. These included Health Canada’s ‘National Strategy against Tuberculosis for First Nations On-Reserve’, the ‘National Framework for Addressing BBSTIs in First Nations On-Reserve’, currently being developed through extensive consultations with stakeholders and First Nations communities, Health Canada’s national ‘Infection Prevention and Control Action Plan for First Nations Communities’, and a wide range of guidelines and templates.

FNIHB regional offices reported, via the Performance Measurement Systems reporting templates, developing a large number of health research and evaluation reports for each program, relevant to their needs and priorities. Much of the research and reports focused on identifying needs related to community emergency responsiveness and evaluations of H1N1 response.

At the regional level, the surveillance data and other research and evaluation reports were used to identify priorities, plan, and focus activities on specific areas of need (e.g., implement targeted responses). About half of regional staff members noted that their region had access to communicable disease surveillance data and evaluation reports. Others noted that there was limited access to data, highlighting gaps in the existing data (e.g., not all communities reported data), a lack of First Nations identifiers, and a lack of data on certain programs (e.g., lack of BBSTI and HIV/AIDS First Nations identifiers in Public Health Agency of Canada data).  FNIHB regional offices reported the development of a number of policy standards, guidelines and frameworks. They often worked with provincial partners to develop or tailor provincially developed resources, policies, guidelines and tools, particularly with respect to TB and the provincial immunization schedules and, where available, the BBSTI & HIV/AIDS protocols and guidelines. 

Community health representatives somewhat agreed that, at the community level, they had access to surveillance data, research and evaluation reports needed to plan activities and effectively deliver CDCM programming. According to the Community Based Reporting Template (CBRT) reports, 57% of communities delivering CDCM programs reported receiving a Health Status Report on communicable diseases in 2011-2012 and 61% reported the same the following year. FNIHB remained the major source of data and research. CBRT reports indicated that a majority of communities (78%) received reports directly from FNIHB, although 64% also reported receiving information from the provincial government or Regional Health Authorities. Over half of community health representatives surveyed reported reviewing surveillance data on communicable diseases relevant to the target populations, and 43% reviewed research reports and other publications relevant to the delivery of CDCM programs.

All Health Canada representatives and the majority of regional representatives reported that they had access to the operational policies, guidelines, practice manuals and tools needed to deliver the CDCM Program effectively. Similarly, most community health-care representatives reported using the CDCM related policies, guidelines, practice manuals and other tools to a great extent (54%) or some extent (29%).

Overall, the research, policies and guidelines were perceived to have filled gaps and reflected the current state of knowledge. However, some regional and community health representatives suggested that there were gaps related to inconsistent adherence and implementation of policies, guidelines and manuals among communities. This was due to a variety of factors such as insufficient support to ensure resources are up-to-date or to adapt resources to the needs of the community, a lack of oversight over transferred communities, and insufficient knowledge of the policies and guidelines among the band councils responsible for health management.

Other gaps were identified with respect to the availability, access and use of data, particularly with respect to BBSTIs and HIV/AIDS (e.g., limited sharing of data due to privacy concerns). Use of the surveillance and performance data at the regional and community levels could be constrained by the timeliness of reporting (e.g., some surveillance reports take two years before they reach communities) as well as competing community priorities (e.g., some communities had other pressing social/economic and environmental priorities which took precedence over CDCM at the time). Further constraints included the capacity of community representatives to interpret and make decisions based on surveillance data, and the reliability of the surveillance data reported (e.g., incomplete data was not sufficient to make decisions). According to the majority of regional staff, the lack of surveillance data created problems in identifying the needs of communities, planning and targeting programming, properly allocating resources, measuring impacts and improvements, and/or addressing health issues and gaps in the communities. A few suggested that it was much more difficult to engage community leadership, create partnerships, and leverage resources without sufficient data.

Immediate outcome #4: Increased community and stakeholder engagement and collaboration in policy/program development and delivery.

Significant progress has been made in terms of building collaboration and partnerships at the national, regional and community levels to share information and achieve integrated service delivery. However, engagement and collaboration at the community level was more challenging due to staff turnover in some communities, changes in leadership, competing community priorities, and the political environment.

Partner Collaboration and Engagement

Significant progress was made in increasing collaboration and engagement with partners at the national level, which in turn helped clarify roles and responsibilities, increased information sharing, and supported policy development.

The healthcare system for Canadian First Nations communities is complex, and involves numerous other federal departments and national organizations.

Health Canada and FNIHB engaged federal departments and national organizations through participating in various horizontal initiatives (e.g., Federal Initiative to Address HIV/AIDS, Preparedness for Avian and Pandemic Influenza Initiative (AI/PI)); working groups, national meetings and committees (e.g., Trilateral First Nations Pandemic Committee with the Public Health Agency of Canada and Assembly of First Nations). FNIHB was also involved in the development of work plans and communication and information protocols in an effort to ensure a coordinated approach to addressing the specific needs of First Nations communities (e.g., joint work plan with AANDC entitled Work Plan Emergency Management and Pandemics, Communications Protocol: Working Together on H1N1 Preparedness with AANDC and Assembly of First Nations). 

According to key informants, as a result of these and other activities, significant progress was made in facilitating collaboration, coordination and integration between FNIHB and other national stakeholders, particularly the Public Health Agency of Canada, AANDC, the Canadian Institute for Health Research and national organizations including Assembly of First Nations, Inuit Tapiriit Kanatami and the Canadian Aboriginal AIDS Network. Key informants reported a number of activities undertaken to facilitate collaboration, coordination and partnership development which had notably improved CDCM policy and program delivery for First Nations on-reserve communities including:

  • Increased access to data.  Notwithstanding the continuing challenges associated with the use and sharing of information accessed through the Public Health Agency of Canada, the MOU  signed between Public Health Agency of Canada and FNIHB on information sharing was an important step towards increasing the ‘visibility of First Nations health outcomes’ in data collected by the Agency and ultimately by provincial governments. The Public Health Agency of Canada worked with provincial partners to improve access and sharing of data.
  • Clarified roles and responsibilities. Signing of an MOU between Health Canada and AANDC helped clarify the roles and responsibilities of the departments, and improved the coordination of pandemic response activities for First Nations on-reserve communities. Greater clarity around roles and responsibilities helped to avoid duplication and overlap among national stakeholders.
  • Increased preparedness to respond to pandemics.  Close collaboration between Health Canada and  First Nations and Inuit organizations such as Assembly of First Nations and Inuit Tapiriit Kanatami, helped to respond to the H1N1 outbreak. Health Canada representatives reported that greater collaboration with AANDC helped to achieve closer working relations between the two departments on the community emergency preparedness file; increased communication and collaboration with Assembly of First Nations led to better integration and a more responsive approach to community health emergencies.

Despite the progress made, some Health Canada program representatives suggested that continued coordination and collaboration with partners and further clarification of roles with respect to surveillance data is needed. For example, the need for strengthening the role and leadership of Surveillance Health Information and Policy Coordination UnitFootnote 25 with respect to developing a national surveillance strategy, collecting and analysing surveillance data and better defining the roles of the national and regional offices.

Engagement with Provincial Partners

At the regional level, FNIHB’s collaboration and engagement with provincial partners increased.  In most regions, this resulted in improved and better integrated program delivery.

According to the literature and document reviews, each provincial government operated a communicable disease program, which encompassed surveillance, IPC, pandemic planning, and some level of information-sharing and awareness building for residents. In addition to provincial governments and regional/district health authorities,  numerous stakeholders were involved in health promotion and delivery of health care services for First Nations living on-reserve and off-reserve including tribal councils, regional First Nations associations, emergency management agencies, and organizations involved in HIV/AIDS programming and support.

According to the document review, over the past five years and particularly over the past two years, each FNIHB region implemented a large number of formal contribution agreements, MOUs, formal and informal collaborative partnerships, working groups and committees in an effort to increase collaboration and better integrate program service delivery. According to the Performance Measurement Systems reports, almost all collaborative strategies implemented by   FNIHB regions involved  engagement of provincial partners in development of common educational activities, development of new policies, and facilitating data sharing. For example:

  • Tuberculosis: The Manitoba FNIHB regional office collaborated with the province and other stakeholders in the development of the Manitoba TB Strategy, three regions developed formal TB-related MOUs with provincial governments, and five regions identified approximately 30 TB related working groups.
  • VPD & Immunization: Four regions identified strategic alliances with other organizations such as educational institutions, Tribal Councils, associations, and health and wellness organizations. In 2011-2012 and 2012-2013, regions reported participating in over 20 committees and regional working groups.
  • BBSTI and HIV/AIDS: All regions reported a number of formal collaborative agreements to share data, integrate program and service delivery, and develop strategic alliances with other organizations at the regional level. For example, the Alberta, Saskatchewan, and Atlantic regions each reported formal collaborative agreements with Aboriginal AIDS Service Organizations, and the Atlantic, Ontario and Québec regions reported formal collaborative agreements with First Nations Provincial-Territorial Organizations.
  • CDE: In 2010-2011, five regions reported 262 contribution agreements developed with their respective communities and over 90 linkages and committees with emergency preparedness response and pandemic influenza program staff at the regional, provincial and national level. In 2011-2012, three regions identified formal CDE Interdepartmental Letter of Agreement and MOUs with other federal and/or provincial government entities and five regions identified at least one working group or committee related to CDE.
  • Infection Prevention and Control: All regions reported participating in key IPC working groups and other committees and networks. Examples included: the Ongoing Regional Infection Prevention and Control Team, which met at least 8 times per year to direct ICP activities; the Regional Infection Prevention and Control Task Group with representation from across FNIHB, which met at least 4 times per year; the Technical Advisory Group on Infection Prevention, a provincial group that provided advice, developed and reviewed guidelines; the Respiratory Protection Task Group; the Regional Emergency Preparedness Committee; the IPAC Steering Committee implemented in 2009 to develop IPAC practices; and, the CDE Sub Working Group.

Most regional representatives reported that significant progress was made in facilitating collaboration and networking between FNIHB and other stakeholders, particularly provincial partners. They noted that alignment with Regional Health Authorities and provincial structures, policies and priorities was important for strengthening health services (e.g., increasing access and use of data, improving consistency and responsiveness of health services and integrating programming). Provincial designation of the Medical Officer of Health in some regions allowed for more authority over reporting, surveillance, enforcement when required, and better collaboration with the province.

According to the document review and interviews, increased coordination and integration with partners expanded the reach to key target populations (e.g., individuals with mental health and addictions, those with co-morbidities, and youth), improved access to testing, treatment and counselling services for HIV/AIDS, and fostered a more integrated or holistic approach to health service delivery. For example, in Ontario, partnership efforts resulted in enhanced opportunities to support standardization in HIV/AIDS testing and counselling practices. Greater information sharing and integration with provincial partners was reported to be particularly important for ensuring consistent services for individuals who received services on and off-reserve.

The extent of efforts to work with other organizations towards more coordinated and integrated responses varied across programs. Regional coordinators ranked most initiatives from some extent to a great extent for TB, VPD & Immunization and CDE, BBSTI and HIV/AIDS,. They noted that collaboration on the BBSTI and HIV/AIDS file is an on-going effort and more work is needed to increase information sharing and reduce stigma. 

These collaborative efforts also helped to clarify roles and responsibilities and reduce duplication and overlap of programs and services. The majority of regional representatives agreed that the FNIHB CDCM programs were well coordinated with, and complementary to, other regional programs, suggesting that the roles between the federal and provincial governments were clear and, that increased efforts were made to better integrate the FNIHB CDCM programs with provincial programs.

Engaging Community Leaders

The CDCM programs were somewhat successful in engaging community leaders in program activities. Engagement with community leaders was more challenging due to frequent changes in leadership, competing priorities, high turnover of nurses in some communities, the political environment and recent travel restrictions beginning in 2013-2014.

About half of regional representatives agreed that CDCM programs were successful in engaging community leaders on issues related to communicable diseases. Others noted that engagement of community leaders varied significantly across communities and programs. Regional coordinators generally perceived their respective programs to be somewhat successful in engaging community leaders and collaborating and networking with other organizations at the local level, but noted that the level of engagement depended on the community health staff’s ability and capacity to work directly with leaders to establish effective professional relationships. FNIHB regional representatives engaged with community leaders through face-to-face meetings, conferences, workshops, phone calls, and correspondence and participated in working groups, committees and consultations with community leaders.

Recent travel restrictions that began in 2013-2014, particularly for communities that do not have good access to technology or the internet, undermined the regions’ ability to facilitate communication, collaboration and engagement of community leaders. Other challenges included competing priorities for community leaders, limited resources and funding (including the prevalence of part-time community health staff), high turn-over, limited understanding about communicable diseases and their impact on public health amongst leaders, and  changes in community leadership. 

Perceptions of community health representatives with respect to CDCM programs in engaging community leadership varied across regions. Those who perceived programs to be less successful suggested that more time and resources were necessary to build relationships, educate community leaders about the importance of CDCM programs and engage them through formal and informal meetings and discussions in the design and delivery of health programs overall.

Regional representatives suggested that FNIHB should follow a more holistic approach to healthcare. This would involve engaging leaders in the early stages of the planning and implementation processes and on multiple health issues simultaneously, and provide further education, training and resources to facilitate their engagement. More capacity building was specifically requested for First Nations’ elders related to BBSTI and HIV/AIDS in communities that face significant challenges with stigma.

4.4.2 To what extent have the intermediate outcomes been achieved? 

Intermediate Outcome #1: Increased use and uptake of communicable disease prevention and control measures.

CDCM programs contributed to increased uptake of communicable disease prevention and control measures by implementing prevention and disease control strategies. These strategies resulted in increased staff knowledge and skills for improved health care delivery within First Nations communities.

Prevention and Treatment Measures

CDCM programs contributed to high levels of prevention and treatment measures related to the FNIHB-targeted communicable diseases.
  
Some of the factors that influenced the level of personal and public measures taken to prevent and control communicable diseases included knowledge and awareness of preventative measures (e.g., the importance of immunization), access to testing and treatment services, and reduced stigma. The findings of the evaluation with respect to increased use and uptake of communicable disease prevention and control measures were based partially on the success of the programs in addressing the factors noted above, as well as the available (however limited) data and the perspectives of those interviewed and surveyed. Some of the information presented below serves as a proxy of the Program’s contribution towards increased prevention and control measures and should be interpreted with caution.

  • Data collected via FNIHB’s performance measurement templates suggested immunization coverage rates have remained relatively stable in most regions and increased for some vaccines in the last few years. For example, MMR rate for children in on-reserve communities remained high and in some cases increased, and was similar for other CDCM targeted diseases.Footnote 26
  • While the data presented above provided some indication of the relative uptake of the immunization rates for specific vaccines in some regions, there were many limitations of the immunization surveillance data and consequently an inability to provide a comprehensive review of the coverage rates among First Nations on-reserve communities. Some of the limitations include variability of immunization schedules across regions and provinces and the absence of a national immunization strategy, variations in the proportion of communities that report immunization coverage data to the region in any given year, and differences in immunization data collection and reporting mechanisms across FNIHB regions.
  • As some First Nations individuals living on-reserve may receive their vaccines off-reserve, the coverage data reported by FNIHB may be underestimated. Community health representatives and regional coordinators report some progress in improving immunization coverage rates  due to program efforts. However, according to regional coordinators, the improvements are inconsistent and not systematic across communities and regions. Limited community capacity (i.e., ability to manage the delivery of  CDCM programs) and resources, competing priorities at the community level, under-staffing and/or high turnover among community health nurses, inadequate data collection and tracking, and persisting myths and biases against vaccines are among the factors constraining the uptake.
  • Active TB treatment rates in on-reserve communities (2009 and 2010) were similar to overall Canadian TB treatment rates. According to the Public Health Agency of Canada pre-release reports, 86.3% of reported active TB cases completed treatment in 2009 (13.7% of cases were cured with negative culture at the end of the treatment); and 90.5% of reported active TB cases completed treatment in 2010, of which 39.6% were cured with negative culture (data includes BC). The rates were comparable to the total number of TB cases in Canada for which the treatment outcome was available (within a percentage point) in 2009, and better outcomes were reported for First Nations on-reserve in 2010.Footnote 27
  • Latent TB Infection was an important factor in the management and control of TB, given its potential to develop into active disease. Over the period covered under the evaluation, most regions started implementing enhanced screening programs for early detection and management of Latent TB Infections in children. The intensity of targeted or enhanced screening programs varied based on the local epidemiology of TB.
  • Regional coordinators and community health representatives reported that the TB Program made progress in detecting and diagnosing TB infections among those exposed and in completing treatment of those with active TB. The Program utilized targeted testing, awareness and treatment activities in communities with high incidences of TB and contact investigation to identify possible new cases. The efforts to treat active TB cases included one-on-one services by health care representatives and/or directly observed therapy workers and building the skills and capacities of health care staff to implement early detection. Low levels of staffing training combined with high levels of staff turnover and poor understanding of TB in some communities were some of the factors that constrained the progress.
  • According to the First Nations Regional Health Survey, half of all First Nations adults who are sexually active reported having been tested for an STI and 41% have been tested for HIV/AIDS. Women were more likely than men to be tested (58% vs. 44% for STI and 50% vs. 33% for HIV). The survey reported that approximately one-fifth of First Nations adults who reported being sexually active reported “always” using a condom. The reason given most commonly for not (always) using a condom was having a regular partner (60%).Footnote 28 According to CBRT reports, most communities reported having access to HIV testing (84% in 2011-2012 and 85% in 2012-2013), and a majority report having access to HIV treatment services (66% in 2011 and 58% in 2012-2013) on or near reserve.
  • Regional coordinators and community health representatives suggested that the BBSTI and HIV/AIDS Program made some progress towards increased access to testing for HIV and progress in increased access to education for those vulnerable to/living with HIV. Access to HIV testing increased through provision of transportation in remote communities, recruitment of sexual health nurses or doctors, creation of on-site testing facilities, and referral systems between the local family health team and a specialist.
  • Access to education for those vulnerable to and living with HIV was strengthened through partnerships with provincial programs, communities and grassroots organizations that provided similar services and increased focus and attention on HIV/AIDS prevalence among First Nations peoples. Stigma and privacy concerns, limited human and financial resources, insufficient training of health representatives in the communities, and the failure of some communities to recognize HIV/AIDS as a significant issue affecting their community members were identified as factors limiting further progress.
Strategies to Increase Uptake

Various strategies were utilized to increase uptake of communicable disease prevention and control measures such as working with provincial and local partners, facilitating access to surveillance data, and implementing program-specific approaches such as targeted TB testing, increasing privacy of testing for BBSTIs and HIV, and use of effective incentives to increase immunization.

Community health representatives used a variety of techniques and strategies to increase uptake of communicable disease prevention and control measures. The document review and input from regional and community health representatives suggested that some strategies are important for all programs. For example, working with provincial and local partners ensured consistent messaging and helped reach out to the most vulnerable target groups. Increased access to surveillance data enabled programs to their understanding of trends in the uptake of prevention measures identify needs and develop targeted responses. Uptake also benefited from building awareness through local media (TV, local radio) and other avenues that were culturally sensitive and appropriate to the targeted age groups. For example, community events were appropriate for raising awareness among the elderly, in-person workshops were effective for building community awareness and delivery of programs, and presentations in schools and use of social media were more effective for youth.

In addition, various program-specific strategies were implemented effectively including: 

  • Strategies that increased immunization uptake, including those that included community health representatives who developed closerelationships with parents in the pre- and post- natal stage. Community health representatives used incentives and appointment reminders such as calendars and mugs with the provincial immunization schedule.
  • Strategies that increased testing and support for BBSTI and HIV/AIDS included improved privacy for on-reserve testing and treatment, awareness building activities implemented to reduce stigma, and implementing harm reduction strategies that targeted high risk groups. Regional representatives noted that efforts to increase testing and support for HIV/AIDS affected individuals focused on training community health nurses to encourage testing among vulnerable populations, and be more comfortable in conducting testing and providing pre- and post-test counseling. Other contributing factors included creating onsite testing facilities, hiring sexual health nurses and/or doctors, creating a community referral system to increase access to various specialists, and providing transportation to testing facilities in communities that face barriers related to access and privacy.
  • Strategies that increased treatment of TB through targeted education which were implemented and tested with high-risk groups. Strategies included one-on-one education and counselling support, recruiting and training Directly Observed Therapy workers and other community health staff, and mandatory TB screenings for clients entering a treatment centre.
Intermediate Outcome #2: Increased community capacity to plan and manage programs.

There are some indications that community capacity to manage CDCM programs has begun to increase, however, there are challenges at the community level including governance, accountability and human resource capacity. Key informants indicated that they perceived community capacity building to be critical for ensuring successful future transfer of communicable disease control and management to communities.

Importance of Community Capacity Building

Developing the capacity of the community health staff and communities overall is crucial to the success and continuity of services, as well as on-going efforts to increase community involvement and management of the CDCM programming. 

First Nations on-reserve communities have been funded through contribution agreements that varied in levels of responsibility and in various programs and services based on their capacity and willingness to take control. Health Canada was engaged in negotiations, led by AANDC, for transferring health services to First Nations governments in all regions. To support these efforts and devolution of health services to First Nations communities, it was crucial for FNIHB to provide ongoing support for capacity development and support for communities to plan, manage and monitor CDCM programming.  Some regional staff noted that the impact and success of CDCM programming ultimately rest with community health staff who delivered services, built trusting relationships and effective partnerships, provided support, and communicated the importance of communicable disease prevention. 

Strategies for Building Community Capacity

The strategies employed to build community capacity to plan and manage (i.e. delivery) CDCM programs depended on the type of agreements signed between the community and FNIHB, but generally involved providing support for the development of community health plans and ongoing engagement and educational activities.

CDCM programs defined community capacity development as helping communities to identify community needs through surveillance evidence, plan and organize services (including the short and long-term planning of financial and human resources), and maintain and/or improve the quality of services through relevant training for staff and the implementation of evidence-based policy, standards, guidelines, etc.

The planning requirements were set in the FNIHB Contribution Funding Framework, which meant that the funding model depends whether communities have capacity and are willing to take the responsibilities for program planning. For example, in the set funding model, FNIHB designed the programs and the recipient used a multi-year plan as provided by FNIHB to deliver and manage services. Under the flexible funding model, the recipient established a multi-year work plan which included a health management structure. Community-based agreements required recipients to develop a health plan which included a health management structure. FNIHB regional staff were involved in helping communities in developing the plans, through which key priorities, objectives, activities, and other programs and services were identified.

According to regional representatives, in communities where services were managed and delivered by FNIHB, regional FNIHB representatives worked directly with community health nurses, health directors and others to develop plans, set priorities and discuss strategies. In communities where management of the programming was transferred to the Tribal Council, FNIHB provided support in preparing plans, helping communities prepare proposals for health project funding, and provided advice, information and knowledge materials, resources, manuals and tools. Prior to 2013-2014, FNIHB regional offices developed and submitted regional work plans which identified resources, priorities, and strategies for collaboration, education, data collection and evaluation of the programming and capacity building. Although regional work plans were discontinued, regions were still required to submit detailed operational plans.

FNIHB regional representatives indicated that capacity was built through: 

  • Organizing and attending conferences and symposiums at the regional and community levels dedicated to various aspects of CDCM (e.g., nurse conferences, TB-HIV integrated symposium, Directly Observed Therapy workers conference, sexual health conference, National Immunization Conference, etc.). Representatives in some regions indicated this was decreased or was discontinued as a result of recent (starting in 2013-2014) restrictions on travel.
  • Delivering training, workshops and teleconferences to help communities plan for programming, and prepare for emergencies (e.g., workshops delivered on First Nations outbreak training, CDE training and emergency planning).
  • Direct one-on-one support through participation in community meetings and events. Face to face meetings were particularly important for engaging community leaders and other stakeholders (e.g., Regional Health Authorities, Bands and Aboriginal organizations, etc.) and in communities with limited access to technology. Meetings were used for various purposes such as increasing capacity for TB and HIV testing, providing education (e.g., to reduce stigma), improving pandemic awareness, and other projects (e.g., in Saskatchewan, the LEAN Project was conducted to analyze and reduce vaccine wastage).
  • Support in using surveillance data to plan and manage programs. Capacity to collect report and use data for planning and management purposes varied greatly across communities.
  • Development and distribution of manuals, tools and resources. Many regions developed and distributed planning and preparedness guidelines, guides, case scenarios, resources and templates. For example, the Atlantic region supported the development of policies and procedures for immunization practice and program delivery. In the Ontario region, an Immunization Protocol Binder was developed and introduced to communities. Some regions worked with other branches of Health Canada (e.g., Environmental Health Officers) and adapted their plans to assist First Nations communities.  

As with developing the capacity of community health nurses, the approaches used by regional offices to assess community needs, define services, and develop health plans and strategies differed across regions and programs. For example, according to regional staff, in Alberta, community needs were most often identified through regular and ongoing phone and video communication with community health nurses and health staff and through joint initiatives; in Manitoba, needs were identified through field visits to communities and working with community health staff to develop health plans (especially for transferred communities). In addition to those strategies, Atlantic Canada, for example, assisted community health nurses to respond to CDC requests for proposals. In Ontario, environmental scans and audits were commonly used to identify needs.

In some regions (e.g., Saskatchewan), the regional office did not have adequate human resources to conduct systemic needs assessments and develop strategies. The identification of needs also differed across CDCM program areas. For example, CDE Program needs were identified and communicated through an RFP process undertaken by communities;  VPD and Immunization and BBSTI and HIV/AIDS Programs most commonly used performance and, where available, surveillance data; and the TB Program received input from numerous working groups and stakeholders.

Progress in Developing Community Capacity

Progress was made in developing community capacity with respect to CDE planning and preparedness. The ability of FNIHB to determine the quality, completeness and robustness of plans, as well as the extent to which they are regularly tested, was limited. 

According to CBRT reports, 89% of communities in 2011-2012 and 92% of communities in 2012-2013 reported they had a pandemic plan. FNIHB established close collaboration with AANDC in order to provide necessary support and guidelines to most communities to develop CDE plans as part of their all hazards emergency plans. Community health representatives reported conducting regular meetings and collaborating with community leaders and other stakeholders, such as Regional Health Authorities, Bands and Aboriginal organizations (e.g., Assembly of First Nations) to develop the plans. To facilitate development of the plans, most FNIHB regional offices hired CDE coordinators, who worked directly with communities, provided the necessary tools and templates, and delivered workshops, training and teleconferences for community health nurses and leaders. FNIHB piloted an assessment tool to review CDE preparedness for First Nations communities. However, it was difficult to determine the quality, completeness and robustness of all CDE plans because communities with health transfer agreementsFootnote 29 were not required to submit their pandemic plans to FNIHB.

Communities were also required to regularly test their plans. Some communities test plans through their emergency response mechanism and others conducted testing through mock emergency scenarios, table-top exercises, and meetings with stakeholders and community leaders. Communities that were more affected by natural disasters were more likely to have their CDE plans integrated with all hazard emergency plans and tested during real emergencies. According to CBRT, the majority of communities who reported having an all hazard emergency plan integrated their pandemic plan with their emergency management plan (over 70%). However, not allFootnote 30 regions had all hazard emergency plans which made it more challenging for FNIHB to engage communities in testing their communicable disease plans.

It was more difficult to determine the extent to which communities regularly test their plans. For example, 77% of communities in 2011-2012, and 66% of communities in 2012-2013 self-reported that their pandemic plans had been tested. However, in the CBRT reports of the communities who reported having developed CDE plans, only 27% in 2011-2012 and 24% in 2012-2013 reported having tested the plan in a manner that involved engaging the appropriate stakeholders. Among surveyed community health representatives, 55% indicated that they had recently tested or had regularly tested their CDE plans, 30% indicated that plans were put to test during actual emergencies (e.g., H1N1 pandemic in 2009) but had not been tested in recent years, and 7% indicated that they had not tested their CDE plan at all.

Regional Roles in Building Community Capacity

The roles of regional offices with respect to building community capacities were not clearly defined for some aspects of the programming. One half of FNIHB regional staff reported that the roles of the regional office vis-à-vis the national office were not defined clearly with respect to issues such as providing support for communities, allocation of funding as needed, authorization for travel, and participation in training activities. While the majority of Health Canada representatives reported that improvements were made to clarify roles, others suggested that some challenges remain, particularly with understanding the mandate and constraints of each office. 

Some key informants, including federal partners, noted that the role between the FNIHB national and regional offices is not clear with respect to supporting data collection and monitoring practices and building community capacity in data collection. There also appears to be some overlap in responsibilities with respect to training and capacity building of community health staff.  As mentioned earlier, training is primarily the role of FNIHB regions; however, decisions on travel to and participation in training activities (e.g., conferences) were made by national office and, while very limited, some training activities were developed and delivered through national office.

4.4.3 To what extent has the longer term outcome been achieved?  

Long-Term Outcome #1: Reduced incidence and prevalence, spread and health effects of communicable diseases.

Although it is difficult to measure trends in incidence and prevalence rates and to assess CDCM contributions to those changes, preliminary data indicated that the incidence of communicable diseases remained fairly stable in most regions and was reduced in some other regions over the period covered under this evaluation.

Measuring Incidence and Prevalence Trends

It was difficult to measure trends in incidence and prevalence rates in First Nations on-reserve communities, and even more difficult to determine the contribution of CDCM programs to those trends due to various factors including, most notably, issues associated with surveillance data, the cyclical nature of communicable diseases, and the strong influence of socio-economic factors on incidence and prevalence rates. 

Differences in systems used to collect data across communities, differences in the number of communities reporting any given year, an inability to account for individuals receiving health services off-reserve, and the lack of access to provincial data in some regions made it very difficult to measure incidence rates within a particular region and almost impossible to do so nationally.

In addition, incidence rates were cyclical in nature, making it difficult to conduct a meaningful comparison of rates across years. For example, one outbreak of Hepatitis B reported in 2012 increased the incidence rate for the entire year, and the arrival of new flu-like viruses (e.g., H1N1) quickly spread and result in a pandemic. Community health staff noted that an increase in TB rates, for example, may be a factor of improved surveillance in the last few years rather than an increase in actual rates.

Another difficulty in determining the CDCM Program’s contribution to reducing incidence and prevalence rates was that other socio-economic and environmental factors  strongly influenced changes in communicable disease incidence trends but are outside of the CDCM programming mandate. The examples of such factors include: prevalence of other chronic health conditions (e.g., diabetes, asthma, vascular diseases, etc.); environmental factors such as housing conditions including air quality and ventilation, overcrowding and mold which increased susceptibility and transmission of airborne diseases including TB; personal behaviours such as smoking and substance abuse which increased communicable disease risk by weakening an individual’s immune system and increased their likelihood of engaging in high risk behaviour; the contribution of poverty and unemployment which prevented First Nations on-reserve people seeking adequate, appropriate or timely care; and the existence of cultural and historical barriers for health care (e.g., residential school experiences, lack of culturally-sensitive health care services, language barriers, etc.).

Program Contributions and Achievements

CDCM activities contributed to efforts to reduce the incidence, prevalence, spread and health effects of communicable diseases among First Nations on-reserve communities.

When asked to rate the success of the Program in reducing the incidence and prevalence of the CDCM targeted communicable diseases in their communities, the average ratings among community health representatives ranged from 3.0 to 3.6 across regions (where 5 is very successful). While most community health representatives and regional coordinators agree that the Program contributed to reducing prevalence and incidence rates, some noted that other determinants of health played a more significant role in influencing incidence and prevalence rates.

Although it was difficult to determine long term trends, surveillance data suggested that some improvements were made over the last few years, with respect to incidence rates. For example:

  • According to the Epidemiology of Tuberculosis in First Nations Living On-Reserve in Canada 2000-2008, TB incidence rates varied from a low of 24.1 per 100,000 in 2000, to a high of 32.4 per 100,000 in 2005 before declining to 26.6 per 100,000 in 2008. From 2010 to 2012, TB incidence rate ranged from 21.5 to 23.3 (Public Health Agency of Canada, 2009 was not reported). According to FNIHB data, the number of deaths in on-reserve First Nations due to TB was very low (7 in 2009 and 4 in 2010). The Public Health Agency of Canada reported 134 deaths during TB treatment in Canada in 2011.
  • According to the Program Performance Measurement data, some BBSTI have decreased in certain regions (note that this data is an indication of trends rather than reliable trend statistics). BBSTI data collected by FNIHB through performance measurement reports was preliminary available only for certain regions and for two years: 2011 and 2012. Based on this data, newly reported HIV cases per 100,000 people increased slightly in AB and MB and decreased in ON and SK. Hep C incidence rates decreased in four regions (AB, SK, MB and ATL) and slightly increased in ON. Chlamydia incidence rates decreased in MB, ON and ATL and increased in AB and SK. Gonorrhea incidence rates decreased in MB and ON and increased in AB and SK. Syphilis incidence rates decreased in SK and increased slightly in AB.

Data collection tools for VPD were revised in 2011 and are not comparable to previous years (up to 2010 only VPD cases for children 0 to 6 years of age were reported, while in 2010 and 2011 cases for all ages were reported). With the exception of pertussis and invasive pneumococcal disease, all other VPD counts were low for all age groups in 2011 and 2012 and there were no cases of polio, diphtheria or tetanus reported during these periods.

4.5 Performance: Issue #5 – Demonstration of Economy and Efficiency 

4.5.1    Has the program undertaken its activities in the most economical and efficient manner?

A review of the literature demonstrates the inherent economic benefit of programs designed to prevent communicable diseases. FNIHB has introduced structural and other program changes that were perceived to have increased efficiency of program operations and introduced more strategic and coordinated approaches in the design and implementation of CDCM programming

The Treasury Board of Canada’s Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Demonstration of efficiency can be approached from two perspectives: one is operational efficiency, which focuses on the relationship between resources and outputs, and the other is allocative efficiency, which focuses on the relationship between the resources and program outcomes. 

The financial systems and program budgeting structure that FNIHB had in place during the years covered by this evaluation did not allow for a comprehensive operational assessment of efficiency.  The analysis of efficiency and economy was therefore focused largely on ‘allocative efficiency’ using a utility based approach. In addition to some observations of allocation using a descriptive approach (providing available empirical data with limited analysis), the evaluation relied on a mixed approach (qualitative and quantitative data, including limited comparative analysis) to arrive at findings and conclusions related to the quality of program outputs, benefits of the program design and delivery in achieving outcomes, factors that contributed to the efficiency and economy of the CDCM Program, and potential alternatives for program delivery.

Socio-Economic Benefits

The literature review demonstrated that the types of programming delivered under the CDCM programs areas have significant socio-economic benefits for Canada in terms of direct medical costs associated with treating communicable diseases, costs associated with loss of labour productivity amongst infected people and their families, and loss of quality of life.

Further, the literature demonstrated the inherent cost-effectiveness of programs that prevented communicable diseases. A 2011 national study found the direct and indirect costs of people recently infected with HIV in Canada totaled $4.03 billion, or $1.3 million per person (lifetime cost), of which the direct medical costs accounted for 19%, labour productivity costs accounted for 52%, and loss in the quality of life accounted for 29%Footnote 31. Another study estimated that, for every dollar invested in community-based HIV prevention programs in Ontario, $51 in direct medical costs was avoided. An Australian studyFootnote 32 estimated that, for every $1 expended on HIV prevention programs, an estimated clinical care cost saving of $13 is achieved. As such, improvements to the effectiveness of programming for preventing communicable diseases improved overall efficiency.

Review of Budget and Expenditures

The financial information including the funding authority (budgeted amounts) and actual expenditures are presented in Table 2 below. Program contributions included funding allocated to prevention, promotion and education activities, coordination planning and reporting, outbreak management for TB and VPDs, research and knowledge for BBSTIs & HIV/AIDS, CDE planning and response, and IPC policy and program development.

Of the total $139 million direct program expenditures, about 30% was coded to TB Programming, 24% to BBSTIs and HIV/AIDS programming, 16% to Immunization and CDE (about 0.5% was allocated to IPC), and 14% was coded to policy and program development. Communities had the opportunity to readjust funding depending on their needs.

Table 2: Expenditures Related to CDCM Programming
Year Salaries O&M Gs&Cs Net Vote Revenue 315 Total
2008-2009 7,863,868 6,448,240 11,704,021 0 26,016,129
2009-2010 9,025,540 11,399,247 17,776,814 0 38,201,601*
2010-2011 7,807,835 5,680,464 13,277,718 0 26,766,017
2011-2012 7,799,079 4,894,964 13,240,015 -16,327 25,917,731
2012-2013 7,685,632 3,989,875 10,689,177 -16,327 22,348,357
Total $40,181,954 $32,412,790 $66,687,745 -$32,654 $139,249,835

*  Funding received as part of Supplementary Estimates B for Access to the Pandemic Influenza Contingency to Offset Planned Costs associated with the spring 2009 H1N1 Flu Virus Outbreak.

System Changes to Monitor Costs

For the five year period ending March 31, 2013, expenditures coded to the CDCM programs totalled $139.2 million, of which about $72.6 million was spent on operations and management, including salaries, and $66.6 million in contributions to communities. It should be noted that expenditures were understated to the extent that some costs (e.g., nurse salaries) were not allocated directly to the CDCM programs, but were included in Public Health Protection - Program Oversight and Nursing Delivery. Health Canada representatives noted that changes will be made to the accounting system for 2014-2015 which will better capture the total costs of the CDCM programs to address this latter issue of nurse salaries.

Structural Changes to Support Efficiency and Economy

Over the period covered under the evaluation, structural changes were made to increase the efficiency and economy of program operations and to take a more strategic and coordinated approach in the design and implementation of CDCM programming.  

The deficit reduction action plan led to the reorganization of the national office structure, which was intended to increase efficiencies. Prior to the reorganization in 2012, each program area operated as a separate group at the national level. In response to the recommendations of the previous evaluation (2010), the national office restructured into four functional groups combining the STBBI (including HIV/AIDS) and TB Assurance Team; the CDE, the IPC and Immunization Assurance Team; the Program Administration and Logistics team; and, the Policy and Assessment Team. Most Health Canada representatives interviewed noted that the new structure and implementation of the horizontal or clustered approach to program delivery allowed the programs to better coordinate their activities at the national level and to take a more strategic approach to implementation of clustered activities.

There were also some changes at the regional level. The regional units used to report to a separate branch within Health Canada. As part of the recent restructuring, the regional units now report directly to FNIHB management. According to program representatives, improvements were made to ensure a more seamless transfer of funds to regional offices and provide more flexibility to regional offices over funding decisions. As well, the Branch moved to a zero-based budgeting approach in the past two years under which the regions received block CDCM funding and now no longer submit detailed work plans to the national office for each CDCM program area. In addition, each region developed a transitional plan (2012-2015) to identify opportunities to better align their structures, operations and management practices with the strategic goals of the 2012 FNIHB Strategic Plan. The plans mapped out the management systems and organizational structures, challenges, and outlined the opportunities for improvement and strategic direction.

In addition to organizational restructuring, FNIHB developed the first strategic plan which described the directions and activities for the department in collaboration with a multi-sectorial and multijurisdictional approach to providing health services. This led, according to key informants to the development of national strategies for many CDCM activities, the clustering of TB and HIV activities, and amalgamating CDC programming with environmental health issues, all of which contributed to more strategic approach of program design and delivery. The IPC Plan and activities were also noted for having had a cross-cutting approach that applied to all programming areas, bringing them closer together.

Responding to Changes in Program Governance, Development and Implementation

Key informants noted that a review may be required to ensure that adequate financial and human resources are in place to implement regional transitional plans, support the increasing role of First Nations communities in the programming, and respond to the needs of the growing First Nations on-reserve population.  
 
The majority of Health Canada national office representatives, and over half of FNIHB regional staff, believed that the program resources were being used efficiently. However, some community health representatives and the majority of FNIHB regional coordinators disagreed that the programs had sufficient resources to address the needs of First Nations communities. Most regional staff reported that the community populations and workload of community health staff had increased significantly, and that existing staffing and resource levels in the communities were inadequate. They suggested that increased efforts are needed to address staff turnover and ensure staff have adequate time and resources dedicated to CDCM programming.

According to both FNIHB regional staff and community health representatives, over the past decade, the work load for community health staff increased significantly in terms of additional responsibilities related to data collection, reporting and other activities, impacted by population growth and without any increase in funding and resource provision. According to Aboriginal Demographics from the National Household Survey in 2011, the population of First Nations people living on-reserve grew by 2% from 2001 to 2006 and by 1% from 2006 to 2011. Some specifically recommended that more funding be allocated to developing capacity regionally and at the community level to collect data and evaluate programs. Other recommendations included continuing efforts to further coordinate and integrate communication, coordination and service delivery among the CDCM programs, as well as between the FNIHB national office and the regions, and to shift focus from delivering individual CDCM programs to a community health planning approach that will more efficiently address community needs.

4.5.2    Is there appropriate performance measurement in place?

While access to surveillance information has improved and was utilized to varying degrees to support policy development and program planning, further improvements are needed to increase the usefulness of the surveillance data for ongoing program monitoring, making policy and program decisions and setting priorities

Improving Data Collection and Reporting

The revised Performance Measurement System and data collection system provided some useful information for program planning and management. Further improvements are needed to streamline and better integrate reporting templates across different programs, focus more on health outcomes rather than outputs, and include efficiency measurements.

A review of the program documents indicated that FNIHB collected extensive performance information and surveillance data for each CDCM program through performance measurement templates. Other avenues used to collect surveillance data included the Public Health Agency of Canada reports, and the CBRT, of which the CDCM program activities comprise one component.

Over the period covered under this evaluation, a number of improvements were made to the Performance Measurement Systems for surveillance data collection and reporting. Significant revisions to the FNIHB performance templates were completed in 2011-2012 to improve quality of the performance data collected from the communities. In 2013, a system was created to allow FNIHB’s national office to produce and supply regional offices with a set of Communicable Disease Summary Surveillance Tables with detailed regional surveillance data for TB, Immunization, and BBSTI & HIV/AIDS. When asked whether the Performance Measurement System provided useful and reliable data and information needed to track performance and manage programs, most regional staff indicated that it was partially successful but added that provincial data as well as Canadian Tuberculosis Reporting System and Public Health Agency of Canada data was more reliable. While some indicators were useful for planning and adjusting programming (e.g., data on immunization), the disproportionate focus on outputs (number of activities delivered) rather than health outcomes was not useful in helping them better understand the changes in the communities. 
 
Another difficulty with data collection was highlighted indicating that communities were using a wide range of different data collection tools which impacted the reliability of information. For example, data collection in some communities in SaskatchewanFootnote 33 was still paper based. Furthermore, about half of regional staff noted that the outputs, including reporting, were often not produced in a timely manner and, the quality varied.

Key informants and regional staff suggested that the Performance Measurement Systems could be improved by streamlining the templates, focusing on the outcomes of CDCM programming as a whole rather than on individual programs, and aligning data collection and reporting to the extent possible with provincial strategies. Some Health Canada representatives noted that the PMS needs to be adjusted to reflect the changes in the structure of the programming, better integrated across various partners and assessed and aligned with the logic model and program terms and conditions. A review of the system highlighted the absence of clearly defined measures to assess the efficiency of the Program.

5.0 Conclusions

The CDCM program continues to fulfill the department’s obligation to delivery health services and meet the continued need to address health disparities.

The evaluation demonstrates that there is a continued need for CDCM program activities due to ongoing health disparities among Fist Nations related to communicable diseases. The activities within the CDCM program are priorities of the federal government including increasing access to health care and addressing the health status inequalities of First Nations communities.

While the increased program flexibility and more targeted approach is believed to be more effective in reacting to communicable disease issues, the absence of comparable and reliable information and ongoing surveillance of First Nations health outcomes makes it very challenging to assess the responsiveness to the needs of communities over time.   

Although it was difficult to measure trends in incidence and prevalence rates and to assess CDCM contributions to those changes, preliminary data indicated that the incidence of communicable diseases remained fairly stable in most regions and was reduced in some other regions over the period covered under this evaluation.

The CDCM is making progress in addressing the health effects of communicable diseases as evidenced by continued investments in community health resources, ongoing training and collaborative efforts between regional offices and the provinces, and efforts focused on improving community capacity to manage CDCM programs in terms of overall governance, accountability and human resource capacity.

Further, significant progress has been made in terms of building collaboration and partnerships at the national, regional and community levels to share information and achieve integrated service delivery leading to increased uptake of various prevention and disease control strategies and measures.

Although FNIHB’s access to surveillance data on health outcomes has improved and was utilized to varying degrees to support policy development and program planning, the complexity of surveillance data remains a challenge given differences in key surveillance data sources. The access to quality surveillance data and its reliability across programs and regions further constrains the utility of the information.

6.0 Recommendations

Recommendation #1

Continue efforts in building community capacity by ensuring consistency in training and support for community health staff.

Development of community capacity to manage the delivery of CDCM programs includes various supports which help communities to identify community needs, plan and organize services (including the short and long-term planning of financial and human resources), and maintain and/or improve the quality of services through relevant training for staff and the implementation of evidence-based policy, standards, guidelines, etc. The evaluation found that developing capacity of the community health staff and communities overall is crucial to success and continuity of CDCM programming. While some communities had high level of capacity for planning and management of the CDCM programming, other continued to face challenges with competing priorities and limited human resource capacity, and opportunities to engage with FNHIB regional offices.  

Recommendation #2

Strengthen program ability to address diverse ongoing as well as emerging community needs related to communicable diseases.

Program flexibility and targeted programming and investment were considered to be successful in meeting wide range of needs and priorities of communities across and within FNIHB regions. However, challenges remain with respect to Program’s ability to monitor and respond to wide range of needs and factors affecting communicable diseases such as further increasing collaboration and engagement with partners, integration and alignment of programming with other service providers and ensuring program resources are allocated appropriately based on priorities and need within communities.

Recommendation #3

Building on past successes, continue efforts to collect, access and use reliable surveillance data.

Numerous data limitations including issues with quality and reliability, timeliness and the ability to share data among partners and stakeholders remain a significant challenge for the Program and its efforts to increase community capacity to manage CDCM programs. Improved collection, reporting and overall access to surveillance data and health outcomes for First Nations living on-reserve will assist programs to continue to identify needs, planning and managing of programs, develop targeted responses and produce evidence based policies.

Appendix 1 – References

  • Canadian Supplement to The State of the World’s Children 2009 - Aboriginal Children’s Health: Leaving no Child Behind. 2009. pg. 2. http://www.nccah-ccnsa.ca/docs/nccah%20partner%20documents/UNICEF%20Report,%20English.pdf
  • CBRT Reports (2011/12 and preliminary findings for 2012/13)
  • First Nations Information Governance Centre, (2012). First Nations Regional Health Survey 2008/10: National report on adults, youth and children living in First Nations communities. Ottawa: FNIGC.
  • FNIHB PMS data 2008 to 2012
  • Government of Canada, HIV/AIDS. http://healthycanadians.gc.ca/health-sante/disease-maladie/hiv-vih-eng.php
  • Health Canada, Epidemiology of Tuberculosis in First Nations Living On-Reserve in Canada, 2000-2008. pg. vi.
  • Health Canada, “Knowledge, Perceptions, Awareness and Behaviours Relating to Immunization amongst First Nations and Inuit”, February, 2011.
  • Health Canada, Mandate, Plans and Priorities. http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/fnihb-dgspni/mandat-eng.php
  • Health Canada’s Report on Plans and Priorities 2014-2015
  • Indian and Northern Affairs Canada – Fact Sheet: First Nations and Inuit Health in Canada. http://www.ainc-inac.gc.ca/ai/mr/is/fnihb-eng.asp
  • Johnston Research Inc. (2011). 10-Year First Nations and Inuit Home and Community Care Strategy Literature Review Final Submission. Ottawa, ON: Health Canada
  • Kirby Institute for Infection and Immunity in Society (University of New South Wales). 2013. Bloodborne viral and sexually transmitted infections in Aboriginal and Torres Strait Islander people: Surveillance and Evaluation Report 2013. http://www.kirby.unsw.edu.au/sites/ default/files/hiv/resources/2013ATSIP-ASR.pdf.
  • National Collaborating Centre for Aboriginal Health, Achieving Strength Through Numbers: First Nations, Inuit and Métis Health Information, 2010. http://www.nccah-ccnsa.ca/docs/fact%20sheets/setting%20the%20context/NCCAH_fs_context_EN.pdf
  • Public Health Agency of Canada, Evaluation of the Federal Initiative to Address HIV/AIDS in Canada, 2008/09 to 2012/13
  • Public Health Agency of Canada, Immunization Coverage in Canada (2002 to 2012), http://www.phac-aspc.gc.ca/im/nics-enva/icc-cvc-eng.php#fig-1
  • Public Health Agency of Canada, Tuberculosis in Canada 2012 - Pre-Release.
  • Public Health Agency of Canada, Summary - Estimates of HIV Prevalence and Incidence in Canada, 2008. http://www.phac-aspc.gc.ca/aids-sida/publication/survreport/estimat08-eng.php
  • Smylie, J. (2010). Achieving strength through numbers: First Nations, Inuit, and Métis health information. Prince George: National Collaborating Centre for Aboriginal Health.
  • Speech from The Throne The First Session Thirty-Eighth Parliament of Canada. House of Commons Journals:  2004.10.05, vol.  140, p.  15.
  • The Chief Public Health Officer’s Report on the State of Public Health in Canada, 2013: Infectious Disease—The Never-ending Threat; Public Health Agency of Canada

Appendix 2 – Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation question and issue have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Questions Indicators Overall Rating Summary
Continued Need for the Program
Does CDCM continue to address a health need? Has the need changed over time? Has the program adapted to changing needs. Incidence and prevalence of communicable diseases among First Nations populations.  Immunization coverage rates.
Stakeholder perspectives on the relevance of the programs design. Perceptions on how efforts could be more helpful or accomplish more including effectiveness of efforts compared to other interventions.
High The CDCM program continues to fulfill the department’s obligation to delivery health services and meet the continued need to address health disparities.
Communicable Disease Control (including immunization and pandemic planning) is one of the mandatory programs designed to meet legislated standards such as provincial and territorial public health legislation and health acts that ensures public health and safety. The continuing need for the CDCM program activities is also attributed to the disparities in health outcomes experienced by First Nations living on-reserve in comparison to the general population, and an ongoing need for communicable disease emergency preparedness. The flexible design of the CDCM programs is largely responsive to the varied needs of communities; however, challenges still exist for both regional partners and individual communities to address competing health priorities and the prevalence of other contributing factors such as chronic health conditions and variations in community capacity and ability to plan and manage programs.
Alignment with Government Priorities
Do the objectives and priorities of CDCM align with current Government of Canada and Health Canada priorities and strategic outcomes? Evidence demonstrating CDCM remains a priority of the federal government. Key drivers/ determinants of CDCM targeted communicable diseases. Alignment of the CDCM activities with departmental strategic priorities and outcomes. High The CDCM objectives and priorities align with the Government of Canada and departmental current priorities and strategic outcomes by increasing access to health care and addressing the health status inequalities affecting First Nations and Inuit communities
Alignment with Federal Roles and Responsibilities
Do Health Canada activities related to CDCM align with federal government jurisdictional, mandated and/or legislated roles and responsibilities? Legitimacy of federal role in the program area. Examination of regional roles including strengths and opportunities and inclusion of strategic partnerships for engaging community leadership. High The CDCM activities align with federal government jurisdictional, mandated and/or legislated roles and responsibilities through fulfilling various policies and authorities enabling the provision of health services for First Nations and Inuit.
Legend – Relevance Rating Symbols and Significance:

High: There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.

Partial: There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.

Low: There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Questions Indicators Overall Rating Summary
Achievement of Expected Outcomes (Effectiveness)
Is the Program achieving the outcomes expected as outlined in the Logic Model? Immediate Outcomes:
Increased workforce skills, knowledge and/or expertise in CDCM.
Increased public awareness and knowledge of prevention and control measures as well as communicable disease health effects.
Service delivery increasingly informed by surveillance data, evidence-based policy, standards, guidelines and frameworks.
Increased community and stakeholder engagement and collaboration in policy/ program development and service delivery.
Progress Made; Further Work Warranted Although it was difficult to measure trends in incidence and prevalence rates and to assess CDCM contributions to those changes, preliminary data indicated that the incidence of communicable diseases remained fairly stable in most regions and was reduced in some other regions over the period covered under this evaluation.

The CDCM is making progress in addressing the health effects of communicable diseases as evidenced by continued investments in community health resources, ongoing training and collaborative efforts between regional offices and the provinces, and efforts focused on improving community capacity in terms of overall governance, accountability and human resource capacity.

Further, significant progress was made in terms of building collaboration and partnerships at the national, regional and community levels to share information and achieve integrated service delivery leading to increased uptake of various prevention and disease control strategies and measures.

Intermediate Outcomes:
Increased use/ uptake of CDCM prevention and control measures.
Increased community capacity to manage CDCM programs.

Progress Made; Further Work Warranted

Longer Term Outcome:
Reduced incidence, spread and health effects of communicable diseases.

Progress Made; Further Work Warranted

Appropriate performance measurement in place.

Progress Made; Further Work Warranted
Although, FNIHB’s access to surveillance data on health outcomes has improved and was utilized to varying degrees to support policy development and program planning, the complexity of surveillance data remains a challenge given differences in key surveillance data sources. The access to quality surveillance data and its reliability across programs and regions further constrains the utility of the information.
Demonstration of Economy and Efficiency
Has the CDCM been managed efficiently and economically? Economy
Has the program undertaken its activities in the most economical and efficient manner?
Efficiency
Is there appropriate performance measurement in place?
Progress Made; Further Work Warranted A review of the literature demonstrates the inherent economic benefit of programs designed to prevent communicable diseases. FNIHB has introduced structural and other program changes which were perceived to have increased efficiency of program operations and introduced more strategic and coordinated approaches in the design and implementation of CDCM programming.
Legend – Performance Rating Symbols and Significance:

Achieved: The intended outcomes or goals have been achieved or met.

Progress Made; Further Work Warranted: Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.

Little Progress; Priority for Attention: Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.
Table 3: Summary of Relevance Ratings
Evaluation Issue High Partial Low
Issue 1: Continued need for the program
Does CDCM continue to address a health need? Has the need changed over time? Has the program adapted to changing needs? High N/A N/A
Issue 2: Aligned to federal government priorities
Do the objectives and priorities of CDCM align with current Government of Canada and Health Canada priorities and strategic outcomes? High N/A N/A
Issue 3: Program consistent with federal roles and responsibilities
Do Health Canada activities related to CDCM align with federal government jurisdictional, mandated and/or legislated roles and responsibilities? High N/A N/A
Legend - Relevance Rating Symbols:

High: There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.

Partial: There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.

Low: There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.
Table 4: Summary of Performance Ratings
Evaluation Issue Achieved Progress Made;
Further Work Warranted
Little Progress;
Priority for Attention
Issue 4: Achievement of intended outcomes (effectiveness)
Is the Program achieving the outcomes expected as outlined in the Logic Model? N/A Progress Made; Further Work Warranted N/A
Issue 5: Demonstrated economy and efficiency
Has the CDCM been managed efficiently and economically? N/A Progress Made;
Further Work Warranted
N/A
Legend - Performance Rating Symbols:

Achieved: The intended outcomes or goals have been achieved or met.

Progress Made; Further Work Warranted: Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.

Little Progress; Priority for Attention: Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.

Appendix 3 – Logic Model

COMMUNICABLE DISEASE CONTROL AND MANAGEMENT LOGIC MODEL
Target Group First Nations on-reserve (all programs/activities) and some communicable disease support for Inuit south of the 60th parallel
Theme Service
Provision
Capacity
Building
Stakeholder
Engagement
and Collaboration
Data Collection,
Research and
Surveillance
Policy Development & Knowledge Sharing
Outputs Communicable disease
prevention and control
measures
offered
/ available
Workforce training activities offered and completed

Public education and awareness activities (e.g., community events and social marketing campaigns)

Community health planning and management support
Collaborative process and practices in place:

Data sharing agreementsContribution

Agreements/Memoranda of Understanding/Letters of Intent

Joint projects

Committees/Working Groups
Communicable disease
surveillance and research
reports
produced
Policies,
standards and service delivery guidelines developed/
refined Knowledge
transfer
products
Reach First Nation reserves and one Inuit region (Nunatsiavut) Nurses and support workers (Immunization, DOT and HIV/AIDS)

First Nation reserves and one Inuit region (Nunatsiavut)
Provincial governments

Regional Health Authorities


FNIHB regions First Nation and one
Inuit region (Nunatsiavut)

Public Health Agency of Canada
FNIHB regions

First Nation and one Inuit region (Nunatsiavut)
Immediate Outcomes Increased workforce skills, knowledge
and / or expertise in CDCM

Increased public awareness and
knowledge of prevention and
control measures as well as
communicable disease health effects
Service delivery increasingly informed by surveillance data, evidence-based policy, standards, guidelines and frameworks

Increased community and stakeholder engagement and collaboration in policy/program development and service
delivery
Intermediate Outcomes Increased use / uptake of CDCM prevention and control measures Increased community capacity to manage CDCM programs
Longer Term Outcomes Reduced incidence, spread and health effects of communicable diseases

Logic Model Narrative for the First Nations and Inuit Communicable Disease Control and Management

Target Group: First Nations on-reserve (all programs/activities) and some communicable disease support for Inuit south of the 60th parallel

Theme Activities and Related Outputs and Outcomes:

Theme 1: Service Provision

Related Output: Communicable disease prevention and control measures offered / availableFootnote 34
Related Immediate Outcomes:

  • Increased workforce skills, knowledge and/or expertise in CDCM
  • Increased public awareness and knowledge of prevention and control measures as well as communicable disease health effects

Related Intermediate Outcome: Increased use/uptake of CDCM prevention and control measures

Theme 2: Capacity Building

Related Outputs:

  • Workforce training activities offered and completed
  • Public education and awareness activities (e.g., community events and social marketing campaigns)
  • Community health planning and management support

Related Immediate Outcomes:

  • Increased workforce skills, knowledge and/or expertise in CDCM
  • Increased public awareness and knowledge of prevention and control measures as well as communicable disease health effects

Related Intermediate Outcome:

  • Increased use/uptake of CDCM prevention and control measures
Theme 3: Stakeholder Engagement and Collaboration

Related Output

  • Collaborative process and practices in place including: data sharing agreements, contribution agreements, memoranda of understanding, letters of interest; joint projects; committees and working groups

Related Immediate Outcomes:

  • Service delivery increasingly informed by surveillance data, evidence-based policy, standards, guidelines and frameworks
  • Increased community and stakeholder engagement and collaboration in policy/program development and service delivery

Related Intermediate Outcome:

  • Increased community capacity to manage CDCM programs
Theme 4: Data Collection, Research and Surveillance

Related Output:

  • Communicable disease surveillance and research reports produced

Related Immediate Outcomes:

  • Service delivery increasingly informed by surveillance data, evidence-based policy, standards, guidelines and frameworks
  • Increased community and stakeholder engagement and collaboration in policy/program development and service delivery

Related Intermediate Outcome:

  • Increased community capacity to manage CDCM programs
Theme 5: Policy Development and Knowledge Sharing

Related Outputs:

  • Policies, standards and service delivery guidelines developed/refined
  • Knowledge transfer products

Related Immediate Outcomes:

  • Service delivery increasingly informed by surveillance data, evidence-based policy, standards, guidelines and frameworks
  • Increased community and stakeholder engagement and collaboration in policy/program development and service delivery

Related Intermediate Outcome:

  • Increased community capacity to manage CDCM programs
Long-term Outcomes

All activities, outputs, immediate and intermediate outcomes lead to the same long-terms outcome of:

  • Reduced incidence, spread and health effects of communicable disease

Evaluation Purpose and Scope  

The main objective of this evaluation is to provide Health Canada with comprehensive and reliable evidence to support decisions regarding continued implementation of communicable diseases interventions or initiatives. The evaluation is required by the 2009 Treasury Board Policy on Evaluation and, as per Health Canada’s 5-Year Departmental Evaluation Plan, the Evaluation Report and Management Action Plan is to be completed in 2014-15.

The scope of the evaluation covered the period from April 2008 to March, and all the FNIHB CDCM programs/services as defined by the renewed 2011 Authorities (i.e., Immunization, Tuberculosis, HIV/AIDS and CDE including IPC).

The evaluation issues are aligned with the Treasury Board of Canada’s Policy on Evaluation (2009) and consider the five core issues under the two themes of relevance and performance. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process. An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

Evaluation Issues and Questions

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada’s Directive on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Core Evaluation Issues and Questions
Evaluation Issues Description Evaluation Question
Relevance
Issue #1: Continued Need for program Assessment of the extent to which the programs continue to address a demonstrable need and are responsive to the needs of Canadians
  • Does CDCM continue to address a health need? Has the need changed over time? Has the program adapted to changing needs?
Issue #2: Alignment with Government Priorities Assessment of the linkages between the programs’ objectives and (i) federal government priorities and (ii) departmental strategic outcomes
  • Do the objectives and priorities of CDCM align with current Government of Canada and Health Canada priorities and strategic outcomes?
Issue #3: Alignment with Federal Roles and Responsibilities Assessment of the role and responsibilities for the federal government in delivering the programs
  • Do Health Canada activities related to CDCM align with federal government jurisdictional, mandated and/or legislated roles and responsibilities?
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement
of Expected Outcomes
Assessment of progress toward expected outcomes (including immediate, intermediate and longer term outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes.
  • To what extent has the CDCM achieved its expected immediate, intermediate and long-term outcomes?
  • What have been the challenges, barriers and opportunities?
  • What changes have been made as a result of the last evaluation?
  • Are there positive or negative unintended consequences as a result of program implementation?
Issue #5: Demonstration
of Efficiency and
Economy
Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.
  • Has the program undertaken its activities in the most economical and efficient manner?
  • Is there appropriate performance measurement in place?

Data Collection Methods

Evaluators collected and analyzed data from multiple sources, including primary and secondary data sources. Primary data sources include community representatives, FNIHB regional coordinators, and key informants. The regional breakdown and the description of each groups is provided below.

Table 1: Number of Regional Representatives
Regional Representatives AB SASK MB ON QC ATL Total
Regional Staff 2 7 4 6 1 4 24
Regional Coordinators, Nurse Managers and Program Managers 3 2 5 2 1 2 15
Community Reps 39 18 22 15 13 17 124
Total 44 27 31 23 15 23 163
  • Phone interviews with 41 key informants, including 10 national office program staff, 24 regional staff, 2 representatives of  the Public Health Agency of Canada, and 5 national program partners and experts including Assembly of First Nations and Inuit Tapiriit Kanatami. A list of key informants was provided by Health Canada. Key informants were selected in consultation with the evaluation committee and included those who were directly involved in the program at the national and regional levels, as well as program partners and experts working in one or more areas of FNIHB targeted communicable diseases. An email was sent to each interviewee, which described the purpose of the evaluation and solicited their participation in a telephone interview. A relevant interview guide was attached to each email. A total 41 interviews was conducted which was significantly higher than initial target of 30. The results of the interviews were summarized and presented in a separate Key Informant Technical Report.
       
  • A survey of 15 regional coordinators who oversee the CDCM program activities at regional offices.  A list of 17 regional coordinators was obtained from the FNIHB regional offices, including contact information. A link to the online survey was emailed to these coordinators along with an invitation letter explaining purpose of the evaluation, importance of participation and confidentiality of the responses. The original objective was to administer these surveys online. However, Health Canada employees were not able to access the survey site due to restrictions imposed by an internal firewall within the Department. Consequently, the decision was made to conduct the surveys by telephone. Of the 17 regional coordinators contacted, 15 completed the survey for an 88% response rate. The results of the surveys were summarized and presented in a separate Survey Technical Report.
      
  • A survey of 105 community representatives involved in program delivery including community health nurses (CHNs) and health directors. A list of 305 community representatives was developed based on the contact information provided by FNIHB regional offices. A link to an online survey was emailed to these representatives along with an invitation letter explaining the purpose of the evaluation, importance of participation and confidentiality of the responses. At least three email reminders and multiple phone follow-ups were conducted with each representative on the list. Representatives were provided with an option to complete surveys by telephone, online or to be mailed a copy of the questionnaire. About 7% choose to complete the interview over the phone. Of the 305 community representatives contacted, 23 were not reached (e.g., emails bounced back, were away during the survey, etc.), 100 completed the questionnaire, 33 provided partial answers and 28 visited the survey site but did not provide any useful information. After initial data cleaning, 105 surveys were used in analysis, resulting in a response rate of 34%. The results of the surveys were summarized and presented in a separate Survey Technical Report. 

Various scales were used in the surveys. For example: progress made is measured on a scale of 1 to 5, where 1 is no progress at all, 3 is some progress and 5 is major progress; satisfaction is measured on a scale of 1 to 5, where 1 is not at all satisfied, 3 is somewhat satisfied and 5 is very satisfied, use of policies is measured on a scale of 1 to 5, where 1 is not at all, 3 is to some extent and 5 is to a great extent. Agreement on a scale of 1 to 5 was also used, where 1 represented do not agree, 3 somewhat agree and 5 strongly agree. 

The secondary data sources used in the evaluation were as follows:

  • An extensive review of the program documents and files. The types of files and documents reviewed included internal government documents such as Government of Canada Budget Announcements, Speeches from the Throne, Health Canada Reports on Plans and Priorities and program descriptions; documents produced by the FNIHB headquarters and regional offices, such as regional progress reports and work plans, FNIHB Strategic Plan, FNIHB Accountability Framework, departmental operational plans, Community Based Reporting Template (CBRT) data, financial information, CDCM program developed policies, Health Canada’s TB Strategy, and past evaluations and audits of CDCM programs; and financial information and expenditure reports as well as limited data available from the Management of Contracts and Contributions System database. The results of the document and data review were summarized and presented in a separate Document and Data Technical Report.
  • A review of the relevant literature which included an extensive review of information from sources external to the FNIHB Program, including both peer-reviewed (scientific and academic) and grey literature (newspapers and websites). The literature was searched for based on a review of references in recent reports, evaluations and program documents as well as a keyword search. Examples of key words used to conduct the online search of related literature in the Canadian context and other countries included: Aboriginal, First Nations, Inuit, Indigenous, communicable disease, infectious disease, vaccine preventable disease, immunization, Tuberculosis, TB, HIV/AIDs, sexually-transmitted infection, social, environmental and other health determinants, health awareness, awareness-building, promotion, prevention, control, strategy, pandemic planning, emergency plan, emergency planning, outbreak, infection prevention and control, best practices, etc. Where available, abstracts of the key articles and reports were reviewed to determine their relevance and their bibliographies used to identify additional material for review. The review also included an overview of other national and regional/provincial/territorial programs similar to the CDCM programs. The information on the other programs included this review was obtained from relevant websites, journals and grey literature articles. Information on other similar programs/initiatives operating in jurisdictions outside of Canada (e.g., United States, New Zealand, Australia) which intervene on the drivers considered key in the development and transmission of CDCM targeted communicable diseases were sourced using a keyword search. In addition, group telephone interviews were conducted with a total of 9 representatives from other jurisdictions including 2 representatives from the United States Indian Health Service HIV Program, 6 representatives from the Australian Department of Health (representatives from the Office of Health Protection, including representatives responsible for health emergency management, immunization, and bloodborne viruses and STI policy, as well as a representative from the Indigenous and Rural Health Division), and one representative from the New Zealand Ministry of Health Maori Health Unit. The interviews followed a discussion guide which included questions pertaining to the role of organizations responsible for CDCM for indigenous people and how they collaborate, the resources dedicated to the programs and services, surveillance and data collection methods and challenges, and best practices and lessons learned in program delivery. The results of the literature review were summarized and presented in a separate Literature Review Technical Report.
  • A regional comparative analysis. A detailed profile of program design and delivery, success in reaching objectives, barriers, challenges, and learned lessons were developed and reviewed across five regions. This line of evidence did not involve additional research but rather consisted of applying a regional lens to the data already collected, particularly the results of the document and data review, key informant interviews, and survey of program representatives and health professionals involved in the programs. The analysis allowed for a further examination of program performance across regions and highlighting of best practices. A regional dashboard was created that provided an overview of the strengths and challenges facing various regions in the implementation of the CDCM program including an assessment of strategic partnerships for engaging key stakeholders and community leadership. The results of this line of evidence was summarized and presented in a Regional Profile Technical Report.

Data Analysis 

The information gathered from each line of evidence was summarized and presented in six technical reports which included detailed information and extensive analysis of data associated with relevant evaluation questions. SPSS software was used for cleaning and analysis of both qualitative and quantitative data collected via interviews and surveys. Data analysis techniques included statistical analysis of quantitative data from databases and surveys and key informant interviews and thematic analysis of qualitative data. Preliminary findings were presented in PPT form to the Evaluation Directorate, Evaluation Committee and Assembly of First Nations/Inuit Tapiriit Kanatami partners. Minor changes were made in order to ensure that the context is provided for some data and differences across regions are better illustrated. The comments and feedback from the committee was also considered in writing the final report. Triangulation of multiple lines of evidence was used to arrive at conclusions and test the strengths and limitations of each line of inquiry when writing final report. Conclusions that were supported by more than one line of evidence were presented in the final report.

List of Acronyms

AANDC
Aboriginal Affairs and Northern Development Canada
BBSTI
Blood-Borne and Sexually Transmitted Infections
CBRT
Community Based Reporting Templates
CDCM
Communicable Disease Control and Management
CDE
Communicable Disease Emergencies
FNIHB
First Nations and Inuit Health Branch
IPC
Infection Prevention and Control
MOU
Memorandum of Understanding
PMS
Performance Measurement Systems
TB
Tuberculosis
VPD
Vaccine Preventable Diseases

Page details

Date modified: