Evaluation of the First Nations National Nursing Innovation Strategy Program 2008-2009 to 2011-2012

(PDF Version - 522 K)

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

June 2013

List of Acronyms

FNIHB: First Nations and Inuit Health Branch
IT: Information Technology
HIV: Human Immunodeficiency Virus
LPN: Licensed Practical Nurse
NNISP: National Nursing Innovation Strategy Program
NP: Nurse Practitioner
RN: Registered Nurse

Executive Summary
Management Response
1.0 Evaluation Purpose
2.0 Program Description
3. Evaluation Description
4.0 Findings
5.0 Conclusions
- 5.1 Relevance Conclusions
- 5.2 Performance Conclusions
6.0 Implications
7.0 Lessons Learned
Appendix 1: Details on Data Collection and Analysis Methods
Appendix 2: References

List of Tables

Table 1 - Summary Table of Evaluation Findings, Conclusions and Implications
Table 2 - NNISP Logic Model
Table 3 - Limitations and Mitigation Strategies
Figure 1 - NNISP Annual Expenditures

Executive Summary

Evaluation Purpose, Scope and Design

The evaluation of the National Nursing Innovation Strategy Program (NNISP) was part of Health Canada's Five-Year Departmental Evaluation Plan to fulfill its reporting obligations under the First Nations and Inuit Health Branch (FNIHB) Renewal of Authorities (2011) as well as to gain insight into the achievements of the pilot projects funded through this program.

The evaluation covered the period from 2008-09 to 2011-12, prior to the completion of the NNISP, which ended March 31, 2013. The evaluation assessed the relevance and performance of NNISP.

Specifically, the evaluation focused on the success of pilot projects and progress made toward the achievement of the following immediate outcomes: improved access to primary care services, increased capacity of the primary care workforce, increased access to primary care nursing education, improved collaboration among health care providers and broader stakeholders, and increased use of evidence-based information to inform quality primary care service delivery. Due to the time-limited nature of funded pilot project implementation, this evaluation focused on the achievement of immediate outcomes.

Program Description

Nurses play a central role in the delivery of health services in remote and isolated First Nation communities. As often the first and only point of contact, nurses require innovative approaches and adaptability within their regulated authority to manage ever changing needs including the use of medical technologies and changing infrastructure.

As a result, the federal government committed funding for five years (2008-13) to pilot innovative practices and approaches under the NNISP. The NNISP used a proposal-based funding strategy to fund pilot projects within a time-limited five-year period. Proposals were accepted within four innovation streams, each focusing on different aspects of primary care service delivery as follows: Stream 1--Collaborative Teams; Stream 2 -- Integration of Nursing and Technology: Stream 3 - New Hours of Operation, and; Stream 4 -- a National Education Strategy.

Health Canada headquarters in the National Capital Region was ultimately responsible for the program, in partnership with the regional offices of FNIHB. FNIHB senior management and program staff managed and coordinated the program. Health Canada regional offices, who assisted in the pilot project implementation, consisted of regional directors, regional nursing officers/directors, nurse educators, nursing staff, project managers/leads, and project staff.

External partners included First Nations Bands and Tribal Councils in remote and isolated communities, which responsible for implementing specific NNISP pilot projects within their communities.

Evaluation Conclusions and Implications

Conclusions: Relevance

Need for the NNISP

The NNISP was appropriately designed to address specific challenges identified in remote and isolated First Nation communities through the use of innovation in primary care service delivery. There is a need for continued innovative approaches in primary care service delivery.

Alignment with Government Priorities

The NNISP was aligned with federal government priorities and departmental strategic outcomes. While the NNISP concluded March 31, 2013, the federal government continues to prioritize the delivery of primary care services in remote and isolated First Nations communities.

Alignment with Federal Roles and Responsibilities

The NNISP was aligned with the federal roles and responsibilities of improving First Nations health by demonstrating innovative practices and approaches in primary care service delivery.

Conclusions: Performance

Achievement of Outcomes

The NNISP demonstrated progress towards the achievement of immediate outcomes. The innovative practices and approaches to primary care made by many of the pilot projects, combined with the lessons learned from this initiative, could be incorporated in future models of primary care service delivery.

Economy and Efficiency

Many pilot projects provided examples of ways to impact primary care service delivery efficiency that should be considered in future models for primary care delivery. Improved overall performance measurement data would better support reporting requirements and the conduct of future evaluations of primary care service delivery.

Implications

No formal recommendations were proposed for this evaluation given that the NNISP ended March 31, 2013. Based on the findings and conclusions outlined in this evaluation report, there are lessons learned from the innovation pilot projects that will be valuable to consider with the ongoing provision of primary care service delivery in remote and isolated First Nation communities.

Lessons Learned

The lessons learned can be classified into two themes as outlined below and include ways to implement innovative practices and approaches that would improve the future of primary care service delivery in remote and isolated First Nation communities.

Project-Specific Lessons

Implement partnership strategies with other jurisdictional stakeholders in order to promote sustainability of successful innovation through long-term community and/or provincial involvement and support.
Improve jurisdictional partnerships in the delivery of health care services through collaborative approaches to primary care service provision (e.g., telerobotics).
Ensure adequate time and expertise to facilitate timely project development and funding; identify the appropriate skills required for project implementation (e.g., technology skills), and ensure appropriate and consistent performance monitoring and measurement across all projects.
Develop strategies to manage the growing community expectations for additional health care services.

Primary Care Service Delivery Lessons

Recognize and address the ongoing challenges of continuing education and professional development for nurses which impact recruitment and/or retention (i.e., increase on-line and other electronic resources that improve access to remote education opportunities for nurses within remote or isolated communities).
Provide alternative modes to support education (i.e., paper copies and courses on CDs) where high-speed internet is not available in remote communities.
Develop clear job descriptions for newly created positions.
Improve the collaborative team approach (i.e., address issues of resistance to change, duplication of work effort and the working capacity of care teams; develop community and team capacity to support the change process, clarify and communicate new team roles and the distribution of client workload among team members).

Table 1: Summary Table of Evaluation Findings, Conclusions and Implications
Evaluation Issue		Findings	Conclusions	Implications
Relevance	Need for the NNISP	The NNISP was required in order to find innovative ways to address the provision of primary care services in remote and isolated First Nation communities, including addressing the challenges of recruitment, retention and training of nursing staff and allied health professionals.	Relevance of the Program Continued Need 1. The NNISP was appropriately designed to address specific challenges identified in remote and isolated First Nation communities through the use of innovation in primary care service delivery. There is a need for continued innovative approaches in primary care service delivery.	No formal recommendations were proposed for this evaluation given that the NNISP ended March 31, 2013. Based on the findings and conclusions outlined in this evaluation report, there are lessons learned from the innovation pilot projects that will be valuable to consider with the ongoing provision of primary care service delivery in remote and isolated First Nation communities.
	Alignment with Government Priorities	The NNISP aligned with the federal government and departmental strategic priorities, particularly with respect to its emphasis on innovation in primary health care delivery, collaborative health teams and increased access to primary care services. It also aligned with the priority of ensuring that remote and isolated First Nation communities are receiving responsive health services and benefits in order to improve their health status.	Government Priorities 2. The NNISP was aligned with federal government priorities and departmental strategic outcomes. While the NNISP concluded March 31, 2013, the federal government continues to prioritize the delivery of primary care services in remote and isolated First Nations communities.
	Alignment with Federal Roles and Responsibilities	The NNISP pilot projects aligned with Health Canada's role in and responsibilities of improving the recruitment and retention of nursing staff through addressing issues of health infrastructure support. The NNISP also aligned closely with the department's role and responsibilities to support action on health status inequalities affecting First Nation communities.	Alignment with Federal Roles and Responsibilities 3. The NNISP was aligned with the federal roles and responsibilities of improving First Nations health by demonstrating innovative practices and approaches in primary care service delivery.
Performance	Achievement of Expected Outcomes	Although there was a lack of a standardized reporting system in place to assist with assessing progress towards outcomes, the evaluation identified innovative pilot projects that demonstrated progress was made to support collaborative teams, integration of nursing and technology, and improved access to continuing education and professional development for nurses. Collectively, these successes demonstrate movement towards achieving the expected immediate outcomes. Additional outcome findings include: 1. Some of the projects funded through the NNISP increased First Nations' access to primary care services in pilot communities. 2. Projects that had the goal of increasing the capacity of the primary care workforce generally reported that capacity had increased. 3. The NNISP contributed to increased access to primary care nursing education for practitioners in remote and isolated communities.	Performance of the Program Achievement of Expected Outcomes 4. The NNISP demonstrated progress towards the achievement of immediate outcomes. The innovative practices and approaches to primary care made by many of the pilot projects, combined with the lessons learned from this initiative, could be incorporated in future models of primary care service delivery.
		5. The NNISP contributed to the development of collaboration among health care providers and stakeholders but few formal partnerships resulted from the pilot projects. 6. Evidence-based information was only used for developing some of the NNISP pilot projects proposals but many projects attempted to collect and analyze project data.
	Demonstration of Efficiency and Economy	Despite under-spending the funds available to the NNISP, many of the pilot projects demonstrated program delivery efficiencies in providing primary care services.	Efficiency and Economy 4. Many pilot projects provided examples of ways to impact primary care service delivery efficiency that should be considered in future models for primary care delivery. Improved overall performance measurement data would better support reporting requirements and the conduct of future evaluations of primary care service delivery.

Management Response

Evaluation of the First Nations National Nursing Innovation Strategy Program (NNISP)

Evaluation of the First Nations National Nursing Innovation Strategy Program (NNISP)
Conclusions	Management Response	Further Considerations/ Comments/ Action
Continued Need The NNISP was appropriately designed to address specific challenges identified in remote and isolated First Nation communities through the use of innovation in primary care service delivery. There is a need for continued innovative approaches in primary care service delivery. Government Priorities The NNISP was aligned with federal government priorities and departmental strategic outcomes. While the NNISP concluded March 31, 2013, the federal government continues to prioritize the delivery of primary care services in remote and isolated First Nations communities. Alignment with Federal Roles and Responsibilities The NNISP was aligned with the federal roles and responsibilities of improving First Nations health by demonstrating innovative practices and approaches in primary care service delivery. Achievement of Expected Outcomes The NNISP demonstrated progress towards the achievement of immediate outcomes. The innovative practices and approaches to primary care made by many of the pilot projects, combined with the lessons learned from this initiative, could be incorporated in future models of primary care service delivery. Efficiency and Economy Many pilot projects provided examples of ways to impact primary care service delivery efficiency that should be considered in future models for primary care delivery. Improved overall performance measurement data would better support reporting requirements and the conduct of future evaluations of primary care service delivery.	Management agrees with the conclusions of this evaluation as this funding contributed to the testing of new models of primary care service delivery along with the strengthening of collaboration and partnerships in remote and isolated First Nations communities. The lessons learned from this evaluation will assist the overall goal of improving future quality and access to primary care through the Clinical and Client Care program. Nursing Innovation funding has resulted in a number of successful pilots that demonstrated improvements in primary care service delivery in First Nation communities and identified a number of efficiencies. FNIHB has committed to move forward to implement many of these new practices to improve primary care service delivery such as: the introduction of interdisciplinary collaborative practice teams; the introduction of technology; online training approaches; and the introduction of flexible nurse schedules. FNIHB will apply the many lessons learned from these pilot projects to inform future initiatives including recruitment and retention strategies and will work to build on this new information.	The NNISP contributed to policy initiatives such as the Health Services Delivery Model (HSDM), approved by the Branch in 2012 which aims to develop more effective approaches for Clinical and Client Care services in remote and isolated First Nation communities through a 3-5 year multi-pronged approach. The HSDM project implementation and transformation activities are delivered in collaboration with the regions in order to better respond to the evolving needs of the First Nations communities.

1.0 Evaluation Purpose

The evaluation covered the period from 2008-09 to 2011-12, prior to the completion of the NNISP, which ended March 31, 2013. The evaluation assessed the relevance and performance of NNISP.

2.0 Program Description

2.1 Program Context

Primary care delivery relies predominantly on nurses for care delivery. Socio-economic, demographic and geographical factors such as professional isolation, and adapting to a culturally appropriate health care approach further increases the complexity of primary care particularly in remote and isolated First Nation communities settings. Nurses play a central role in the delivery of health services in remote and isolated First Nation communities. As often the first and only point of contact, nurses require innovative approaches and adaptability within their regulated authority to manage ever changing needs including the use of medical technologies and changing infrastructure. As a result, the federal government committed funding for five years (2008-13) to pilot innovative practices and approaches under the NNISP .

2.2 Program Profile

The NNISP was an initiative to investigate new practices and approaches (innovation) to improve primary care services in remote and isolated First Nations communities. This included annual non-permanent funding, over five years, to pilot these innovations.

Proposals for NNISP pilot project funding were accepted under four streams, each focusing on different aspects of nursing innovation including:

Stream 1: Collaborative Teams involved projects that introduced new health care providers into existing nursing teams. It also involved the inclusion of new nursing staff such as nurse practitioners, licensed/registered practical nurses, and mental health nurses. It also involved the addition of allied health care professionals, such as x-ray personnel, pharmacy clerks, and midwives.
Stream 2: Integration of Nursing and Technology focused on incorporating certain technologies into health care delivery, such as telehealth, mobile devices, on-line tools (e.g., instructional websites), software applications, and pharmacy inventory/dispensing management systems.
Stream 3: New Hours of Operation involved extending the regular business hours in select nursing stations to improve access to walk-in health care.
Stream 4: National Education Strategy involved National and Regional Nursing Education Pilot Projects as well as Regional Nursing Education Activities that provided increased access to continuing education and professional development for nurses.

There were a number of stakeholders associated with the NNISP. Health Canada headquarters in the National Capital Region was ultimately responsible for the program, along with the regional offices of the First Nations and Inuit Health Branch (FNIHB). FNIHB senior management and program staff managed and coordinated the program. Health Canada regional offices, who assisted in the pilot project implementation, consisted of regional directors, regional nursing officers/directors, nurse educators, nursing staff, project managers/leads, and project staff.

The external partners were First Nations Bands and Tribal Councils in remote and isolated communities responsible for specific NNISP pilot projects. In addition, they may have had dedicated staff working with Health Canada on specific pilot projects.

2.3 Program Logic Model and Narrative

The program logic model, on the following page, depicts the results chain for the NNISP. The overall objective of the NNISP was "to promote innovation and partnership in health care delivery that better meets primary care needs" with the target group being First Nations living in remote and isolated First Nations communities. There were five main themes in the logic model: service provision; capacity building; stakeholder engagement and collaboration; data collection, research, and surveillance; and, policy development and knowledge sharing. Each main theme was linked to various outputs, which are intended to lead to immediate, intermediate, and long-term outcomes, as outlined in Table 2 (below).

Table 2: NNISP Logic Model
Theme	Service Provision	Capacity Building	Stakeholder Engagement and Collaboration	Data Collection, Research, and Surveillance	Policy Development and Knowledge Sharing
Objective	To promote innovation and partnership in primary health care service delivery
Target Group	First Nations on reserve in Remote and Isolated First Nations Communities
Outputs	New primary care service approaches/ models piloted	Nursing education and training activities offered and completed	Collaborative primary care teams established	Change management processes established	Analysis and recommendations that inform innovative practices in primary care service delivery in First Nations communities presented
Reach	First Nations on-reserve/in First Nations communities	Primary care nursing workforce in remote and isolated First Nation communities: NPs, RNs, LPNs	Internal primary care providers: RN, L/RPN, pharmacy/lab/x-ray techs., paramedics External primary care service providers	Internal stakeholders/partners External stakeholders/ partners including remote and isolated First Nations organizations	Internal stakeholders and partners
Immediate Outcomes	Improved access to primary care services	Increased capacity (knowledge, skills, and ability) of the primary care workforce Increased access to primary care nursing education for remote and isolated practice	Improved collaboration among health care providers and broader stakeholders	Increased use of evidence-based information to inform quality primary care service delivery
Intermediate Outcomes	Improved quality of primary care services accessed		Improved integration and coordination of primary care programs, services, and technologies	Implementation of continuous quality improvement for primary care service delivery
Longer Term Outcomes	Sustainable primary care services that are responsive to the needs of First Nation communities

2.4 Program Alignment and Resources

The NNISP contributes to Health Canada's Strategic Outcome 3: First Nations (and Inuit^{Footnote 1}) communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status. The Nursing Innovation sub-sub activity was identified in the department's Program Alignment Architecture 3.3: Health Infrastructure Support for First Nations and Inuit, and the sub-activity of Health System Transformation. The NNISP had approved funding of approximately $22M across the period of 2008-09 to 2011-12 for the funding of pilot projects.

3. Evaluation Description

3.1 Evaluation Scope

The evaluation covered the period from 2008-09 to 2011-12 and included all 52 NNISP pilot projects. Due to the short time-frame for funded project implementation (i.e., 1-2 years on average), the evaluation focused on the achievement of immediate outcomes.

3.2 Evaluation Issues

The evaluation considered the five core evaluation issues as per the 2009 Treasury Board Policy on Evaluation, under the two themes of relevance and performance. Specific questions were developed based on program considerations, and used to guide the evaluation process. Table 3 below presents the issues and questions addressed by this evaluation.

Table 3: Core Evaluation Issues and Questions
Core Issue	Evaluation Questions
Relevance #1: Need For Program	1.1: Does the program address a demonstrable need? 1.2: Is the program responsive to the needs of First Nation communities in remote and isolated communities?
#2: Alignment With Government Priorities	2.1: Does the program align with federal government priorities?
#2: Alignment With Government Priorities	2.2: Does the program align with departmental strategic priorities/outcomes?
#3: Alignment With Federal Roles and Responsibilities	3.1: Does the program align with departmental roles and responsibilities?
	3.2: Do the program's key stakeholders see the program's activities as relevant and aligned to its roles and responsibilities?
	3.3: Are the program's activities aligned with the department's jurisdictional, mandated, and/or legislated role?
Performance #4: Achievement Of Expected Outcomes	4.1: Has the program achieved its immediate outcomes?
#5: Demonstration of Efficiency and Economy	5.1: Has the program demonstrated resource utilization in relation to the production of outputs and progress toward expected outcomes?

3.3 Evaluation Approach

The evaluation used an outcome-based approach to assess the progress made towards the achievement of immediate outcomes. The approach included collaboration with key internal and external stakeholders in the development of the evaluation framework, conduct of the evaluation, review of technical data as well as the evaluation report and management response.

3.4 Evaluation Design

This evaluation used a non-experimental and retrospective design. The evaluation was non-experimental because evidence on the progress toward the achievement of expected outcomes was observational in nature. Furthermore, not only did the evaluation require a retrospective design because the data was based on past years of pilot project funding (2008-09 through 2011-12) but also because there was an absence of baseline data.

3.5 Data Collection and Analysis Methods

The evaluation involved four main data collection methods: a document and data review (see additional information in Appendix 1); key informant interviews and focus group discussions; a brief survey of First Nations representatives (hereafter referred to as the Community Survey); and a review of project proposals and reports. A review of program-level financial data was used to complete the analysis of resource allocation and utilization. Additional resources are listed in Appendix 2.

Involvement of key stakeholders in the evaluation was achieved through a series of individual and group interviews, and telephone focus groups. Community participants involved in the pilot projects were identified by regional staff who participated in a telephone survey.

The collected data was analyzed using the following methods:

a systematic review of data extracted from the documents, which included the creation of summary table and led to the formulation of conclusions based on the summary data;
statistical analysis of quantitative data, which involved the creation of charts summarizing the results;
qualitative data (from key informant interview/focus group questions) analysis using a thematic analysis technique, where responses were systematically reviewed and emergent themes were identified and categorized; and
comparison of data from document reviews and stakeholder surveys to synthesize data from disparate sources, and validate information as part of the findings of this assessment.

3.6 Limitations and Mitigation Strategies

Challenges are inherent in any evaluation and limitations can impact evaluation findings. As such, mitigation strategies are used to ensure that the data collected produces a credible evaluation report with evidence-based conclusions and recommendations.

The following table outlines the limitations, their impact/potential impact on the evaluation and the mitigation strategies employed in this evaluation to limit these impacts.

Limitation	Challenge	Mitigation Strategy
Performance Data Lack of consistent, standardized performance data on expected outcomes in pilot project reports Limited departmental financial data	Much of the information contained in the individual pilot project reports (mid-year and annual) contained only anecdotal information on outcomes. Also, despite the fact that FNIHB provided a reporting template, information tended to be inconsistently reported across the individual pilot projects. Furthermore, few projects completed formal evaluations, although several are planned to occur after the conclusion of the NNISP. Lack of financial object costing data does not allow for a full assessment of economy and efficiency.	Additional review of proposals and project reports was required to determine/find and assess the level of outcome information and plans for evaluating projects. A focused assessment of resource allocation and utilization was included in the evaluation using only approved data provided by the Branch Senior Financial Officer.
Key Informant Interviews Challenges accessing project-level key informants	The design included 10 telephone group interviews. Difficulties with scheduling groups made this approach too difficult to manage. Consequently, this method did not provide a sufficient number and range of key informants. The Community Survey was unable to reach a large enough group of respondents to be statistically significant.	Additional interviews with Regional Directors and Regional Nurse Directors were added during the data collection phase. Group interviews were conducted with Regional Nurse Educators and the Steering Committee. Additional interviews with senior management/executive directors were conducted individually or in pairs. Despite a small sample (N = 37), it was selected purposefully with interviews and surveys providing a relatively high response rate (61%). The survey provided observations of community members on the achievements of the pilot projects.

Limitation

Challenge

Mitigation Strategy

Performance Data

Lack of consistent, standardized performance data on expected outcomes in pilot project reports

Limited departmental financial data

Much of the information contained in the individual pilot project reports (mid-year and annual) contained only anecdotal information on outcomes. Also, despite the fact that FNIHB provided a reporting template, information tended to be inconsistently reported across the individual pilot projects. Furthermore, few projects completed formal evaluations, although several are planned to occur after the conclusion of the NNISP.

Lack of financial object costing data does not allow for a full assessment of economy and efficiency.

Additional review of proposals and project reports was required to determine/find and assess the level of outcome information and plans for evaluating projects.

A focused assessment of resource allocation and utilization was included in the evaluation using only approved data provided by the Branch Senior Financial Officer.

Key Informant Interviews

Challenges accessing project-level key informants

The design included 10 telephone group interviews. Difficulties with scheduling groups made this approach too difficult to manage. Consequently, this method did not provide a sufficient number and range of key informants.

The Community Survey was unable to reach a large enough group of respondents to be statistically significant.

Additional interviews with Regional Directors and Regional Nurse Directors were added during the data collection phase. Group interviews were conducted with Regional Nurse Educators and the Steering Committee. Additional interviews with senior management/executive directors were conducted individually or in pairs.

Despite a small sample (N = 37), it was selected purposefully with interviews and surveys providing a relatively high response rate (61%). The survey provided observations of community members on the achievements of the pilot projects.

4.0 Findings

4.1 Relevance: Issue #1 -- Need for the NNISP

The NNISP was required in order to find innovative ways to address the provision of primary care services in remote and isolated First Nation communities, including addressing the challenges of recruitment, retention and training of nursing staff and allied health professionals.

First Nations are distinct in terms of health issues. They experience higher rates of certain acute and chronic diseases than the Canadian population overall and, therefore, have different health care delivery needs than the rest of Canada.

Health and Health Care Needs

Both the document review and the key informant interviews found evidence to suggest the NNISP was designed to address a number of pertinent needs for the health of First Nations. Several studies have demonstrated the unique health-related challenges faced by First Nations communities. The document review indicated that the NNISP was developed in response to these challenges. Accordingly, the NNISP responded to the health-related needs of First Nations through the delivery of health care services in remote communities with innovative projects. Key informants identified several health-related needs and challenges of delivering services in remote and isolated First Nation communities that had an impact on the development of funded innovative projects. This indicates that the NNISP is relevant to these communities.

First Nations living on-reserve are distinct from other Canadians in terms of health issues. According to Statistics Canada, they experience higher rates of certain diseases than the Canadian population overall (Statistics Canada, 2007). Discrepancies in disease rates, in turn, are often linked to socio-economic factors such as income, education, and employment levels -- all important determinants when it comes to health.

In addition, according to the 2006 Canadian Community Health Survey, 62% of the Canadian population overall (including adults aged 15 years and older) reported having excellent or very good health, whereas 53% of First Nations adults living off-reserve reported the same (Statistics Canada, 2010). In 2002-03, 88.0% of the general Canadian population reported having "good," "very good," or "excellent" health, whereas 79.7% of First Nations living on-reserve reported the same (Health Canada, 2009, p. iii). Many key informants also identified health-related challenges for remote and isolated First Nation communities, including chronic conditions (cardiovascular disease, respiratory disease, diabetes, hypertension, kidney disease, HIV, tuberculosis, etc.), mental health issues as well as addictions and drug abuse.

Another significant challenge related to First Nations health is that health care delivery in remote and isolated First Nation Communities is substantially different from health care delivery in the rest of Canada (FNIHB, 2011; Misener et al., 2008; Smith, 2010). Not only do these communities have greater reliance on nurses in the delivery of primary care, but the role of nurses working in these areas is different from that of nurses working elsewhere in non-isolated communities. Key informants said that remote and isolated First Nation communities often lack support from primary care services and emergency care. Other factors, such as geography, isolation, and cultural dynamics make nursing in remote communities a complex and challenging practice. Often, these challenges can lead to high turnover rates and shortages of nurses. Studies on nurses providing primary health care in remote and isolated communities have noted that nurses need to be provided with additional competencies to practice in remote and isolated First Nation communities as well as supplementary education that pertains to the practice context in these communities (FNIHB, 2011; Misener et al., 2008; Smith, 2010).

Given these challenges, the document review confirmed several needs for the NNISP to address including: increasing the supply of health care service providers; improving accessibility of care and capacity to deliver care within remote and isolated First Nation communities; making 24/7 nursing stations available for walk-in access after regular business hours; and, providing education to equip nurses with the knowledge and skills required to practice in remote and isolated communities (ONS, 2010b, p. 10).

Key informants identified specific needs that they felt the NNISP was designed to address, such as: improving access to primary health care services; enhancing available primary care services and supports; and, providing communities with the opportunity to try new, innovative approaches to health services delivery with the goal of supporting a new model of primary care service delivery in remote and isolated communities.

Based on the document review and responses from key informants, there is a continued need for nurses and allied health care professionals who are trained to work in remote and isolated First Nation communities. It is clear that health care delivery in remote and isolated First Nation communities, where nurses are expected to provide a wide range of primary care services is substantially different from delivery in less isolated communities. Health systems in remote and isolated First Nation communities are faced with many challenges, such as disproportionately high rates of chronic disease, communicable disease, and substance abuse. Health care staff members working in remote and isolated First Nation communities are also often faced with other types of challenges, such as geographical isolation, resource shortages, and provision of a culturally appropriate health care approach.

Responsiveness to the Needs of First Nation Peoples in Remote and Isolated Communities

The key informant interview results indicated that the overall design of the NNISP as well as the design of specific NNISP-funded projects was appropriate to meet the needs of those targeted by the program.

With regard to overall program design, key informants mentioned that regional stakeholders were involved in the early stages of the NNISP's development, which was essential to ensuring the program was appropriately designed. A few mentioned the literature on innovation in primary care service delivery was consulted. Other key informants, mainly education project representatives, noted that education activities were designed to meet the needs of participants in terms of primary care competencies and to respond to the particular needs of health care workers in remote and isolated First Nation communities (e.g., by offering distance or online education in addition to, or instead of, providing on-site, hands-on skill training). A few key informants also said that, since the program focuses on these communities, projects were appropriately targeted in communities that have the least access to primary care. Overall, many key informants agreed that the program was appropriately designed to meet identified health needs of First Nations living in remote and isolated communities.

As for the design of NNISP pilot projects, key informants identified ways in which specific projects were designed and uniquely tailored to meet specific identified health care service needs of First Nations communities. Some examples included projects involving task shifting/collaborative practice approaches that focused on addressing specific health concerns relevant to remote and isolated First Nation communities, such as diabetes, hypertension, and kidney disease. Other projects focused on hiring health care workers (such as mental health nurses and nurses specializing in diabetes) who were needed in certain communities.

While key informants were generally satisfied with the appropriateness of the program's design, some identified implementation issues that may have affected the NNISP's ability to meet certain goals and address certain identified needs. A few respondents said they felt some projects were not innovative; however, this does not necessarily take into account regional differences, where standard practice in one region may be considered innovative in another. Other key informants mentioned challenges related to the timely release of funding to regions, the lack of sustainability, and the short time period for the program design, development and implementation of the project funding process, which made undertaking and achieving sustainable innovative projects difficult.

For the most part, the NNISP was designed to meet actual needs of remote and isolated First Nation communities related to health care delivery and resource challenges. The document review showed that the NNISP was designed to address specific challenges identified in remote and isolated First Nation communities. Key informants also mostly agreed that the design of the NNISP factored in findings from the literature, and consulted appropriate stakeholders early in the design process. Several informants also mentioned consultations occurring in the development of individual pilot projects. Overall, these factors supported the relevance of the NNISP.

4.2 Relevance: Issue #2 -- Alignment with Government Priorities

The NNISP aligned with the federal government and departmental strategic priorities, particularly with respect to its emphasis on innovation in primary health care delivery, collaborative health teams and increased access to primary care services. It also aligned with the priority of ensuring that remote and isolated First Nation communities are receiving responsive health services and benefits in order to improve their health status.

Alignment with Federal Government Priorities

The document review indicated that the federal government intended to use the NNISP to help align First Nations health care systems with those of the provinces. The core concept is that investment in health care innovation would lead to structural change in the long run. Some key informants discussed pilot projects that partnered with regional health authorities, but it was not clear from the interviews what these efforts achieved.

Although there is no specific mention of the NNISP in the 2012-2013 Report on Plans and Priorities, it describes how Health Canada intends to address its strategic outcomes and related priorities for First Nations health-related programming in order to improve the health outcomes of First Nations peoples (Health Canada, 2012a, pp. 38-39, 42). The 2012-13 Report on Plans and Priorities suggested that the federal government will continue to prioritize improving the health care system, addressing health inequalities for First Nations, increasing access to health care, and encouraging collaborative health teams, which are all central aspects of the NNISP.

It is clear from the document review that the federal government prioritizes health care service delivery in remote communities and continues to support a host of related programs, including the ongoing provision of primary care services, even though funding for the innovation pilot projects ended in March 2013.

Alignment with Departmental Strategic Priorities and Outcomes

The document review and key informant interviews found that the NNISP aligned with the strategic outcomes of Health Canada and the mandate of FNIHB.

In terms of Health Canada's strategic outcomes, documents and key informants indicated that the NNISP aligned closely with Strategic Outcome 3: First Nations communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status (Health Canada, 2012a, p. 37). This strategic outcome seeks to increase access to health care services in First Nations communities and also aims to bridge the gap in health status between First Nations and the rest of Canada.

The document review found that the NNISP also aligned with the mandate of FNIHB, which is consistent with Health Canada's overall approach to health care service delivery in remote and isolated First Nation communities. FNIHB focuses on improving access to health care services, including remote communities, and increasing the health status of First Nations (Health Canada, 2005), which were central activities for the NNISP.

Both the document review and key informant interviews indicated strong links between the NNISP and Health Canada's Strategic Outcomes. The NNISP also appears to have aligned with the mandate of FNIHB, which involved ensuring access to health care services and addressing health barriers in remote and isolated First Nation communities.

4.3 Relevance: Issue #3 -- Alignment with Federal Roles and Responsibilities

The NNISP pilot projects aligned with Health Canada's role in and responsibilities of improving the recruitment and retention of nursing staff through addressing issues of health infrastructure support. The NNISP also aligned closely with the department's role and responsibilities to support action on health status inequalities affecting First Nation communities.

The document review found that the NNISP aligned most closely with three program activities and the priorities of FNIHB.

Human resource issues are a focus of Program Activity 3.1 (First Nations and Inuit Primary Health Care). According to Health Canada's 2012-13 Report on Plans and Priorities (p. 38), Health Canada planned to address multiple human resource issues related to nursing innovation. The priorities included supporting professional practices for nurses in remote communities; addressing recruitment and retention challenges; facilitating consultation services for nurses; and investing in health care innovation to help bridge the health status gaps between First Nations communities and the rest of Canada. The NNISP's focus on innovation, recruitment and retention, and health care service delivery meant the program aligned with Health Canada's Program Activity 3.1 for the 2012-13 fiscal year. An important part of the NNISP's alignment with this Program Activity was the program's focus on community-level initiatives and health care innovation.

The NNISP's approach to innovative projects was aligned with a key performance indicator for Program Activity 3.3, that is, addressing health infrastructure support. This was identified in Health Canada's 2012-13 Report on Plans and Priorities (p. 16) which indicated that health system innovation, sustainability, efficiency, effectiveness, and accountability are part of the department's main priorities. While this Program Activity does not apply exclusively to First Nations, it does relate to some of the key aspects of the NNISP including health system transformation. For example, one of the performance targets of this Program Activity is the adoption of new health care models and best practices, with which the goals of the NNISP aligned.

In terms of Health Canada's Organizational Priorities, the document review found the NNISP aligned with two in particular: Promote Health System Innovation (#1) and Strengthen First Nations [and Inuit] Health Programming (#3) (Health Canada, 2012a, pp. 5-6). Two important aspects of Organizational Priority #1 are encouraging innovation and developing long-term sustainability of health systems. However, key informants were divided on whether NNISP projects were truly innovative. Some said they were innovative in the sense that they had not been done before, whereas others said the pilot projects were not very different from "business as usual". Many key informants did identify aspects of particular projects they felt were innovative, such as the employment of nurse practitioners in remote and isolated First Nation communities; the provision of electronic/mobile devices to nursing staff; the integration of licensed nurse practitioners and/or registered practical nurses into primary care teams; the use of telehealth technology; and agreements with some Provincial Health Authorities.

From the document review, the key aspects of Organizational Priority #3 (Strengthen First Nations [and Inuit] Health Programming) that are related to the NNISP include encouraging innovation, implementing interdisciplinary health care teams, and ensuring quality service delivery. Key informants generally agreed that the NNISP did align with strengthening First Nations health programming; however, a few key informants said that the NNISP was more specifically about strengthening service delivery than indirectly strengthened health programming.

The NNISP also aligned closely with the FNIHB priority to "support action on health status inequalities affecting First Nations and Inuit communities, according to their identified priorities" (Health Canada, 2005). Arguably, it was the NNISP's focus on the community level that aligned the program with this FNIHB priority.

4.4 Performance: Issue #4 -- Achievement of Immediate Outcomes (Effectiveness)

Although there was a lack of a standardized reporting system in place to assist with assessing progress towards outcomes, the evaluation identified innovative pilot projects that demonstrated progress was made to support collaborative teams, integration of nursing and technology, and improved access to continuing education and professional development for nurses. Collectively, these successes demonstrated movement towards achieving the expected immediate outcomes.

Immediate Outcome #1: Improved Access to Primary Care Services

Some of the projects funded through the NNISP increased First Nations' access to primary care services in pilot communities.

NNISP pilot projects have:

provided access to Nurse Practitioner consultations, diagnosis support and electronic treatment orders to support nurses in remote practice settings;
provided opportunities to more closely align with provincial primary care models to facilitate ongoing collaboration and integration with provincial healthcare systems;
improved access through technology by making more services accessible by distance;
added new staff members to primary care teams as part of pilot projects, which provided additional services and balanced the workload with existing team members, allowing all team members to work to their full capacity;
improved access to specific services needed by community members (such as eye, kidney, and blood glucose screenings for diabetic community members, and cancer screening, wound care pilot project, etc.); and
expanded the hours of service which provided increased access in one jurisdiction.

A few interviewees noted that improving access to primary care services for community members is not a direct goal of all NNISP projects. Those involving implementation of computerized records systems or distribution of personal electronic devices, for example, focused instead on improving the organization of, and access to, information for health care workers. Therefore, expected improvement in access to and/or quality of primary care services could be an indirect result of better work organization.

Immediate Outcome #2: Increased Capacity (Knowledge, Skills, and Ability) of the Primary Care Work Force

Projects that had the goal of increasing the capacity of the primary care workforce generally reported that capacity had increased.

NNISP pilot projects have:

increased the capacity of Nurse Practitioner services through the use of technologies to allow remote consultation and treatment, thereby reducing client travel costs;
reduced reliance on external contracting agencies through the introduction of paramedics and the expansion of nurse resource teams;
piloted flexible hours of operation in a nursing station to reduced overtime, meet community needs, and increased nurse satisfaction;
introduced hand-held technologies to provide clinical care information that supports patient care and education, test results as well as the reduction of wait-times; and
introduced pharmacy technicians and pharmacy inventory management systems to address the need for client medication management and improve processing of client prescriptions.

For projects where there was an addition or reallocation of resources, project reports often noted that there was an increased capacity to deliver services to the community, and that the community appreciated the enhanced services. In some cases, hiring additional staff allowed for a reduction in overtime worked by existing staff. Reports indicated that this offset the cost of the additional resources somewhat and also lowered stress levels of staff from working extended hours to maintain service levels, thus improving overall capacity of the health care team.

Key informants identified that collaborative practice and technology have removed some pressure from primary care service providers, allowing them to focus more on patient care. New staff members (such as nurse practitioners) added to project teams have increased team capacity, and new technologies have improved the knowledge levels of staff (e.g., by improving access to information).

Surveys of project staff, included in the pilot project annual reports, indicated that improvements in knowledge, skills and/or ability were noticed following specific training. In some projects, increased capacity was not fully realized, as positions were difficult to fill. Some challenges included a lack of qualified candidates and the need to refill positions due to high turnover rates. Often this required re-training for some specialized positions affecting the time a project had to reach its goals. Additionally, administrative issues and community sensitivities affected the hiring of staff for project positions. A few key informants also said there was occasionally some resistance from existing primary care staff to changing their work practices to accommodate a nurse practitioner, which led to duplication of work.

In the Community Survey, 84% of respondents said that nurses were better able to provide services compared to three years ago, while 11% did not agree that nurses were better able to provide services. In addition, 76% said that the methods used in the delivery of information and services in their pilot project increased training and competency, but 22% did not agree that the pilot project increased training and competency. It is worth noting that not every NNISP pilot project was intended to increase capacity, which might explain some of the 22% who disagreed.

Immediate Outcome #3: Increased Access to Primary Care Nursing Education for Remote and Isolated Practice

The NNISP contributed to increased access to primary care nursing education for practitioners in remote and isolated communities.

The NNISP pilot projects increased the number of nurses trained with remote practice competencies through access to new education programs and increased access to on-line education and training programs. According to the annual regional nursing education reports between 2010 and 2012 from each of the six regions (BC, AB, SK, MB, ON, and QC) and the evidence in the individual education project reports, the NNISP contributed to increased access to primary care nursing education for isolated and remote practitioners. This included both the Regional Nursing Education Activities and the National/Regional Education Project Pilots.

In many cases, training was delivered through online courses or video conferencing that nurses obtained over the Internet through a secure web portal. This offered nurses the option to complete the training at their convenience and/or remotely, and allowed those who did not have easy access to educational facilities to take the training offered. A few key informants noted that online courses reduced travel costs and allowed more nurses to participate. However, both the documents and a few key informants indicated that the remoteness of some communities made it difficult to access online training, as high-speed Internet was often not available. Under these circumstances, other forms of distance training, such as "e-learning," were employed (e-learning uses pre-recorded videos to train staff). This substituted for less accessible forms of training, but it may have also impacted the students' ability to ask questions or receive feedback on issues that might have been locally relevant, but not covered in the core course material.

In a few instances, the reports noted that obtaining certification from a recognized educational facility for specific training courses was challenging or did not occur at all. One report mentioned that differences in university programs and provincial legislation impacted whether the earned university credits in one province are transferable to a different institution in another province. Although this issue was resolved, it did create concern among those affected.

In the Community Survey, 68% of respondents agreed that access to education or professional development opportunities for nurses occurred due to the pilot project in their community. The remainder (32%) did not agree this was the case. Some of the variation might be explained by the fact that not all NNISP pilot projects were intended to improve access to education or professional development opportunities.

Immediate Outcome #4: Improved Collaboration among Health Care Providers and Broader Stakeholders

The NNISP contributed to the development of collaboration among health care providers and stakeholders but few formal partnerships resulted from the pilot projects.

NNISP pilot projects have:

addressed care delivery challenges through the implementation of collaborative teams involving a mix of health care providers (e.g., nurse practitioners, licensed/registered practical nurses, pharmacy technicians, midwives and paramedics);
distributed primary care services among a team of health care providers, allowing nurses to focus on nursing services; and
provided access for prenatal clients through Nurse Practitioner Midwifery clinics.

While many of the pilot project reports mentioned improved collaboration among health care providers and stakeholders, only a few noted partnerships with specific organizations and none mentioned formal agreements to solidify these partnerships. Most often, the reports noted that there was increased cooperation between health care practitioners within or outside communities as a result of the pilot project. There was also mention of better communication between health care workers and their clients, and this led to better services for patients and other individuals seeking services within and outside of the community. In several cases, the reports indicated that there were regular meetings between various stakeholders to better coordinate services and plan implementation of activities aimed at providing enhanced services to their clients. The Community Survey also indicated improved collaboration, with 81% of respondents agreeing there had been more collaboration among health care providers compared to three years ago.

Many key informants mentioned attempts to involve First Nations leaders in the design and delivery of projects. However, there was variation on the success of these attempts: some project managers said there was substantial community involvement, whereas a few project managers said that community members did not always attend planning or information sessions, or did not become involved as much as they would have liked.

According to the Community Survey, First Nations partners appeared to be very involved or somewhat involved in the design (63%), implementation (57%), and delivery (54%) of the pilot projects. Few respondents indicated they were not at all involved; however, about one third did not respond, either because they were unsure or did not know. For several projects, the project reports mentioned that having the First Nation community involved in the project was crucial for moving forward, and that this community involvement (collaboration) helped to develop additional services for residents.

In general, key informants seemed to agree that many projects improved collaboration among health care providers and between health care providers and community members. Some NNISP streams were more focused than others on collaboration. Key informants provided specific examples of collaboration such as:

licensed practical nurses hired through NNISP projects have worked closely with other health professionals (primary care nurses, doctors, etc.) to deliver services and help to bridge gaps in service provision (e.g., by working to support agency nurses by filling in during times of staff turnover);
nurse practitioners hired through NNISP projects have worked to consult with nurses already working in a community, benefiting from the additional skills and expertise brought by the nurse practitioner; and
nurse practitioners have worked with community health nurses and have undertaken a community development role.

Key informants also said that some NNISP projects facilitated broader stakeholder partnerships with regional health authorities, provincial health organizations, hospitals, and national health-related organizations. However, some key informants mentioned barriers to improved collaboration, particularly among service providers. It is clear that attempts to connect some remote communities to provincial health systems encountered problems, largely technical in nature. For example, Health Canada staff did not have access to a national electronic database which complicated the potential for data sharing.

Immediate Outcome #5: Increased Use of Evidence-based Information to Inform Quality Primary Care Service Delivery

Evidence-based information was only used for developing some of the NNISP pilot project proposals but many projects attempted to collect and analyze project data.

The logic model indicates that the NNISP supports the achievement of increased use of evidence-based information to inform how pilot projects were developed, designed and implemented in order to inform the delivery of quality primary care service.

Based on the review of all proposals, it is evident that evidence-based information (in the form of literature reviews; needs assessments; established frameworks, best practices, and competencies; consultations with health care professionals, researchers, and IT experts; consumer reports; statistical data; etc.) was only used for developing some (64%) of the NNISP projects. A few proposals involved academics, and typically, these proposals drew on scientific evidence, created clear work plans, and identified measureable outcomes. This detail in the proposal tended to carry through to the detail in the report, which usually used more than just anecdotal evidence of outcomes.

The results from the interviews were different from the project review. Key informants indicated that the development of the NNISP and its pilot projects was largely influenced by evidence-based information in the context of project design. Almost all key informants representing educational projects mentioned that the projects were developed using established evidence, such as literature reviews, established frameworks, best practices, consultations with health care professionals, and other sources.

Many key informants who were project managers and/or team members mentioned that their projects were based on research evidence and established guidelines/conventions. That being said, the interviews suggested that not all projects considered evidence-based information(e.g., a few project managers mentioned their projects were based on recognized needs in the communities).

In terms of the projects themselves, based on the document review and interviews, there is evidence that project staff members attempted to collect and analyze project performance data, although most reports provided primarily anecdotal or qualitative evidence. In several projects, opinions from stakeholders or health care providers were obtained casually. Discussions of these findings in the project reports, while not highly detailed, did provide a sense of the level of success for the projects. Whether the project recorded outcome data in a systematic fashion or informally, there did appear to be analyses and actions based on the information collected, as projects were often modified or re-focused when there were difficulties in achieving their objectives.

Challenges and/or Barriers Encountered

At the program level, key challenges were identified such as the short time frame for reviewing project proposals, staff changes that altered the overall program management approach, and challenges with balancing the project streams.

According to key informants, measures required to more effectively address program-level issues include: establishing clearer guidelines and standards to guide the proposal review process, achieving more timely release of program funding as well as achieving better collaboration between Band and Health Canada IT staff on technological initiatives implemented in transferred communities.

Project reports indicated and key informants said that because of the remoteness of the communities, technology such as access to reliable high-speed Internet often used in communications, training, and evaluations was frequently not available to health care providers. In some cases, especially for training-related projects, this was overcome by offering pre-recorded training videos distributed to nurses in remote locations where high-speed Internet was not readily available. Respondents to the Community Survey identified a wide variety of challenges, but the most common factor was difficulty with integrating technology into primary care service delivery.

Aside from technological barriers, there were often human resource challenges in the recruitment and retention of suitable staff for positions created by the project. The remoteness of the positions and the often high workloads contributed to staffing challenges. Key informants noted that there was continuous turnover and a lack of time to keep training new staff members. Several project reports also cited health crises such as the H1N1 epidemic as needing priority attention, thus contributing to difficulties in meeting timelines (i.e., hiring suitable candidates), and as a result delaying the project implementation. In these cases, once the crisis was over, there was more progress made in hiring qualified staff.

Key informants also identified some unexpected challenges related to achieving a collaborative approach to primary care service provision. Some key informants did not expect the level of resistance that they encountered to newly hired health professionals among health care staff already servicing a community. Furthermore, a few key informants reported that they did not expect that differences in provincial education standards/credit recognition between institutions would deter nurses from taking part in NNISP educational initiatives to the extent that they did.

Where the challenges were too great to overcome, six projects were cancelled, with two of the six project reports mentioning difficulties with engaging the community and provincial stakeholders in the project's implementation that contributed to the cancellation. When a report indicated that an alternative approach was devised, but did not succeed, a lack of resources was usually the reason cited.

Unintended Results and/or Consequences as a Result of Program Implementation

The evaluation found that there were raised expectations of sustainability, that is, the pilot projects have raised community expectations for services that may not be sustainable once NNISP funds expire.

Many key informants (representing all stakeholder groups) expressed concern about this expectation, with several project funding committee members suggesting that the fact that NNISP projects would not be receiving federal funding after March 2013 may not have been as clearly articulated to the regions as it could have been. Some project proponents are exploring funding support through provincial systems.

4.5 Performance: Issue #5 -- Assessment of Economy and Efficiency

Despite under-spending the funds available to the NNISP, many of the pilot projects demonstrated program delivery efficiencies in providing primary care services.

The Treasury Board Policy on Evaluation (2009) defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment of economy and efficiency is based on the assumption that departments have standardized performance measurement systems and that financial systems use object costing.

Given the lack of departmental financial data that is linked to the quantity and type of outputs coupled with incomplete program-level expenditure reporting, the evaluation could not conduct an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Further, there were changes in financial coding across the evaluation period and challenges in tracking innovation funding within the broader First Nations primary care service delivery envelope.

Considering these issues, the evaluation provided observations on economy and efficiency based on limited financial data, findings from the document review as well as key informant perspectives.

Observations on Economy

This section provides a review of resource expenditures, resource utilization in terms of approved funding levels versus actual expenditures (i.e., direct and indirect; including salary, operating/ maintenance and contributions for funded projects), including regional differences in overall expenditures. It also provides a summary of factors that impacted overall expenditures for the NNISP.

Funding for the NNISP program was approved for five years (2008-09 through 2012-13). No expenditures were recorded against the NNISP during the initial year (2008-09) as funding was only allocated from the department in December 2008. In addition, no program-level financial coding for the NNISP had yet been created in order to track expenditures. It should be noted that $1.5M was allocated in the first year to regions to support consultations with First Nation partners with the aim of identifying potential innovation projects and project priorities as well as to support proposal development. Given that the NNISP Steering Committee did not require regions to report on these funds, these were not counted as expenditures specific to the NNISP (but to primary care generally)and were not included in the analysis. The evaluation did not include any expenditure information for the final year given the timing of the evaluation (underway during 2012-13).

Expenditures^{Footnote 2} were modest in 2009-10 and began to increase in 2010-11 with the majority of expenditures occurring in 2010-11 and 2011-12. Although expenditures increased over time, the review found under-spending by the program. Of the $22,270,441^{Footnote 3} approved by the Government of Canada for this program between 2008-09 and 2011-12, only $17,276,279 (78%) was allocated by the department for expenditure in the NNISP program. Of those funds, the program expended $12,862,561 (74% of funds allocated and 58% of funds approved).^{Footnote 4} There was no information provided to track the re-allocation of the funds.

The review also identified several factors that influenced the overall resource utilization of the NNISP (i.e., ability of the program to spend resources).

The 2009-10 fiscal year represented the first full year of NNISP project start up. However, the H1N1 crisis prevented the program from utilizing all available funding and anticipated unspent funds were re-allocated within FNIHB to address surge capacity and the outbreak throughout the regions. In addition, at this time, there was a general government freeze on public servant travel and general unanticipated spending reductions by FNIHB that prevented the program from expending all allocated funding.
It was noted by key informants that there was insufficient human resource capacity within regions to implement projects which impacted expenditure levels, that is, the ability of the NNISP to spend available resources.
Delays in contracting and negotiating with third parties (universities) created challenges in spending the allocated funds, which resulted in surpluses each year.
The lack of technology infrastructure and delays in the procurement of software influenced the degree to which the program could fully utilize funding overall.

For the years included in the evaluation, expenditures by regional and national offices varied greatly, with the greatest percentage of expenditures in Manitoba (38%), Headquarters (15%), Ontario (14%), British Colombia (12%), and Saskatchewan (10%). Quebec accounted for 6% of total expenditures with only 1% in the Atlantic region.^{Footnote 5}

Each region, with the exception of the Atlantic region, funded, on average, six projects. However, key informants indicated that project expenditures were greater than indicated by this contributions (funded projects) financial breakdown (i.e., some projects incurred unanticipated costs and most regions where this occurred internally reallocated funds to address the added costs). For example, operating costs often included project-related expenditures such as contracts for project staff, project supplies and equipment as well as facility costs is part of this category. Further, salary and wage expenditures included those of project staff who were hired to either manage or implement funded projects within the remote and isolated communities.

Observations on Efficiency

This section provides a review of the factors that impacted program delivery as well as examples of pilot projects demonstrating program delivery efficiency.

The distribution of projects and resources was determined by the project approval process. As such, funds were distributed based on successful project proposals. In total, the NNISP funded fifty-two projects, most lasting two years on average.

Although 53 projects were funded and many demonstrated progress towards the achievement of immediate outcomes, key informants identified issues that impacted the efficiency of overall program delivery.

There were delays in the approvals of projects by the Branch, which affected the timing of implementing projects, and thus the overall distribution of projects. As a result, some projects were not able to proceed due to the delay in funding allocations and/or funding cycles for contribution agreements within the communities. This impacted the overall efficiency of program delivery.
Many reported that there were difficulties in recruiting and/or retaining staff for short periods of time across the five years of the NNISP project funding, including negotiating contracts with agencies to find and provide qualified staff, particularly for remote and isolated communities, which impacted the efficiency of project implementation. Members of the Steering Committee identified that recruitment and hiring challenges were compounded by challenges encountered with paperwork tasks (security clearances, medicals, etc.) and completion of these tasks was often prolonged for those working in remote and isolated communities. These delays in staffing resulted in delays in project implementation impacting the efficiency of overall program delivery.
A change in implementation approach occurred in the second year of the program (2010-11) when funding allocations became project-based instead of the previous approach of percentage funding by region, with regions submitting funding proposals to be reviewed by a Steering Committee and approved by the Branch ADM. This change was implemented to ensure projects that were likely to be successful received funding and were appropriately resourced. However, this resulted in delays in getting projects funded and disqualified some regions (i.e., Atlantic), where FNIHB does not deliver primary care to any remote and isolated communities impacting the efficiency of overall program delivery.

Key informants provided examples of ways in which NNISP pilot projects improved the efficiency of primary care service delivery. From a human resources perspective, the following examples were noted:

the hiring of nurse practitioners to be 'on call', and adhering to their regulated authorities provided access to expanded services at less cost than hiring a physician;
initiatives that increased access to services in the community also reduced the cost of medical transportation, such as collaborative initiatives that brought in additional primary care staff to work in the community and integrated nursing technology initiatives allowing community members to receive services at a distance;
initiatives that improved the satisfaction levels and decreased the stress and administrative workloads of nurses and physicians helped to improve retention (e.g., in some pilot communities, the hiring of new staff, including pharmacy technicians, licensed practical nurses, etc., reduced the administrative burden on nurses and allowed them to focus more on patient care);
the hiring of licensed practical nurses trained to assume a range of duties offered a cost-effective way of providing services; and
the use of online technologies for education and training initiatives offered through the NNISP (such as in the University of Ottawa pilot project) decreased costs and allowed for more participation of nurses, however, key informants indicated that some remote and/or isolated communities still struggle with internet access and therefore, online educational opportunities may not improve access for nurses in all regions.

From a financial management perspective, the following example of overall program efficiency was provided:

Overall program expenditures were monitored quarterly to track expenditures across all the four streams and in some cases, project numbers were used to track expenditures by project, although this was tracked inconsistently. Nonetheless, funds were reallocated when projects ran into difficulties in order to maximize efficiency.

Observations on the Adequacy of Performance Measurement Data

The lack of a standardized reporting system to assess outcomes for the NNISP funded projects hindered the ability for projects to provide information that would support the evaluation, in particular, data to support the achievement of all the expected outcomes outlined in the NNISP logic model. For example, the reporting templates for the NNISP generated vastly disparate information and did not consistently provide data on outcome measures. Improved overall performance measurement data would better support the conduct of evaluations for these programs.

5.0 Conclusions

5.1 Relevance Conclusions

Core Issue #1: Need for the NNISP

There is a continued need for nurses and allied health care professionals who are trained to work in remote and isolated First Nation communities. It is clear that health care delivery in these communities, where nurses are expected to provide a wide range of primary care services is substantially different from the rest of Canada. Health systems in remote and isolated First Nation communities are faced with many health-related challenges, such as disproportionately high rates of chronic disease, communicable disease, and substance abuse. Health care staff working in these communities often face other types of challenges such as geographical isolation, resource shortages, and cultural differences.

As a result, the evidence suggests that there is a need to continue to support innovative practices and approaches to improve primary care services in remote and isolated First Nation communities. The application of these practices and approaches will ultimately improve recruitment, retention and training of nursing and other health care staff.

For the most part, the NNISP pilot projects were specifically designed and implemented to meet the unique health care delivery needs of First Nation peoples living in remote and isolated communities.

Core Issue #2: Alignment with Government Priorities

Overall, the NNISP aligned with federal government priorities. There was alignment between the NNISP and Health Canada's Strategic Outcome 3 (First Nations [and Inuit] communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status). The NNISP also aligned with the mandate of FNIHB, which involves ensuring access to health care services and addressing health barriers in remote and isolated First Nation communities.

The federal government continues to prioritize improving the health care system, addressing health inequalities for First Nations, increasing access to health care, and encouraging collaborative health teams, which were all central aspects of the NNISP. Although the NNISP concluded in March 2013, the federal government will continue to prioritize the delivery of primary care in remote and isolated First Nation communities.

Core Issue #3: Alignment with Federal Roles and Responsibilities

The NNISP was aligned with the federal roles and responsibilities of improving First Nations health by demonstrating innovative practices and approaches in primary care service delivery.

Overall, the NNISP appeared to align with departmental key program activities and related responsibilities fairly well, although key informants were divided on some subjects. There was general agreement that the NNISP aligns with Program Activities 3.1 (First Nations and Inuit Primary Health Care) and 3.3 (Health Infrastructure Support for First Nations and Inuit). The NNISP also supported the department's organizational priority of strengthening First Nations and Inuit Health Program.

5.2 Performance Conclusions

Expected Outcomes (Effectiveness)

Although only the immediate outcomes were assessed in this evaluation, there is evidence that the NNISP used innovation in progressing towards the achievement of improved access to primary care services, increased capacity of the primary care workforce (knowledge, skills and ability); increased access to primary care nursing education for remote and isolated practice, and improved collaboration among health care providers and broader stakeholders. There were fewer findings to support the achievement of increased use of evidence-based information for the design, development and implementation of pilot projects that inform the delivery of quality primary care service delivery.

Economy and Efficiency

Many external and internal factors impacted the overall ability of the NNISP to expend available resources on pilot projects. Although many of these factors were beyond the control of the program (i.e., H1N1 outbreak and response), improved management and coordination in the planning and implementation of the pilot projects would have provided greater opportunities for additional pilot projects.

The capacity of the department to plan, design and implement projects that lasted for two years (on average) in such a short time period (four years) impacted the ability to fully achieve the immediate outcomes.

6.0 Implications

7.0 Lessons Learned

Project-Specific Lessons

Implement partnership strategies with other jurisdictional stakeholders in order to promote sustainability of successful innovation through long-term community and/or provincial involvement and support.
Improve jurisdictional partnerships in the delivery of health care services through collaborative approaches to primary care service provision (e.g., telerobotics).
Ensure adequate time and expertise to facilitate timely project development and funding; identify the appropriate skills required for project implementation (e.g., technology skills), and ensure appropriate and consistent performance monitoring and measurement across all projects.
Develop strategies to manage the growing community expectations for additional health care services.

Primary Care Service Delivery Lessons

Recognize and address the ongoing challenges of continuing education and professional development for nurses which impact recruitment and/or retention; increase on-line and other electronic resources that improve access to remote education opportunities for nurses within remote or isolated communities.
Provide alternative modes to support education (i.e., paper copies and courses on CDs) where high-speed internet is not available in remote communities.
Develop clear job descriptions for newly created positions.
Improve the collaborative team approach: address issues of resistance to change, duplication of work effort and the working capacity of care teams; develop community and team capacity to support the change process, clarify and communicate new team roles and the distribution of client workload among team members.

Appendix 1: Details on Data Collection and Analysis Methods

Data Collection Methods

The evaluation involved four main data collection methods: a document and data review; key informant interviews; a brief survey of First Nations representatives (hereafter referred to as the Community Survey); and a review of project proposals and reports. A supplementary data collection activity was conducted (review of project finances) to support the analysis for Issue #5 (economy and efficiency). The methods for each data collection activity are described below.

Document and data review: The documents had been categorized by fiscal year. In addition, there were separate folders for administrative documents and documents with tracking information. A master list of documents was provided in Excel format. Upon receiving the documents, they were entered into a Zotero^{Footnote 6} library for the review.

Key informant interviews: The key informant interviews used a multiple-step process for identifying and recruiting key informants. The evaluation working group was able to identify key informants responsible for higher level oversight and administration of the NNISP (i.e., Steering Committee members, Regional Nursing Directors and Regional Nursing Officers, and those involved in NNISP-funded education projects and activities). The interview process began by contacting these individuals including project managers and team members were identified by Regional Nursing Directors/Officers. Multiple interview guides were tailored to each of these groups of key informants and designed to address all relevant evaluation questions.^{Footnote 7}

Potential key informants received a letter from Health Canada describing the purpose and nature of evaluation and inviting them to participate.^{Footnote 8} After receiving the letter, a follow-up with each key informant commenced, and individual or group telephone interviews were scheduled. With key informants' permission, interviews were digitally recorded to ensure the accuracy of the information collected; however, key informants were assured of the confidentiality and anonymity of their responses. Once the interviews were completed, interview notes were imported into NVivo qualitative data analysis software for analysis. Responses were coded according to the evaluation questions, and sources were assigned attributes (stakeholder group, region, project stream, and funding arrangement) to allow for a comprehensive consideration of relevant variables with the potential to affect responses.

In total, 30 group and individual interviews were completed with 61 participants. While all four project streams were represented to some extent by key informants, most represented Stream 1 and Stream 2. This overrepresentation was due to the high number of projects in these two streams. In addition, not all evaluation questions were addressed by key informants in all streams, which precluded a stream-based analysis of the results.

Community survey: The third data collection method was the Community Survey. After conducting the key informant interviews, regional project leads were asked to request permission from First Nations partners to participate in a brief telephone survey. The timeline and the budget for this assignment did not allow for locating and obtaining consent from direct recipients of the innovation projects. The alternative, therefore, was to obtain information from individuals who could comment on the satisfaction of those who received services from one of the innovation projects, and therefore serve as a proxy for recipients. The department assisted this process by preparing a letter for regional directors to send to project managers to encourage the partners to participate.

Sixty-one names were received for the Community Survey. Thirty-seven (61%) of the 61 individuals provided responded to the telephone survey. The following table shows the breakdown of respondents by region; most were representatives from Manitoba reflecting the vast number of pilot projects implemented. Other regions were notably under-represented based on pilot project distribution.^{Footnote 9}

Responses by region - (n=37)
Region	N	%
Manitoba	12	32%
Alberta	10	27%
British Columbia	6	16%
Saskatchewan	6	16%
Ontario	3	8%
Quebec	0	-
Totals may not sum to 100% due to rounding.

Reasons for non-responses included: not aware of the project/program (n=11); not available/on holidays (n=6); unable to contact (n=5); and answering machine/no reply (n=2).

Review of project proposals and reports: The project documentation received from the program received two levels of review. FNIHB provided all available proposals and reports associated with the pilot projects (52 pilot projects in total). The reports for each project typically included proposals (draft and final) as well as interim and annual reports for each year of activity (2008-09 to 2012-13). In most cases, projects did not start until 2009-10. In some cases, reports sometimes were not provided. In the case of missing reports, it was unclear if projects had been completed or were cancelled. For each project, basic information was recorded (region, stream, value, brief description of the project, as well as description of expected outcomes) in Excel.

Data Analysis Methods

Analysis was conducted of the main data collection activities, as well as tables that presented the findings of the review of project proposals and reports and the review of project finances. This provided the findings for the evaluation report by each core issue. Each data collection method had its own method of analysis, and these methods are described below.

Document and data review: For the document and data review, key findings and observations were entered into a document review table.^{Footnote 10} For each evaluation question, the analysis identified information/evidence (usually with quotations), provided the source (with year and page/paragraph number), and observations (key findings from the evidence). Key findings were summarized, separated by evaluation questions and sub-questions.

Key informant interviews: As interviews are a qualitative exercise, they are not reported in a quantitative fashion with numbers and percentages. The following scale was used to report on key informant interviews: a few = 10-15%, some = 15-40%, many = 40-60%, most = 60-80% and almost all = over 80%.

Relying on the scale above, a summary, categorized by evaluation questions and sub-questions was developed.

Community survey: Summary data was gathered to analyze the findings for each survey question response. The summary analysis listed one of the questions asked, the possible responses (multiple choice), and the number and percentage of responses for each answer. The summary analysis was accompanied by a brief narrative explanation of the findings and included an overall conclusion.

Appendix 2: References

Flaherty, J. M. (2012). Jobs growth and long-term prosperity: Economic action plan 2012. Retrieved from http://www.budget.canada.ca/2012/plan/pdf/Plan2012-eng.pdf
FNIHB. (2011a). National Nursing Innovation Strategy background.
FNIHB. (2011b). Specific project partners of the NIS Program (database).
Gregory, D., Russell, C., Hurd, J., Tyance, J., & Sloan, J. (1992). Canada's Indian Health Transfer Policy: The Gull Bay Band experience. Human Organization, 51(3), 214-222.
Health Canada. (2005). Mandate and priorities. Retrieved June 21, 2012, from /ahc-asc/branch-dirgen/fnihb-dgspni/mandat-eng.php
Health Canada. (2009). A statistical profile on the health of First Nations in Canada: Self-rated health and selected conditions, 2002 to 2005. Retrieved from /fniah-spnia/alt_formats/pdf/pubs/aborig-autoch/2009-stats-profil-vol3/2009-stats-profil-vol3-eng.pdf
Health Canada. (2012a). 2012-13 report on plans and priorities. Retrieved from http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/shc/shc-eng.pdf
Health Canada. (2012b). Request for proposal: Evaluation of the FNIHB National Nursing Innovation Strategy Program (NNISP).
Misener, R. M., MacLeod, M. L. P., Banks, K., Morton, A. M., Vogt, C., & Bentham, D. (2008). "There's rural, and then there's rural": Advice from nurses providing primary healthcare in northern remote communities. Nursing Leadership, 21(3), 54-63.
NCCAH. (2011). Looking for Aboriginal health in legislation and policies, 1970 to 2008: The policy synthesis project. Retrieved from http://www.nccah-ccnsa.ca/docs/Looking%20for%20Aboriginal%20Health%20in%20Legislation%20and%20Policies%20-%20June%202011.pdf
Nursing Innovation funding estimates. (2008).
ONS. (2010a). Process for preparing ONS Proposals for Nursing Innovation Funding.
ONS. (2010b). National Nursing Innovation Strategy Program (NNISP) project charter.
Parliament of Canada. (2010). Government response to the Standing Committee on Health's report: Promoting innovative solutions to health human resources challenges. Retrieved June 6, 2012, from http://www.parl.gc.ca/HousePublications/Publication.aspx?Language=E&Mode=1&Parl=40&Ses=3&DocId=4677841&File=0
Smith, J. (2010). Promoting innovative solutions to health human resources challenges: Report of the Standing Committee on Health. Retrieved from http://www.csrt.com/en/professional/pdf/Standing_Committee_Health_June2010.pdf
Statistics Canada. (2007). Aboriginal health and well-being. Retrieved June 6, 2012, from http://www41.statcan.gc.ca/2007/10000/ceb10000_004-eng.htm
Statistics Canada. (2010). Health: Smaller proportion of Aboriginal people reported excellent or very good health. Retrieved June 6, 2012, from http://www.statcan.gc.ca/pub/89-645-x/2010001/health-sante-eng.htm

Footnotes

Footnote 1

Return to footnote1 Referrer

The NNISP was focused on First Nation remote and isolated communities and did not include any Inuit communities.

Footnote 2

Return to footnote2 Referrer

Expenditures on projects including salary and administrative costs.

Footnote 3

Return to footnote3 Referrer

Government of Canada funding authority.

Footnote 4

Return to footnote4 Referrer

Ibid

Footnote 5

Return to footnote5 Referrer

Atlantic region receives primary care services by the province.

Footnote 6

Return to footnote6 Referrer

Zotero is a program that allows users to manage documents and other sources with high efficiency. The program automatically generates in-text citations and reference lists. For more information, see http://www.zotero.org/.

Footnote 7

Return to footnote7 Referrer

Tailoring interview guides to account for the specific areas of expertise of key informants helped to ensure collection of relevant and reliable data and, therefore, supports the validity of the evaluation.

Footnote 8

Return to footnote8 Referrer

Project managers and team members were informed of the evaluation through their Regional Director.

Footnote 9

Return to footnote9 Referrer

Participation was voluntary however there is no clear reason why some regions are under-represented.

Footnote 10

Return to footnote10 Referrer

Although not a main line of evidence for this evaluation, a brief literature review informed the document review. Therefore, some of the sources appearing in the document review were obtained through a literature search, rather than through FNIHB documents. The literature review focused only on health issues for First Nations in Canada.

Page details

Date modified:: 2020-09-01

Evaluation of the First Nations National Nursing Innovation Strategy Program 2008-2009 to 2011-2012

List of Acronyms

Table of Contents

List of Tables

Executive Summary

Evaluation Purpose, Scope and Design

Program Description

Evaluation Conclusions and Implications

Conclusions: Relevance

Need for the NNISP

Alignment with Government Priorities

Alignment with Federal Roles and Responsibilities

Conclusions: Performance

Achievement of Outcomes

Economy and Efficiency

Implications

Lessons Learned

Management Response

Evaluation of the First Nations National Nursing Innovation Strategy Program (NNISP)

1.0 Evaluation Purpose

2.0 Program Description

2.1 Program Context

2.2 Program Profile

2.3 Program Logic Model and Narrative

2.4 Program Alignment and Resources

3. Evaluation Description

3.1 Evaluation Scope

3.2 Evaluation Issues

3.3 Evaluation Approach

3.4 Evaluation Design

3.5 Data Collection and Analysis Methods

3.6 Limitations and Mitigation Strategies

4.0 Findings

4.1 Relevance: Issue #1 -- Need for the NNISP

Health and Health Care Needs

Responsiveness to the Needs of First Nation Peoples in Remote and Isolated Communities

4.2 Relevance: Issue #2 -- Alignment with Government Priorities

Alignment with Federal Government Priorities

Alignment with Departmental Strategic Priorities and Outcomes

4.3 Relevance: Issue #3 -- Alignment with Federal Roles and Responsibilities

4.4 Performance: Issue #4 -- Achievement of Immediate Outcomes (Effectiveness)

Immediate Outcome #1: Improved Access to Primary Care Services

Immediate Outcome #2: Increased Capacity (Knowledge, Skills, and Ability) of the Primary Care Work Force

Immediate Outcome #3: Increased Access to Primary Care Nursing Education for Remote and Isolated Practice

Immediate Outcome #4: Improved Collaboration among Health Care Providers and Broader Stakeholders

Immediate Outcome #5: Increased Use of Evidence-based Information to Inform Quality Primary Care Service Delivery

Challenges and/or Barriers Encountered

Unintended Results and/or Consequences as a Result of Program Implementation

4.5 Performance: Issue #5 -- Assessment of Economy and Efficiency

Observations on Economy

Observations on Efficiency

Observations on the Adequacy of Performance Measurement Data

5.0 Conclusions

5.1 Relevance Conclusions

Core Issue #1: Need for the NNISP

Core Issue #2: Alignment with Government Priorities

Core Issue #3: Alignment with Federal Roles and Responsibilities

5.2 Performance Conclusions

Expected Outcomes (Effectiveness)

Economy and Efficiency

6.0 Implications

7.0 Lessons Learned

Appendix 1: Details on Data Collection and Analysis Methods

Data Collection Methods

Appendix 2: References

Page details