Evaluation of the First Nations Clinical and Client Care Program 2005-2006 to 2011-2012

Prepared by Evaluation Directorate Health Canada and the Public Health Agency of Canada

June 2013

List of Acronyms

AHHRI

Aboriginal Health Human Resource Initiative

CCC

Clinical and Client Care

EWG

Evaluation Working Group

FNIHB

First Nations and Inuit Health Branch

HR

Human Resources

HSDM-RI

Health Services Delivery Model - Remote and Isolated Communities Project

LPN

Licensed Practical Nurse

NP

Nurse Practitioner

PAA

Program Alignment Architecture

PHAC-HC

Public Health Agency of Canada-Health Canada

RMAF

Results-based Management and Accountability Framework

RNE

Regional Nurse Educator

RN

Registered Nurse

RNM

Regional Nurse Manager

RPN

Registered Practical Nurse

RPP

Report on Plans and Priorities

TBS

Treasury Board Secretariat

Table of Contents

• Executive Summary
• Management Response and Action Plan 2012/2013
• 1. Evaluation Purpose
• 2. Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Alignment and Resources
• 3. Evaluation Description
  • 3.1 Evaluation Scope
  • 3.2 Evaluation Issues
  • 3.3 Evaluation Approach
  • 3.4 Evaluation Design
  • 3.5 Data Collection and Analysis Methods
  • 3.6 Limitations and Mitigation Strategies
• 4. Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Immediate Outcomes (Effectiveness)
  • 4.5 Performance: Issue #4 - Achievement of Intermediate Outcomes (Effectiveness)
  • 4.6 Performance: Issue #4 - Achievement of Long-Term Outcomes (Effectiveness)
  • 4.7 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5. Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1: Details on Data Collection and Analysis Methods
• Appendix 2: References

List of Tables

• Table 1 Summary of findings, conclusions and recommendations Clinical and Client Care Logic Model
• Table 2 Summary of findings, conclusions and recommendations Clinical and Client Care Logic Model
• Table 3 Core Evaluation Issues and Questions
• Table 4 Limitations and Mitigation Strategies
• Table 5 Extent of access to CCC services -- internal service providers
• Table 6 Non-nursing health care provider positions at nursing stations - internal service providers
• Table 7 Barriers to participation in continuing education/professional development - internal service providers
• Table 8 Linkages and collaboration within the community - internal service providers
• Table 9 Linkages and collaboration with outside service providers - internal service providers and band representatives
• Table 10 Policies, standards, or guidelines that are used regularly in providing CCC services - internal service providers

Executive Summary

Evaluation Purpose, Scope and Design

This evaluation covers the First Nations Clinical and Client Care (CCC) program for the period April 1, 2005, to March 31, 2012. This evaluation is the first undertaken of the CCC program as a whole and has been done to fulfill the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009).

The purpose of the evaluation was to assess the relevance and performance of the CCC program. An Evaluation Working Group with representation from First Nations and Inuit Health Branch National and Regional Offices, and led by the Evaluation Directorate, provided guidance and input into the evaluation process. The methodology used in the evaluation included a document, data, and literature review; key informant interviews; and stakeholder surveys.

Program Description

CCC services are typically nurse-led and provide residents of remote and isolated First Nations communities access to urgent and non-urgent health services that are not available through provincial or regional health authorities. Urgent care is provided in consultation with a physician by phone or Internet and may involve stabilizing treatment and/or transportation to a secondary or tertiary care facility. Non-urgent care involves non-life-threatening health issues that are assessed and diagnosed, and a management/treatment plan determined and implemented. This could involve consultation between relevant health care providers. Other elements of CCC services include: coordination of care and case management; access to medical equipment, supplies, and pharmaceuticals; a system of record keeping and data management; continuous quality improvement process; diagnostics; and in-patient federal hospital services (Manitoba only).

Evaluation Conclusions and Recommendations

Conclusions - Relevance

Continued Need
The CCC program continues to address a demonstrable need that is responsive to the health needs of First Nations communities by addressing on-going health demands. These demands are related to higher rates of illness and changing demographics, which require CCC services in communities that would otherwise not have these services.

Alignment with Government Priorities
The CCC program's service delivery in remote and isolated First Nations communities align with federal government priorities as articulated in Health Canada's 2012-13 Report on Plans and Priorities, federal budgets and Speeches from the Throne.

Alignment with Federal Roles and Responsibilities
The CCC program aligns well with the role of the federal government to provide or fund health programs for First Nations through the First Nations and Inuit Health Branch of Health Canada. The Program is consistent with the Indian Health Policy and departmental mission and mandate statements.

Conclusions - Performance

Achievement of Expected Outcomes
The CCC program is progressing towards its intended outcomes and is responsive to the needs of First Nations individuals and communities through the provision of urgent and non-urgent health care services.

The main challenges in program delivery stemmed from staff recruitment and retention, challenges in linking with other service providers, lack of information sharing of client data and systematic tracking of both human resources and performance measurement data.

Economy and Efficiency
There are examples of project implementation that demonstrate ways to improve CCC service efficiency that could be incorporated in future models for primary care delivery within remote and isolated First Nations communities. Improved overall performance measurement data would better support reporting requirements and the conduct of future evaluations of primary care service delivery.

Recommendations

Recommendation 1:

Strengthen efforts concerning nursing recruitment and retention.

Recommendation 2:

Work with regions, First Nations communities, and provincial health departments to strengthen collaboration and improve information sharing and partnerships.

Recommendation 3:

Work with regions, First Nations communities, and nursing stations to develop and implement an improved performance measurement strategy that will assist regions and National Headquarters in measuring the achievement of expected outcomes and planning for resource utilization and service requirements. This should include electronic record keeping on clients and services provided and on all personnel at the nursing stations.

Management Response and Action Plan 2012/2013

1. Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the First Nations Clinical and Client Care (CCC) program for the period April 1, 2005, to March 31, 2012.

This evaluation is the first undertaken of the program as a whole and has been done to fulfill the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009).

2. Program Description

2.1 Program Context

The program was developed in response to meet the primary health care needs of remote and isolated First Nations communities.

2.2 Program Profile

The CCC program consists of essential primary care services developed and implemented in order to meet the health care needs of remote and isolated First Nations communities. CCC services encompass a number of different health care services that are provided by Health Canada or through contribution agreements with Tribal Councils and First Nations bands. Although First Nations individuals of any age living on-reserve are eligible to receive CCC services, non-First Nations individuals may also access services where not otherwise available.

The CCC program's services are provided by interdisciplinary health care teams, which are typically headed by nurses. The composition of the team varies depending on the service needs of a given area. The team can include regulated health professionals, such as registered nurses (RN), nurse practitioners (NP), licensed practical nurses (LPN), registered practical nurses, medical radiation technologists, and medical laboratory technologists. It may also include unregulated health workers, such as health care aides, rehabilitation aides, pharmacy technicians, and support personnel (FNIHB, 2011, p. 50).

The three objectives of CCC program are as follows:

• "Provide access to urgent and non-urgent health services to community members including those who reside in remote/isolated communities where access to health services is not available through provincial or regional health authorities.
• Provide access to coordination and consultation services with other appropriate health care providers and/or institutions as indicated by client needs.
• Provide access to short term in-patient services in Federal Funded Hospitals in Manitoba." (Health Canada, 2011a, p. 3).

The eight main elements of the CCC program are outlined below (Health Canada, 2011a, pp. 3-4):

1. urgent care;
2. non-urgent care;
3. in-patient federal hospital services (Manitoba only);
4. coordination and case management;
5. access to medical equipment, supplies, and pharmaceuticals;
6. system of record keeping and data management;
7. continuous quality improvement process; and
8. diagnostics.

Stakeholders include a wide range of internal and external parties. Internal stakeholders include Health Canada National and Regional Offices, Bands and Tribal Councils, and CCC service providers (e.g., RNs, NPs, LPNs, Registered Practical Nurses, medical radiation technologists, medical laboratory technologists, pharmacy technicians, laboratory/x-ray technicians, and public health physicians). External stakeholders include CCC providers living off-reserve, such as family practice physicians, physician specialists, air ambulance, provincial/federal hospitals, and rehabilitation settings. Beneficiaries include members of First Nations communities (FNIHB, 2011).

2.3 Program Logic Model and Narrative

Table 2 is the logic model for the CCC program.^{Footnote 1} The objective of CCC is to provide clinical and client care services for remote and isolated First Nations communities, and as such, the target recipients are First Nations individuals living on reserves in these communities. According to the logic model, the CCC has five main themes:

1. service provision;
2. capacity building;
3. stakeholder engagement and collaboration;
4. data collection, research, and surveillance; and
5. policy development and knowledge sharing.

Each of these themes has related outputs targeted to specific audiences and intended to produce immediate outcomes. The immediate outcomes expected to flow from the outputs include the following:

• increased First Nations awareness of self-care and illness prevention;
• improved access to CCC services;
• increased appropriate use of CCC services;
• increased capacity (knowledge, skills, and abilities) of the CCC workforce;
• increased First Nations' management of CCC services;
• increased collaboration, awareness, and understanding of service delivery arrangements, service requirements, and accountabilities; and
• increased awareness and understanding of policies, standards, guidelines, and best practice/evidence-based information in service delivery.

Immediate outcomes should lead to the following intermediate outcomes:

• timely collaboration/system response to CCC needs; and
• increased use of policies, standards, guidelines, and best practices and evidence-based information for CCC quality improvements.

Finally, all outcomes should lead to the one long-term outcome:

• CCC services are responsive to the needs of First Nations individuals and communities

2.4 Program Alignment and Resources

The CCC contributes to Health Canada's Strategic Outcome 3: First Nations communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status. The CCC sub-sub activity was identified in the Department's Program Alignment Architecture (PAA) under Program Activity 3.1: First Nations and Inuit Primary Health Care, and the sub-activity of Primary Care (Health Canada, 2011b, p. 4).

Due to overlap between the CCC program and other programs within Health Canada's PAA as well as the financial coding and tracking methods in place for most of the evaluation years, expenditures for only CCC services were not available. According to financial data, CCC expenditures in 2011-12 were $139.7 million. This is the most representative measure of actual CCC resources during the evaluation period.

3. Evaluation Description

3.1 Evaluation Scope

The scope of the evaluation covers the period from April 1, 2005, to March 31, 2012, and includes all FNIHB CCC activities and services, as defined by the 2005 Authorities and renewed 2011 Authorities.

3.2 Evaluation Issues

The evaluation considered the five core evaluation issues as per the 2009 Treasury Board Policy on Evaluation under two themes of relevance and performance. Specific questions were developed based on program considerations, and used to guide the evaluation process. Table 3 below presents the issues and questions addressed by this evaluation.

3.3 Evaluation Approach

The evaluation used an outcome-based approach to assess the progress made towards the achievement of immediate outcomes. The approach included collaboration with key internal and external stakeholders in the planning and conduct of the evaluation, review of technical data and the evaluation report as well as the development of the management response.

3.4 Evaluation Design

This evaluation used a non-experimental and retrospective design. The evaluation was non-experimental because evidence on the progress toward the achievement of expected outcomes was observational in nature. Furthermore, not only did the evaluation require a retrospective design because the data was based on past years of CCC funding, but it also used a retrospective design because there was an absence of baseline data.

3.5 Data Collection and Analysis Methods

The evaluation was conducted using multiple lines of evidence, including a document, data, and literature review; key informant interviews to gain the qualitative perspective of relevant Health Canada stakeholders; and, surveys of stakeholders, including internal and external service providers, as well as representatives of CCC service recipients. An inclusion/exclusion approach was used for assessing documents, data, and literature for relevance and applicability.^{Footnote 6} The sampling approach for internal service providers and recipient representatives was to include all nursing stations providing CCC services as well as all band representatives of communities with nursing stations. External service provider sampling was achieved by requesting regions identify a small target sample. Use of interviews and surveys gave a balanced qualitative and quantitative approach to obtaining stakeholder perspectives. Furthermore, given the evaluation was conducted under very tight timelines, the surveys obtained perspectives from a large number of stakeholders within a relatively short time frame.

Key stakeholders were involved in the evaluation process in various ways. Internal stakeholders from Health Canada's National and Regional Offices participated in key informant interviews, and are referred to throughout the report as key informants. Internal stakeholders providing direct CCC services (nurses) participated in a stakeholder survey. The survey process also included band representatives as proxies for clients. Several external service providers were also surveyed. Surveyed representatives are referred to as surveyed stakeholders. As well, representatives of Health Canada's FNIHB National and Regional Offices, and the Evaluation Directorate of Public Health Agency of Canada-Health Canada (PHAC-HC) participated as members of the Evaluation Working Group (EWG) which provided guidance and input into the evaluation process.

The data collection and analysis methods by line of evidence are summarized below, with detailed descriptions provided in Appendix 1. Information was gathered through each line of evidence according to the evaluation questions each was expected to address.

Document, Data, and Literature Review

The document and data review involved review of key documents and data provided primarily by Health Canada as well as some acquired through relevant websites. Documents included those related to: relevant legislation; departmental planning and reporting; federal budgets; performance measurement and progress reporting; reporting on special studies/projects; and other relevant documents. Data included Community-Based Reporting Template^{Footnote 7} data, samples of nursing station activity logs, and CCC financial expenditures.

The literature review primarily addressed questions on economy and efficiency and involved a scan of peer reviewed journals and grey literature using key words (see Appendix 2).

Documents, datasets, or pieces of literature received were assessed for relevance using a set of inclusion/exclusion criteria, primarily involving their relevance for addressing evaluation questions, whether they referred to a time period within the evaluation period, and whether they were aggregated to at least a regional level (i.e., not raw data).

Key Informant Interviews

This task involved conducting 37 interview sessions with 46 key Health Canada Regional Office and National Headquarters stakeholders that had knowledge of CCC services. Participants were from the following groups:

• Regional Directors, Regional Directors of Nursing, Regional Nursing Officers, Regional Nurse Managers, Zone Directors;
• Regional Nurse Educators/Practice Consultants;
• Regional Human Resource Officers;
• Regional Financial Officers and Health Funding Arrangement personnel; and
• National Headquarters representatives (Executive Director for Primary Care Division, Nurse Consultants, Financial Management Planning representative).

Completed interview notes were analyzed according to key evaluation issues and questions using qualitative data analysis software (NVivo). Qualitative key informant interviews provide insight into a process or problem, and, as such, are not conducive to counting up responses. The following descriptive scale was used to indicate the approximate number of key informants that made the relevant statement, with "a few" being approximately 10-15% or less of respondents, "some" being more than 15% to approximately 40%, "many" being more than 40% to approximately 60%, "most" being more than 60% to approximately 80%, and "almost all" being over 80%.

Stakeholder Surveys

Surveys were conducted with nurses at nursing stations, band representatives, and several external service providers. Band representatives were surveyed as proxies for clients, to provide their perspective on their community members' experiences with CCC services.^{Footnote 8} Survey responses were analyzed through SPSS (statistical analysis software) using frequency tables and descriptive statistics.

3.6 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications on the validity and reliability of evaluation findings and conclusions. This section illustrates the limitations in the design and methods for this particular evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

4. Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

First Nations individuals and communities often experience higher rates of certain diseases, substance abuse and addiction issues, mental health issues, and injuries related to acts of violence and accidents. All of these factors, as well as an aging population and growing communities, are creating increased demand for CCC services. Given the geographic isolation of remote and isolated First Nations communities, members of these communities do not have the same access to health care services as other Canadians.

The CCC program continues to address a demonstrable need that is responsive to the health care needs of First Nations in remote and isolated First Nations communities. First Nations individuals and communities often experience higher rates of certain diseases (including co-morbid conditions), substance abuse and addiction issues, mental health issues, and injuries related to acts of violence and accidents. All of these factors, as well as an aging population and growing communities, are creating increased demand for CCC services. Given the geographic isolation of remote and isolated First Nations communities, members of these communities do not have the same access to health care services as other Canadians. The CCC program is intended to respond to this inequity, providing remote and isolated First Nations community members access to primary care services within their communities.

Health and Health Care Needs

First Nations individuals living in remote and isolated First Nations communities face unique health-related challenges. Disparities in health outcomes and access to health care still exist between First Nations people and other Canadians. Studies have revealed that First Nations people often have a higher rate of certain diseases and conditions than the Canadian population overall. First Nations people often fare worse than the general population with regards to chronic and communicable disease incidence. Statistics indicate that on-reserve First Nations populations experience tuberculosis and diabetes at rates 5.2 and 3.8 times higher than the general Canadian population (Health Canada, 2012b), and the new HIV infection rate is 3.6 times higher in the Aboriginal compared to non-Aboriginal population (Health Canada, 2012b). Some evidence suggests First Nations people experience more challenges in disease management. For example, Harris et al., (2011) noted that compared to the general diabetic population, First Nations people had twice as many diabetes-related visits to a health facility. First Nations people also experience challenges with a wide variety of other health-related issues, such as higher mortality rates, lower life expectancy, and higher youth suicide rates compared to the general Canadian population (Health Canada, 2012b).

Interviewed and surveyed stakeholders further confirmed these trends, identifying that First Nations' medical conditions have increased in complexity and include, for example, co-morbid conditions, substance abuse and addiction issues, mental health issues, and injuries related to acts of violence and accidents. First Nations individuals themselves also do not self-report good health to the same extent as the general population. From the 2008-10 First Nations Longitudinal Regional Health Survey, less than half (44%) of First Nations adults self-reported their health as thriving compared to 60% of the general Canadian population (The First Nations Information Governance Centre, 2012, p. 139). Also from the RHS, close to two thirds (63%) of First Nations adults reported at least one chronic health condition, and just under 40% reported having two or more conditions (The First Nations Information Governance Centre, 2012, pp. 120-121).

Discrepancies in disease rates, in turn, are often linked to socio-economic factors such as income, education, and employment levels, which are important social determinants of health (Statistics Canada, 2007). The First Nations population experiences lower levels of income, education, and employment compared to the general Canadian population. For example, in the Regional Health Survey (2008-10), close to 60% of First Nations adults reported an annual income of less than $20,000 (The First Nations Information Governance Centre, 2012, p. 38). First Nations adults (25-54 years of age) on-reserve had a 2006 unemployment rate of 23% compared to the 5% Canadian rate (Health Canada, 2012b). In addition, just under one half (49%) of the on-reserve First Nations adults did not have a high school degree in 2006, compared to 14% for the general Canadian population (Health Canada, 2012b). Further challenging First Nations individuals in remote and isolated First Nations communities is their geographic isolation and distance from provincial health services.

First Nations' demographic factors are also changing, with many communities experiencing population growth as well as an aging population. For example, the First Nations population has a projected average annual growth rate from 2001 to 2017 of 2.0% compared to the 0.7% for all of Canada (Health Canada, 2012b). As well, despite the health disparities, First Nations people, similar to the general Canadian population, are living longer than they used to (FNIHB, 2012b, p. 7). For example, between 1975 and 2000, life expectancy for the status Indian population increased from 59.2 years to 68.9 years for males, and 65.9 years to 76.3 years for females (Statistics Canada, no date). While First Nations people have a lower life expectancy than other Canadians, this gap has decreased over time. From 1975 to 2000 the disparity in life expectancy between male status Indians and other Canadian males declined from 11.1 years to 7.4 years, and between female status Indians and other Canadian females from 11.7 years to 5.2 years (Statistics Canada, n.d.).

Stakeholders point to all of these factors as creating increased demands for CCC program services, with approximately two thirds of surveyed nurses (68%) and band representatives (65%) reporting that demand for CCC services in their community is increasing. Key informants and surveyed stakeholders attributed increased demands to a variety of health and demographic factors. Health factors that a majority of surveyed stakeholders identified as contributing to increased demands for CCC services included: addiction and other substance abuse issues (reported by 88% of surveyed nurses and 71% of surveyed band representatives); declining physical health status of community members (reported by 71% of surveyed nurses and 53% of surveyed band representatives); and increasing mental health issues (reported by 69% of surveyed nurses and 53% of surveyed band representatives). Demographic factors identified by a majority of surveyed stakeholders included: growing community populations (reported by 72% of surveyed nurses and 94% of surveyed band representatives) and an aging population (reported by 71% of surveyed nurses and 65% of surveyed band representatives). Key informants noted that these changes are placing greater demands on the service providers in terms of time and skill set requirements.

CCC Services

Remote and isolated First Nations communities do not have the same level of health care services available to them as other Canadian and First Nations communities due to their geographic isolation. This was made evident in a number of studies examining access to health services in these First Nations communities. Since access is not easily measured, studies have examined rates of preventable hospitalization for ambulatory sensitive conditions (ASC) as a proxy measure for access to health care. In these studies, the higher rates of preventable hospitalization for ASC in First Nations populations are believed to reflect insufficient access to primary care and secondary care (Gao et al., 2008, p.1011; Shan, Gunraj, & Hux, 2003, p.800).

Most remote and isolated First Nations communities do not have year-round road access and physician and other health care services are located long distances from the community. The CCC program is intended to offer individuals living in remote and isolated First Nations communities, who do not have access to provincial or regional health authority services, clinical care services within their communities (Health Canada, 2011a, pp. 2-3). The CCC program address a demonstrable need in remote and isolated First Nations communities given the health-related challenges experienced by many First Nations individuals and the geographic isolation of these communities. Provision of urgent and non-urgent care in communities where primary care services would otherwise be hours away is a crucial step to improving the health status of First Nations individuals. The other elements of CCC provide a supportive function to optimize urgent and non-urgent care services and their integration with other health care services.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Federal budgets have consistently committed funds to initiatives for improving health outcomes of First Nations. The CCC program contributes to Health Canada's strategic outcome "First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs".

The CCC program's service delivery to remote and isolated First Nations communities remains a priority to the federal government. Health Canada's 2012-13 Report on Plans and Priorities articulates continued financial support to primary health care services for First Nations. Federal budgets have consistently committed significant funds to initiatives for improving health outcomes of First Nations, as has the renewal to 2015 of the Aboriginal Health Human Resources Initiative. Speeches to the Throne also demonstrate the federal government's commitment to building partnerships with First Nations.

The CCC program aligns well with departmental strategic outcomes, as a sub-sub activity of Health Canada's Strategic Outcome 3 and with a long-term outcome that mirrors the strategic outcome ("First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs"). The CCC program also aligns directly with FNIHB's mandate to ensure First Nations and Inuit communities have access to health services.

Several documents provide evidence that the CCC program remains a priority of the federal government. For instance, Health Canada's 2012-13 Report on Plans and Priorities (RPP) commits $2.7 billion over the three-year period of 2012-13 to 2014-15 for addressing the primary health care needs of First Nations and Inuit through "primary health care programs and services that are responsive to the needs of First Nations and Inuit individuals, families, and communities" (Health Canada, 2012c, p. 37).

A series of federal budgets provide continued financial support to improving health outcomes of First Nations. Budget 2008 provided $147 million over two years to support better integration between First Nations and provincial/territorial health systems, with continued support announced for 2009 (Finance Canada, 2009, pp. 107-108). Budget 2009 committed $305 million over two years for strengthening health care programming, such as primary care services, Non-Insured Health Benefits (NIHB), continued support to integration with provincial/territorial systems, and investments in health infrastructure, including health clinics and nurses' residence systems (Finance Canada, 2009, p. 108). In addition, to improve delivery of primary health services to all Canadian rural and remote communities, Budget 2011 and 2012 committed to forgive portions of Canada Student Loans to new physicians, nurses, and nurse practitioners practicing in rural and remote communities (Finance Canada, 2012, p. 176). The 2011 Speech from the Throne also outlined plans to address social and economic barriers experienced by Aboriginal communities through, for example, promoting access to clean water and energy, and supporting education of First Nations children and adults (Government of Canada, 2011, pp. 13-14). These findings demonstrate the federal government's commitment to building partnerships with First Nations and improving outcomes for those living in remote and isolated First Nations communities.

Finally, the Aboriginal Health Human Resources Initiative (AHHRI) provides further evidence of the continued federal priority of health care service delivery to First Nations communities. Through the AHHRI, Health Canada works in partnership with Aboriginal associations and other partners for developing and implementing programs to support increasing the numbers of Aboriginal people pursuing health care careers and for retaining health care workers in Aboriginal communities (Health Canada, 2012d). The AHHRI was first announced in 2004, with $100 million allocated over a five-year period, and then renewed in 2010 to 2014, with another $80 million allocated over the five-year period (AANDC, 2011).

The evaluation found that the CCC program aligns with departmental strategic outcomes and priorities. Specifically, the CCC program aligns with Health Canada's Strategic Outcome 3 as reflected in the 2011-12 Program Alignment Architecture : "First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status" (Health Canada, 2011b, p. 4). The CCC program is a sub-subactivity of Strategic Outcome 3, and the CCC's long-term expected outcome of "clinical and client care services are responsive to the needs of First Nations individuals and communities" (see Figure 2: the CCC Logic Model) directly aligns with this strategic outcome.

FNIHB's mandate is to ensure First Nations and Inuit communities have access to health services, to help First Nations communities "address health barriers, disease threats, and attain health levels comparable to other Canadians living in similar locations," and to improve the health system through strengthened partnerships with First Nations and Inuit (Health Canada, 2005). The CCC program aligns well with the FNIHB mandate by providing members of remote and isolated First Nations communities the access to primary care services, and by doing so through participation in collaborative partnerships with other health care providers.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

The CCC program aligns well with the provision of First Nations and Inuit health programs which are provided or funded by the First Nations and Inuit Health Branch of Health Canada consistent with the Indian Health Policy and departmental mission or mandate statements.

The CCC program is congruent with the department's jurisdictional and mandated role as evidenced in key pieces of legislation that refer to First Nations, including the Constitution Act, 1867 and the Indian Act, 1876. In addition, the Indian Health Policy and the Indian Health Transfer Policy are two policies which relate to First Nations health at the national level. Also, the Canada Health Act, 1984 identifies a main objective of Canadian health policy as facilitating reasonable access to care to residents of Canada.

The CCC program aligns with departmental key program activities and is directly linked as a sub-sub activity to Program Activity 3.1 of the PAA: First Nations and Inuit Primary Health Care. The areas of focus of PAA 3.1 are to fund health initiatives to the benefit of on-reserve First Nations individuals/communities or Inuit communities. This includes, among other health-related programming, providing primary care to individuals in the form of "diagnostic, curative, rehabilitative, supportive, palliative, and referrals services" (Health Canada, 2011b, p. 78). A continued focus under PAA 3.1 for 2012-13 is addressing human resources issues, such as recruitment and retention issues, and supporting professional practices for nurses in remote and isolated First Nations communities(Health Canada, 2012c, p. 38).

In addition, the CCC program aligns with Health Canada's Organizational Priority III: Strengthen First Nations and Inuit Health Programming, as articulated in Health Canada's 2012-13 Report on Plans and Priorities. This priority is strongly related to the CCC program in that it articulates strengthening First Nations and Inuit access to primary care, improving partnerships and integration of health services, and increased First Nations control of health care (Health Canada, 2012c, p. 6).

Furthermore, by addressing health services disparities between remote and isolated First Nations communities and other Canadian communities and by facilitating First Nations management of primary care services, the CCC program also align with FNIHB's priorities to "support action on health status inequalities affecting First Nations and Inuit communities, according to their identified priorities" and to "transfer existing health resources to First Nations and Inuit control within a time-frame to be determined by them" (Health Canada, 2005).^{Footnote 9} No stakeholder, either through interviews or the survey process, identified any activities conducted through the CCC program that are outside of the identified roles and responsibilities.

The CCC program is appropriate based on the roles and responsibilities identified in key legislation and policy documents. Specifically, the CCC program is congruent with the department's jurisdictional and mandated role as evidenced by the following documents and associated findings:

• The Constitution Act, 1867 (Government of Canada, 1867), which gives joint federal/provincial health care delivery responsibilities, and which, under Section 91(24), identifies "Indians, and Lands reserved for the Indians" as a federal responsibility.
• The Indian Act, 1876 (Government of Canada, 1876), which gives the federal government the right to use reserve lands for Indian health projects with consent of the Band Council.
• The Indian Health Policy (1979), which aims to increase the health status of Indian communities "through mechanisms generated and maintained by the communities themselves." The policy focuses on increasing the capacity of communities to deliver health care services and on building relationships between federal, provincial, and local governments (NCCAH, 2011, p. 23).
• The Canada Health Act, 1984 (Government of Canada, 1984), which identifies a main objective of Canadian health policy as the foundation of reasonable access to care to residents of Canada.
• The Indian Health Transfer Policy (1989), which encourages uptake of community-based health care services. The policy provides opportunities for Tribal Councils and communities to take more responsibility in planning and delivering health care programs and services (NCCAH, 2011, p. 24).
• Health Canada's mandate "to help Canadians maintain and improve their health," with one of their responsibilities for fulfilling this mandate being to deliver health care services to First Nations (Health Canada, 2012f).

The Indian Health Policy (1979) and the Indian Health Transfer Policy (1989) provide First Nations communities the opportunity to gain more authority over and control of primary health care service delivery. The legislation and policies also acknowledge federal responsibilities and the relative disadvantages of First Nations communities regarding health status. The CCC Logic Model illustrates the program's alignment with these policies and legislation, through the related expected outcomes of increased capacity of the CCC workforce and increased First Nations management of CCC services.

4.4 Performance: Issue #4 - Achievement of Immediate Outcomes (Effectiveness)

Immediate Outcome #1: Increased First Nations Awareness of Self-care and Illness Prevention; Hospital Services (in Manitoba)

Awareness of self-care and illness prevention varies across communities and individuals but is generally seen as improving.

Key informants reported some increase in the level of awareness of self-care and illness prevention in remote and isolated First Nations communities, although there is variation. Much of this improvement was attributed to disease prevention/health promotion initiatives as well as through interactions with CCC health care providers at nursing stations; however, key challenges remain in translating awareness into behavioural changes.

Surveyed and interviewed stakeholders report there are varying levels of awareness of self-care and illness prevention across communities and individuals. For example, surveyed band representatives were split on the current level of awareness of self-care and illness prevention in their communities, with 27% reporting high/very high awareness, 35% medium awareness, and 38% low/very low awareness.^{Footnote 10} Key informants said that increased awareness can occur when strong leadership is in place, such as when community leaders take the initiative to access and implement programming to meet the needs of the community, or when communities have a stable, consistent health team that engages with the community. Also, the general availability of electronic media (e.g., radio, television, and Internet) is viewed as contributing to a general increased knowledge regarding self-care and illness prevention.

Most key informants who could speak to this outcome reported that First Nations' awareness of self-care and illness prevention is improving, with much of this increase attributed to disease prevention/health promotion programs and initiatives. Key informants gave examples of several programs contributing to increased awareness, such as through other federal health programs, including the Aboriginal Diabetes Initiative, the Canada Prenatal Nutrition Program, the Children's Oral Health Initiative, and home care. Several key informants noted that Health Canada has invested considerable resources into prevention and promotion programs over the past number of years. Key informants and surveyed stakeholders report that interactions with CCC health care providers at nursing stations contribute to a greater awareness and understanding of self-care and illness prevention in remote and isolated First Nations community members, with most surveyed nurses and band representatives indicating this is occurring to at least a medium extent (83% and 73%, respectively).

Increased awareness of self-care and illness prevention is expected to translate into increased participation in activities and/or behavioural changes for improving health status. For example, an increased awareness of the health impacts from smoking was the reason given for quitting smoking by 30% of the ex-smokers in the 2002-03 Regional Health Survey, and 21% of smokers who quit or tried to quit over the previous 12 months in the 2008-10 survey (First Nations Information Governance Centre, 2005, p. 108; The First Nations Information Governance Centre, 2012, p. 102). And 64% of the ex-smokers in the 2002-03 survey and 56% of the smokers in the 2008-10 survey who had quit or tried to quit did so to choose a healthier lifestyle (First Nations Information Governance Centre, 2005, p. 108; The First Nations Information Governance Centre, 2012, p. 102). However, attribution of such changes to the CCC program was not possible.

A few key informants noted that socio-economic conditions experienced by many remote and isolated First Nations communities can challenge individuals' ability to gain awareness of self-care and illness prevention and/or to translate any awareness gained into behavioural changes to positively impact health status. Examples of socio-economic challenges include lack of adequate housing, high food costs, and low literacy levels. With respect to adequate housing, just over one quarter (26%) of First Nations on-reserve were living in over-crowded homes in 2006 compared to 3% of the non-Aboriginal Canadian population (Health Canada, 2012b). The 2008-10 RHS reported that over half (54%) of First Nations households were moderately (40%) to severely (14%) food insecure. In addition, from the same survey, over a third (36%) of First Nations adults said they did not have a year-round safe drinking water supply (The First Nations Information Governance Centre, 2012, pp. 60, 91).

Immediate Outcome #2: Improved Access to CCC Services or Hospital Services (Manitoba)

Remote and isolated First Nations community members have access to CCC services, with levels of access varying according to the service type. Factors that impacted access include staffing shortages and transportation issues.

While remote and isolated First Nations communities have access to CCC services and the level of access varies according to the type of service provided, no determination could be made as to whether access had improved. Nursing stations provide non-urgent care during regular weekday business hours, and most provide urgent care 24 hours per day, seven days per week. The majority of surveyed nurses and band representatives reported good/very good access to a physician for urgent and non-urgent consults by phone/Internet, and a majority of surveyed stakeholders reported at least medium access to most other areas of access asked about. Factors that were believed to negatively impact access to CCC and federal hospital services included, for example, staffing shortages and transportation issues.

Based on all lines of evidence, remote and isolated First Nations communities community members do have access to CCC services, but no determination could be made as to whether access has improved. The evaluation relied primarily on key informant and survey stakeholder opinion regarding the extent of access to the various elements of CCC. The evaluation found little data or documentation that provided aggregated utilization data (one of the two indicators for this outcome) at the regional or national level or over time. Health Canada's project Recognizing Strengths-Building for the Future, Health Services Delivery Model - Remote and Isolated Communities (HSDM-RI) project provided some limited data on service usage for a sample of 26 communities over 14 days (FNIHB, 2012c, pp. 26-27).

Remote and isolated First Nations communities have good access to CCC services with respect to the hours that these services are available. As reported by key informants and surveyed stakeholders, all nursing stations have regular weekday business hours during which urgent and non-urgent services are provided.

Most nursing stations also provide emergency and urgent care services 24-hours per day, seven days a week. Interviewed and surveyed stakeholders generally believe remote and isolated First Nations community members have reasonable access to CCC services, particularly for access to a physician for urgent and non-urgent consults. Surveyed stakeholders were asked to indicate the extent their community members have access to a range of services, with the majority reporting at least a medium level of access in each case (Table 5).^{Footnote 11} In all cases, nurses reported more positively on access than band representatives.

Access to federal hospitals by members of remote and isolated First Nations communities in Manitoba cannot be determined through the survey, as almost all (n=7, 88%) community respondents said members of their community have not visited either of the two federal hospitals (Norway House Hospital or Percy E. Moore Hospital). One respondent (13%) said they did not know if any community members had ever visited either hospital.

Key informants identified a variety of factors that impede access to CCC services. Most notably, throughout the interview process, key informants spoke of the challenges in recruiting and retaining staff, particularly nurses, to work in remote and isolated First Nations communities (see Section 4.2.4). Several stakeholders also reported that the number of assigned front line nursing positions per facility is not sufficient, given the health-related challenges and/or population growth that communities are experiencing. Insufficient staffing levels due to either high vacancy rates or lack of resources are viewed as contributing to both the stress levels of existing nurses and the ability of nursing stations to meet service demands. As well, because urgent care, by necessity, is given the highest priority, a few key informants noted that staff shortages and demands for urgent care can negatively impact the time available for providing other health services, such as non-urgent care, or other health programming responsibilities of nursing station staff. However, the evaluation did not have access to data or information to assess the extent that staff shortages impacts nursing stations' ability to provide services in the various areas of urgent care, non-urgent care, and other health programming. Recruitment and retention challenges and other factors affecting access to CCC services, such as transportation issues, are discussed in more detail in Section 4.2.4.

Immediate Outcome #3: Appropriate Use of CCC Services Based on Assessed Need

A wide variety of CCC services are available and used, however after-hours care is not always accessed only for urgent care purposes.

There is evidence that a wide variety of CCC services are available and used. The HSDM-RI project, while based only on activity over a 14-day period for 26 communities, reported that of the main reasons individuals visited the nursing stations for CCC services, the majority (69%) were related to diagnosis and/or treatment of injuries or conditions, while 24% were ancillary services that supported the diagnosis and treatment component (e.g., laboratory work, X-rays, or prescription dispensing), 4% were emergency services, and 4% were administrative tasks (FNIHB, 2012c, pp. 52-53). The main reasons for the 69% of diagnostic and treatment visits were health assessment (32%), episodic illness (29%), prescription requests/renewals (17%), and chronic disease care (13%).

Perceptions on the appropriate use of CCC services were mainly related to the use of after-hours CCC services. Evidence indicated that after-hour care is not always used appropriately for only urgent care purposes. Just over half of surveyed nurses (56%), but less than one third (31%) of band representatives, reported that after hours CCC services usage is often for non-urgent purposes.^{Footnote 12} A few key informants reported that some remote and isolated First Nations community members expect 24-hour availability of CCC services even for non-urgent purposes, with some individuals complaining to community leaders when requested after-hours service is not provided. The HSDM-RI project found that almost a quarter (22%) of all services over the 14-day period measured were provided after hours. Of the 733 encounters where nurses recorded the relative urgency of the visit, 70% were considered non-urgent (FNIHB, 2012c, pp. 57-58).

According to interviewed and surveyed stakeholders, educating the community on what constitutes an emergency was the most common strategy used to ensure appropriate use of CCC services. This can occur, for example, through informal discussion between CCC staff and patients (reported by 87% of surveyed nurses and 58% of band representatives), or between community leaders and community members (reported by 62% of nurses and 58% of band representatives), as well as discussions/collaboration between community leaders and the nursing station (reported by 58% of nurses and 54% of band representatives). Other educational formats used included posters in the nursing station waiting rooms, or radio announcements. Strategies successfully used by a few communities included changing or extending nursing station business hours to better meet community needs, encouraging the use of a province-wide available nurse-operated telephone help line, offering flat rates to nurses for after-hours coverage to remove the incentive for over-time (reportedly used for some band-operated nursing stations), and using paramedics for after-hours emergencies.

Immediate Outcome #4: Increasing Capacity (knowledge, skills) of CCC Workforce

The CCC workforce has a high level of professional certification yet not all new CCC nurses have the complete range of required competencies to work in remote and isolated First Nations communities. Increasing the capacity of the workforce is challenged by ongoing nurse recruitment and retention issues.

Nurses working in remote and isolated First Nations communities require a broad skill set and not all new CCC nurses have the complete range of required competencies. Health Canada provides opportunities to increase the capacity of CCC nurses, and most nurses report at least a medium level of satisfaction with educational opportunities and a medium level of participation in available opportunities. However, barriers exist to both providing and participating in educational opportunities (e.g., lack of funds, travel/accommodation costs, and lack of time for nurses).

Certification and Skill Levels of CCC Staff

The current CCC nurse workforce has a high level of professional certification. As reported by surveyed nurse managers, most (80%) CCC nurses are registered nurses, and a small percentage is nurse practitioners (6%), LPNs (2%), or registered practical nurses (2%). Of note, all nurses are required to have current professional registration with their provincial registration body. While it was not possible to assess the increase in CCC staff with certified professional registration, enhancing the capacity of the CCC workforce will continue to present challenges until the ongoing nurse recruitment and retention issues are resolved.

While certification levels of other CCC staff were not available, the majority of surveyed nurse managers reported not having the other identified types of CCC health care provider positions at their nursing station, relying primarily on nursing staff to meet all capacity needs (Table 6). Where nursing stations do have other health care provider types to meet capacity needs, this is most frequently a health care aide, with 25% of nurse managers reporting having this type of assistance.

Various documents as well as interviewed key informants observed that nurses working in remote and isolated First Nations communities require a broader range of knowledge and skills than nurses working in urban areas where more supporting services and specialists are available. For example, the 2010 Audit of Primary Care Nursing Services identified "professional and legal risks as nurses in remote and isolated communities typically require a broader scope of practice" (Health Canada, 2010a, p. 8). Some specific examples of the knowledge and skills required include the ability to adapt to difficult situations (e.g., responding to resource and equipment shortages, adverse weather conditions, and clients' unstable health conditions), and the need to integrate into communities to learn more about them and gain the respect of community members (Misener et al., 2008, p. 57). Nurses must also be able to switch rapidly between specific roles (such as emergency care and long-term care) and practice in a wide scope of care, without much support staff (Kulig, 2005, p. 1). In addition, given the large number of hours nurses spend on call, they must be able to achieve a healthy work-life balance (Stewart & MacLeod, 2005, p. 2).

The majority (68%) of surveyed nurses reported that the range of duties and responsibilities required of them in providing CCC services matches the knowledge, skills, and abilities they acquired through their formal training and previous working experiences. However, close to one quarter (23%) of respondents reported that these duties and responsibilities exceed their knowledge, skills, and abilities. Of the 33 respondents this 23% represents, 24% reported that this disparity in duties and skills impacts quality of care, while 21% said it creates workload issues, and 27% said it created no challenges. Some key informants, particularly nurse educators/ practice consultants, observed that nurses new to providing CCC services often do not have the complete range of competencies required to work in remote and isolated First Nations communities. Several reasons provided for this were that because of recruiting and retention challenges there is greater reliance on inexperienced nurses and that most nursing education programs do not prepare nurses for working in remote and isolated locations.

Efforts have been made to attract and retain Aboriginal people in health careers thereby increasing CCC services' workforce capacity. Just over half of both surveyed nurse managers (51%) and band representatives (58%) reported having Aboriginal nurses on staff at their nursing station. Close to half of nurse managers (47%) and over a third (35%) of band representatives said they have Aboriginal workers in other types of health care positions at their nursing station. A few key informants spoke about the efforts to increase the capacity of Aboriginal health care workers, such as through the Aboriginal Health Human Resource Initiative that encourages Aboriginal health training to allow individuals to remain and work in their communities in some health care capacity.

Orientation and Mentoring for New Health Canada Nurses

All regions provided an orientation and mentoring process for new nurses (e.g., a two-to-three- week in-class session followed by a two-to-three- week field session with mentoring from either a practice consultant or senior nurse, such as the nurse manager). Most (87%) surveyed nurses said they received orientation and training when they first started working at a remote and isolated First Nations community nursing station. Most surveyed nurses said this orientation/training prepared them to at least a medium level, and half or more reported they received good/very good preparation in understanding each of the following areas:^{Footnote 13}

• the range of CCC services they would need to provide, where 58% said the orientation gave them good/very preparation and 25% said medium preparation;
• how to provide culturally appropriate services, where 52% reported good/very preparation and 28% medium preparation; and
• the community in which they would be providing services, where 50% reported good/very good preparation and 29% said medium preparation.

Most surveyed nurses also said they received some type of mentoring when they first started as a health care provider at their nursing stations: a sizeable proportion of nurses (43%) said they received some type of formal mentoring arranged by their employer, and 59% of nurses reported receiving informal mentoring by a more experienced colleague. Twenty-one percent of nurses said they had received no mentoring. Key informants identified mentoring as valuable to provide new nurses with support for developing the comfort and competency needed to practice in remote and isolated First Nations communities and to assist them in integrating into the community. Absence of mentoring is viewed by key informants as having the potential to contribute to stressful situations for new nurses and can negatively affect retention rates.

The 2010 Audit of Primary Care Nursing Services noted that orientation generally involves two weeks in class and two to three weeks in the field. The audit also observed, however, that in earlier years (prior to the evaluation period), the department and regions offered much more extensive training programs for CCC nurses. One of the recommendations of the audit was for development of a training program for new nurses that "adequately prepares them to work in remote and isolated locations." FNIHB's management response acknowledged the need for a training program, and indicated that an Education Strategy was being undertaken through the National Nursing Innovation Strategy program. One of the activities of the Education Strategy is to develop "a national education program in an education institution" (Health Canada, 2010a, p. 12).

Continuing Education and Professional Development Opportunities

As indicated by all lines of evidence, Health Canada strives to increase the capacity of the CCC workforce through opportunities for continuing education/professional development and to upgrade nurses' certification levels (e.g., from a diploma to a degree, or from a degree to a master's level nurse practitioner certification). Continuing education is considered important by key informants for providing nurses with the advanced skill set needed for practicing in remote and isolated First Nations communities. Nurses are offered both mandatory and optional continuing education opportunities, with nurses choosing the latter to meet their individual learning plan. Key informants identified a variety of formats through which continuing education is offered to facilitate participation by nurses in remote and isolated First Nations communities, such as through online modules, webinars, teleconferences, or telehealth. Also, nurses may be able to travel to participate in conferences, forums, or other educational opportunities. Nurses can also access funding assistance and educational leave for upgrading their certification levels, such as from a registered nurse to a nurse practitioner. Some regions are actively encouraging nurses to obtain their nurse practitioner, as these nurses have an expanded scope of practice more suitable to the requirements of working in remote and isolated First Nations communities. While contribution agreements include funding provisions for professional development for band-employed nurses, Health Canada also provides these nurses access to the continuing education activities offered by regions to Health Canada nurses.

Surveyed nurses report that they are generally satisfied with educational and professional development opportunities available to them, with 87% of respondents indicating at least a medium level of satisfaction (54% indicated they were satisfied/very satisfied with opportunities available).^{Footnote 14} Surveyed nurses were also asked to what extent they have been able to participate in available opportunities, with most (82%) reporting at least a medium level of participation, but almost half (46%) reporting a high level of participation.^{Footnote 15}

Challenges in Offering and Participating in Continuing Education and Professional Development Opportunities

Stakeholders identified a variety of challenges in both offering and participating in continuing education and professional development opportunities. As shown in Table 6, the challenges most frequently reported by surveyed nurses were that they do not have enough time to participate (65%) or the hours or times offered do not fit their schedule (58%). Interviewed key informants said that balancing work and educational requirements can be challenging for nurses, particularly when nurses are required to complete a full work day at the clinic and then are on call in the evening.

Most interviewed key informants (who were speaking from the perspective of Regional and National Offices that would be offering opportunities) identified the availability of funds and travel costs as the main challenges to both providing and participating in educational opportunities. Travel costs to and from isolated and remote communities are high and must include accommodation costs; 44% of surveyed nurses identified travel and/or accommodation costs as a barrier. Finding and funding relief personnel when the nurse needs to travel out of the community are further challenges mentioned by key informants. Almost half (49%) of surveyed nurses noted that there is no one available to cover their position for them to attend available opportunities, and almost the same proportion (47%) said they would have to take a leave from their position in order to attend. Key informants observed that when nurses leave their positions temporarily to pursue continuing education this can negatively impact continuity of services to clients, particularly if there are already vacant positions at the clinic.
While (as noted above) providing educational opportunities by telephone or Internet is a strategy for providing access to continuing education, Internet connectivity and other technological issues can challenge nurses' ability to participate, as reported by both key informants and 54% of surveyed nurses.

One recent scan conducted in early 2012 of regional educational activities found significant variation across regions for reasons similar to those mentioned above (e.g., available funding, staff recruitment and retention, availability of replacement staff), as well as due to the differences in the numbers of nurses across regions.^{Footnote 16} The scan also found that regions did not have easily accessible or accurate data on educational activities, and identified a "requirement to establish a data system that supports the monitoring of nursing education to ensure professional and fiscal accountability as the Branch evolves" (FNIHB, 2012d).

Immediate Outcome #5: Increased First Nations Management of Clinical and Client Care Services

There are opportunities for increasing the number of First Nations communities managing their CCC services, primarily in Manitoba and Ontario.

About two thirds of nursing stations are currently managed by Health Canada and there are opportunities for continued increased First Nations management of CCC services. Quebec and Saskatchewan's nursing stations are almost entirely band-operated; other regions are mainly Health Canada-operated. Some initiatives are underway for increased First Nations management, such as the recent British Columbia Tripartite Framework Agreement on First Nations Health Governance.

Currently, approximately one third of nursing stations in remote and isolated First Nations communities are band-operated, and the remaining two thirds are Health Canada-operated. Quebec and Saskatchewan's nursing services are almost entirely band-operated, whereas nursing stations in Alberta, Manitoba, Ontario, and the Pacific region are mainly Health Canada-operated. Below are the numbers of nursing stations that are band-operated for each region:

• Saskatchewan: 12 of 12 nursing stations
• Quebec: 9 of 11 nursing stations
• Pacific: 2 of 9 nursing stations
• Alberta: 1 of 5 nursing stations
• Ontario: 4 of 26 nursing stations
• Manitoba: 1 of 22 nursing stations

Key informants generally could not indicate why some communities have chosen to assume management of CCC services and others have not. Some evidence exists on continued progress towards increased First Nations management, such as through the recent British Columbia Tripartite Framework Agreement on First Nation Health Governance. The Tripartite Agreement emphasizes the development of an integrated health system involving many partners, including the First Nations Health Authority, First Nations health providers, British Columbia Health Authorities, the British Columbia Ministry of Health, and Health Canada. The agreement has a number of intended goals, such as improved accessibility, quality, efficiency, and cultural appropriateness of health programs for First Nations. A key aspect of the agreement is transfer of federal health programs to the First Nations Health Authority, which includes the "planning, design, management and delivery of First Nations Health Programs" ("British Columbia tripartite framework agreement on First Nations health governance," 2011, p. 8).

Also, while not a direct measure of First Nations management of CCC services, Community-Based Reporting Template data (which includes more than just remote and isolated First Nations communities) does provide some indication of First Nations involvement in planning health programming. According to the 2009-10 and 2010-11 Community-Based Reporting Template, 90% and 92%, respectively, of responding communities confirmed they regularly consult with community members for planning health programs (FNIHB, unpublished report, 2010, p. 4, 2011, pp. 6-7).^{Footnote 17}

Some key informants from each region spoke of First Nations communities experiencing similar challenges with their nursing stations as Health Canada-operated stations. In particular, challenges included: recruiting and retaining staff; addressing the varying levels of management capacity and turnover rates in communities; and ensuring good relations between the community leadership (including the Health Director) and the nursing staff.

A few key informants observed that contribution agreement funding has not increased to reflect increasing populations or increasing costs. Constraints on resources can challenge a Band's capacity for providing competitive salaries for nurses. Under their contribution agreement requirements, First Nations are responsible for any expenditure above their funding amounts. As the CCC program is a mandatory service, First Nations managing their nursing stations may find it necessary to transfer funds from other programming to meet any budget shortfalls for CCC services. A few key informants, however, also indicated that some First Nations communities are better positioned to manage their nursing station than Health Canada, mainly due to the ability to operate with more flexibility compared to Health Canada. For example, the First Nations communities are not bound by the same collective agreements as Health Canada, and can more easily negotiate with employees regarding overtime pay and have greater flexibility with their employee hiring or dismissal processes.

Immediate Outcome #6: Increased Collaboration and Awareness/Understanding of Service Delivery Arrangements, Service Requirements, and Accountabilities

Collaboration and linkages between service providers are occurring but there is room for improvement specifically with services related to mental health, physiotherapy, and palliative care. Lack of information sharing between provincial health systems and nursing stations can create issues with case coordination and continuity of care.
Community-Based Reporting Template reporting provides some, but not complete, information on the numbers of collaborative arrangements in place. For example, Manitoba's 2011-12 Community-Based Reporting Template report identified 22 agreements in place with, for example, regional health authorities, hospitals, physicians, and universities. Health Canada should be in a position to increase reporting on collaborative arrangements in the future provided all communities submit their completed Community-Based Reporting Template regularly.

The Community-Based Reporting Template reporting for 2009-10 and 2010-11 provides some evidence of collaboration in health care service delivery (FNIHB, unpublished report, 2010, p. 4, 2011, p. 7).

For both years, most responding communities reported that they link their services with other health care organizations (95% and 93%, respectively), and that health and social services collaborate when providing community services and support (93% and 91%, respectively).^{Footnote 18}

The evaluation relied primarily on stakeholder opinion regarding the extent of collaboration and linkages. Most interviewed Regional Directors and Directors of Nursing/Regional Nursing Officers reported that there are collaboration between the various levels of health services providers in their regions, with a few commenting that collaboration are improving between the CCC service providers and the provincial health departments or the regional health authorities. Several examples include committees that meet to resolve issues and discuss improved approaches, with representation from the various levels of government, regional health authorities, and/or First Nations.

Some key informants identified challenges with the sharing of data and information between nursing stations and provincial health systems, due to legislative and policy barriers between provincial and federal systems. For example, challenges with sharing hospital discharge information to facilitate continuity of care when patients return to the community, and providing nurses at nursing stations access to provincially held immunization records were noted. Data inaccessibility creates issues with case coordination and continuity of care as well as with planning at the regional level. Increased information sharing would require negotiation of agreements to resolve differences in legislation and regulations between provincial and federal systems. A specific example is an information sharing agreement negotiated in Alberta so that the province could share information on the Newborn Metabolic Screen Test with nursing stations.

Interviewed Regional Nurse Managers mainly spoke of collaborative efforts at the operational and community level, with a few reporting that nursing stations have good collaborative relationships with other health care providers (e.g., physicians used for consulting, or other community health care workers). A few key informants observed that collaboration and integration with other health care providers varies among communities or that, while collaboration is occurring and is, in some instances, increasing, there is room for improvement.

Surveyed stakeholders also reported varying perspectives on the extent of collaboration with various other health care providers, as summarized in Table 7 and 8. Nurses were asked to report on the extent of collaboration between their nursing station and other health care providers within their community for providing a continuum of care to the community (see Table 8).

While most respondents reported at least a medium level of linkage/collaboration, the majority (64%) of nurses reported good/very good linkages only with home care providers, and less than half of respondents reported good/very good linkages with public health programs (48%) or programs related to health promotion and/or disease prevention (36%).^{Footnote 19} For both of the latter cases, sample sizes were small (n=31 and n=22, respectively).

Both surveyed nurses and band representatives were asked to report on collaboration between their nursing station and external service providers, with nurses generally reporting more positively on such linkages (see Table 9).^{Footnote 20}

Hospitals are the only area where a majority of both nurses and band representatives reported good/very good linkages. Two thirds of nurses but just over one third of band representatives reported good/very good linkages with general practitioner physicians; however, the majority of band representatives reported at least a medium level of linkages in this area. A few key informants also reported that challenges can exist when physicians providing services to remote and isolated First Nations communities stay in their positions for only short time periods, affecting continuity of care to patients. Close to three quarters of nurses, but less than half of band representatives (42%) reported at least a medium level of linkages with specialist physicians. Nurses and band representatives were least positive about linkages to external services related to mental health, physiotherapy, and palliative care.

A few key informants observed that their region does strive to improve relationships and collaboration with provincial and health authority representatives. Collaboration is viewed as important for facilitating transfer of patients to health facilities for urgent care, or for ensuring provision of physician services.

Telehealth initiatives were also identified as a successful means of facilitating consultation and collaboration between nursing stations and physicians/specialists, with a few key informants observing that this type of service delivery was increasing. However, as was reported in Table4, nurses were split when reporting on access to telehealth services, with 32% reporting good/very good access, 29% medium access, and 33% poor/very poor access.

Immediate Outcome #7: Increased Awareness and Understanding of Policies, Standards, Guidelines, and Best Practices/Evidence-based Information in Service Delivery

Awareness of certain policies, standards and guidelines is evident through high levels of usage.

Awareness of certain policies, standards, and guidelines is evident (and understanding is assumed), in particular FNIHB's Clinical Practice Guidelines, the Nursing Station Formulary and Drug Classification System, FNIHB's Policy and Procedures on Controlled Substances, and the Canadian Diabetes Association Practice Guidelines. Health Canada uses various means to ensure nurses are aware of relevant policies, standards, and guidelines.

Client treatment plans are expected to incorporate current best practice guidelines (Health Canada, 2011a, p. 11). For example, FNIHB's Clinical Practice Guidelines were developed as a reference tool to assist nurses working in remote and isolated locations with identifying, diagnosing, and treating illness and other health issues (Health Canada, 2010b). Knowledge and use of such best practices begins with awareness and understanding of relevant policies, standards, and guidelines.

The document and data review revealed no ongoing collection of data to assess and measure awareness and understanding of policies, standards, guidelines, and best practices/evidence-based information in service delivery. However, high current awareness of certain policies, standards, and guidelines is evident through the large proportion of surveyed nurses reporting regular usage of these documents (Table 10). In particular, nurses report a high level of regular use, and therefore awareness, of FNIHB's Clinical Practice Guidelines, the Nursing Station Formulary and Drug Classification System, FNIHB's Policy and Procedures on Controlled Substances, and the Canadian Diabetes Association Practice Guidelines. The survey data does not give an indication of understanding of the information, other than to assume that nurses have an understanding of the information they are using regularly.

As reported by key informants, Health Canada uses various mechanisms for ensuring nurses are informed of the relevant policies, standards, and guidelines they are expected to follow in their practice. A component of the orientation and training provided to new CCC nurses includes introduction to and review of relevant policies, standards, and guidelines. Nurses are informed of new or revised policies, standards, and guidelines through dissemination of the material and communications on the materials to nurses and nursing stations (e.g., emailing electronic copies and/or mailing hard copies). Dissemination of the material is undertaken by the party responsible for the new or revised policy, standard, or guideline (e.g., FNIHB, the province, or the provincial nursing association). Depending on the material, Health Canada may conduct teleconferences or forums to review some materials.

4.5 Performance: Issue #4 - Achievement of Intermediate Outcomes (Effectiveness)

Intermediate Outcome #1: Timely Collaboration/System Response to Clinical and Client Care Needs

Key informants and survey stakeholders reported that collaboration is occurring, but there is need for continued effort in this area.

As already discussed for immediate outcome 6^{Footnote 21}, key informants and survey stakeholders reported that collaboration and linkages are occurring, but that there is a need for continued efforts in this area.

No data or baseline measures were available for assessing if system responses are timely. The only indication is from 2011-12 Community-Based Reporting Template reporting which required nursing stations to report the number of referrals to outside agencies in a variety of categories. However, the 2011-12 aggregated data is not yet available, although the evaluation did have access to Manitoba's report. Manitoba nursing stations made 25,278 referrals in 2011-12, most (65%) of which were to physician specialists, while 15% were to facilities for urgent care, 7% to diagnostic clinics, and 6% to general practitioners/family physicians (FNIHB, 2012e, p. 11).

Key informants identified several factors that facilitate system response. System response, for example, is facilitated by remote and isolated First Nations community members' access to the nursing station for 24-hour urgent care and weekday non-urgent care. A few key informants noted that while nursing stations can be very busy, patients often do not have the same wait times that would be experienced at other health facilities, such as hospital emergency rooms. Key informants who could speak on referrals noted that patients receive effective referrals and that, at times, remote and isolated First Nations community members receive timelier referrals to specialists than individuals in urban areas. As well, a few key informants identified telehealth initiatives as facilitating effective and timely collaboration between caregivers for providing primary care to remote and isolated First Nations communities.

The issues that provide challenges in providing timely system response are detailed in Section 4.2.4, with the key factors being staffing shortages and high turnover rates.

Intermediate Outcome #2: Increased Use of Policies, Standards, Guidelines, Best Practices, and Evidence-based Information for CCC Quality Improvements

Virtually all surveyed nurses reported using certain polices, such as FNIHB's Clinical Practice Guidelines, which was viewed as a valuable reference guide. However, nurses also experienced challenges in accessing/using relevant policies mainly due to internet connectivity issues or a lack of electronic copies that could be easily accessed when seeing patients. About half of responding communities and nursing stations reported having a process for addressing client concerns and improving quality services. Pilots are underway to accredit nursing stations to national standards.

Use of Policies, Standards, and Guidelines

As reported in Table 9 under immediate outcome 7^{Footnote 22}, a large proportion of surveyed CCC nurses regularly used certain policies, standards, and guidelines (however, due to the lack of a baseline measure, it was not possible to conclude if use had increased). In particular, most nurses regularly use FNIHB's Clinical Practice Guidelines (96%); the Nursing Station Formulary and Drug Classification System (92%); FNIHB's Policy and Procedures on Controlled Substances (88%); and the Canadian Diabetes Association Practice Guidelines (77%). As well, a majority (58%) of nurses regularly use the Regional Orientation Manual and close to one third (32%) use the Consensus Guidelines (32%). Key informants also reported wide use of the Clinical Practice Guidelines by nurses and that nurses view them as a valuable reference guide.

Surveyed nurses were asked to identify any challenges or barriers they experience in using the relevant policies, standards, and guidelines. While usage of key policies overall is high, there can also be challenges in accessing some policies in some situations. Just over half of nurses identified internet connectivity issues for accessing online copies as a barrier (58%), as well as the ability to have electronic copies easily available to them while seeing patients (52%). A few key informants also noted that nursing stations rely primarily on printed material, which can present challenges when information is not updated with the most recent versions or when copies go missing. Few respondents (16%) said that no barriers/challenges existed to them in using the relevant policies, standards, or guidelines.

Processes for Managing Client Concerns

As a best practice for quality improvements, most remote and isolated First Nations communities have some type of process for addressing client concerns with services received at the nursing station. Close to half (47%) of surveyed nurses said their nursing station has a formal process in place for addressing concerns, and one third (35%) said they use an informal process. Of band representatives, 42% said their community used a formal approach, 23% said an informal process, 19% said there was no process for addressing concerns, and 15% either did not know or did not respond to the question. From the 2009-10 and 2010-11 Community-Based Reporting Template reporting, half of responding communities (49% and 50%, respectively) said their community had a quality improvement plan, although again this includes more than remote and isolated First Nations communities (FNIHB, unpublished report, 2010, p. 1, 2011, p. 1).^{Footnote 23}

Accreditation for CCC Services

The accreditation of CCC services serves as an indicator for the use of policies and best practices, as these are needed to meet federal, provincial or territorial standards, and also as an indicator for quality improvements. No nursing stations are currently accredited, although two locations are piloting standards for accreditation purposes. The 2010 Audit of Primary Care Nursing Services mentioned that an initiative for accreditation of First Nations and Inuit Community Health Centres began in 2000 and that this has been extended to remote and isolated First Nations community nursing stations, with pilot locations expected for 2011 (Health Canada, 2010a, p. 15). According to a Health Canada representative, the Standards for the Remote/Isolated Health Services were piloted in two locations in October 2012. Accreditation Canada is currently reviewing the evaluation of these pilots. The next step is to clarify requirements, review the standards for opportunities to add to the guidelines where items were not clear, and add a glossary of terms to the Medication Management for Remote/Isolated Health Services Standards.^{Footnote 24}

4.6 Performance: Issue #4 - Achievement of Long-Term Outcomes (Effectiveness)

Long-term Outcome: Clinical and Client Care Services Are Responsive to the Needs of First Nations Individuals and Communities (PAA 3.1.3)
The CCC program is responsive to the needs of First Nations individuals and communities through provision of urgent and non-urgent health care services within individuals' own communities. Most (81%) surveyed band representatives report their community members have a medium to very high level of satisfaction with CCC services received.

The CCC program responds to remote and isolated First Nations communities' lack of access to provincial health systems due to geographical isolation. The CCC program's services provide urgent and non-urgent health care services within individuals' own communities, with urgent care services available mainly 24 hours a day, seven days a week. First Nations individuals and communities have a range of health care needs, and often experience higher rates of certain diseases, as well as higher rates of substance abuse and addiction issues, mental health issues, and injuries related to acts of violence and accidents. All of these, as well as an aging population and growing communities, are creating increased demand for CCC services.

Surveyed band representatives reported on their community members' satisfaction with CCC services received. Most (81%) respondents reported at least a medium level of satisfaction, with 38% reporting their community members were satisfied/very satisfied. The most frequent reason provided for any dissatisfaction was that services were not always sufficiently responsive or timely (38%).

Stakeholders were asked to suggest potential ways to improve CCC services and a variety of suggestions were provided. The most common suggestions were related to increasing resources, with 21% of nurses suggesting increasing the number of nurses/staff, and 23% of band representatives identifying the need for additional funding, such as for hiring more staff. Furthermore, 23% of band representatives identified the need for improvements related to internal service providers, such as recruitment of permanent staff at the nursing stations, and decreased use of agency nurses. Other suggestions related to internal CCC resources (identified by a small proportion of respondents in each case) included increasing availability and accessibility to training opportunities, taking measures to ensure nurses have the appropriate skills/training, improving recruitment and retention strategies, and improving access to equipment and/or laboratory resources. Suggestions related to external health care resources (again identified by a small proportion of surveyed stakeholders in each case) included increasing accessibility to external service providers (e.g., related to physician specialists, or mental health and addictions), providing medical facilities (e.g., mental health facilities, hospital) within the community, and improving collaboration/communication between health care stakeholders (e.g., internal, external, and governments). Key informants also identified the need for increased collaboration between stakeholders and a greater integration of services.

Challenges and/or Barriers Encountered

Staff recruitment and retention is an important and ongoing challenge for providing CCC services, particularly for nursing. These challenges are due to various factors (e.g., an overall nursing shortage in Canada; heavy workloads; geographic isolation; lack of the amenities and support found in urban areas; and safety issues). Health Canada has implemented various strategies to respond to these challenges. Other challenges to providing responsive CCC services relate to: (1) changes being implemented in some regions that increase training and educational requirements; and (2) the provision of services in geographically isolated locations (e.g., Internet connectivity, transportation issues, high shipping costs, access to other services, and hiring and retaining other types of health care workers).

Recruitment and Retention Challenges

CCC services within remote and isolated First Nations communities are primarily nurse-led, with nursing stations, relying primarily on nurses for responding to the community's primary care needs. Staffing shortages, as well as lack of continuity of staff due to high turnover and use of short-term contract agency nurses, can negatively affect responsiveness in terms of continuity of care and the capacity for the nursing station to address the volume and nature of client health care needs. Furthermore, challenges in recruiting and retaining nurses with the full range of required competencies for nursing in remote and isolated locations further affects capacity to respond to community health care needs. While CCC services are being delivered and progress is being made towards the intended outcomes, challenges still exist with attracting and retaining nurses to work in remote and isolated communities. The reasons for recruitment and retention challenges revealed through the evaluation can be categorized into three main overall themes:

1. Remote and isolated communities are not viewed as desirable locations in which to live and work.

   Key informants and documents cited the social, professional, and geographic isolation as the main reasons for the reluctance to reside in remote and isolated communities (Health Canada, 2010a, p. 9, 2011d, p. 4). This includes, for example, lack of access to the amenities, and social and professional supports (friends, family, and colleagues) found in more urban areas, as well as the difficulties and costs involved in travel. As a result of reluctance to live in remote and isolated communities, many nurses prefer to work only part-time, where they rotate in and out of the community over designated time periods, depending on their employment arrangement.

2. Perceived wage disparities, as per key informants, exist between salaries offered to CCC nurses and those available through provincial health systems and private employers.

   Many key informants suggested there were wage disparities between Health Canada nursing salaries and those offered by other employers. Reviewed documents made references to higher provincial nursing salaries compared to federal salaries (FNIHB, 2012f, p. 2), higher federal nursing salaries compared to salaries offered to band-employed nurses (LTSC, 2010, p. 8), and, for nurse practitioners, higher salaries offered by the private sector compared to federal public employers (Health Canada, 2011d, p. 11). It is worth noting, however, that the majority of federal, general-duty nurses working in remote and isolated communities are at the Community Health Nurse level 3 salary (CHN-3). The base salary for this level is roughly equivalent to the base salary for general duty nurses provided by most provinces. Nurses are also provided with various incentives (e.g., sign-on bonuses, retention and education allowances) although sign-on bonuses and other incentives are also provided to their provincial counterparts. It is also noteworthy that given the remote and isolated scope of practice, there are limited provincial comparators. Many key informants said an overall nursing shortage exists in Canada, and, therefore, nurses can be selective in their employment choice. A nursing workforce backgrounder by the Canadian Federation of Nurses Unions identifies that, despite an approximate 2% annual growth in the Canadian nursing workforce since 2006, a shortfall of 22,000 nurses existed in Canada as of early 2012 and was expected to increase to 60,000 by 2022 (Canadian Federation of Nurses Unions, 2012, pp. 1-3).

3. Heavy workloads and a challenging work environment are common at CCC nursing stations.

   Many nurses find the work environment in remote and isolated First Nations communities challenging, with heavy workloads, no on-site physician assistance, lack of other health care and administrative support, requirements to work overtime and on-call hours, and safety and security issues (FNIHB & EPOHD, 2011, pp. 50-51; Health Canada, 2010a, p. 9, 2011d, pp. 4,11-12). An example of a safety and security issue that needs to be addressed was noted in Health Canada's 2011 Northern Nursing Recruitment and Retention Strategy. It identified the need for increased evening and weekend security coverage when nurses are providing urgent and emergency care (Health Canada, 2011d, p. 12). As an example of safety concerns, Critical Incident Stress Management (CISM) reporting shows some increase in both verbal and physical assaults to nurses in First Nations communities for each of the three years between 2008-09 to 2010-11, with verbal assaults increasing overall from 108 to 182 and physical assaults from 5 to 19 (FNIHB & EPOHD, 2011, p. 7).^{Footnote 25} Nurses in general are considered at high risk for on-the-job abuse by patients. Based on findings from a 2005 Statistics Canada survey, 34% of nurses working in hospitals and long-term care facilities reported physical assault by patients, with 40% of nurses who worked evenings and night shifts reporting assaults compared to 23% of nurses working day shifts (Shields & Wilkins, 2009, pp. 1&4).

Recruitment and retention challenges result in high vacancy rates, insufficient staff levels, and increased workload and stress levels for remaining nurses, which can further challenge nurse retention. For example, in the survey of stakeholders, a majority of nurses (70%) and band representatives (62%) identified nursing position vacancies at their nursing station, and just over half of nurses (51%) and one third of band representatives (35%) identified vacancies for other types of health care provider positions.

Insufficient staff levels and high workloads can compromise nurses' capacity for provision of services. As previously noted, a few key informants observed that because urgent care must be a priority, high workloads can result in reductions in non-urgent care services, as well as in other Health Canada-delivered health care programming for which the nurses might have responsibilities, such as public health, or health promotion and disease prevention.

As well, recruitment and retention challenges create a need to use costly contract nursing agencies. The Health Services Delivery Model - Remote and Isolated Communities project estimated that contract agency nursing costs accounted for 42% of all nursing costs between 2008-09 and 2010-11, and that the cost of an agency nurse compared to a Health Canada nurse ranged from 1.5 times higher in the Pacific region to 2.2 times higher in the Manitoba region (FNIHB, 2012c, pp. 67-68). Furthermore, key informants reported that consequences of use of short-term agency nurses as well as rotational nursing staff and high staff turnover, can include a lack of continuity of care, loss of corporate memory, and difficulties in establishing relationships with the community. A large proportion of nurse and band representative survey respondents also identified lack of continuity of services due to high staff turnover (71% and 50%, respectively) and the need to use contract nursing agency staff (57% and 58%, respectively).

Health Canada has implemented various strategies to respond to recruitment and retention challenges. For example, the 2011 Northern Nursing Recruitment and Retention Strategy identifies current strategies and recommends new strategies in nine areas of recruitment and retention (Health Canada, 2011d, pp. 13-25). Other strategies being used to meet human resources needs include the time-limited National Nursing Innovation Strategy Program that funds projects to test new and innovative approaches to health care delivery to remote and isolated First Nations communities, such as collaborative health teams, integration of nursing and technology, new hours of operation for nursing stations, and nurse education. Also, the aforementioned Aboriginal Health Human Resources Initiative has been renewed to 2015 to increase the numbers of Aboriginal health care workers and support human health resources (Health Canada, 2012c, pp. 38-39). Key informants also identified other recruiting initiatives such as advertisements in newspapers and journals, participation in health and job fairs, and presentations at nursing programs, with some key informants observing that Health Canada is always in a recruiting mode. However, a few key informants also said that Health Canada's recruitment process is too lengthy and complicated and, as a result, promising candidates often find other positions in the interim and withdraw their interest.

Further potential retention strategies identified by key informants could include financial incentives (e.g., isolated post allowance), provision of additional educational opportunities, and use of flexible employment arrangements such as the part-time arrangements where nurses rotate between the remote and isolated First Nations communities and their home community (e.g., key informants note approximately 80% of Manitoba nurses are part-time staff who live outside the community).

Other Challenges to Providing Responsive CCC Services

The recent or soon-to-be-implemented changes by provincial nursing associations for additional educational/training requirements of nurses practicing in remote and isolated areas can pose a challenge for providing CCC services. For example, British Columbia now has a requirement for a remote nursing certification, and Saskatchewan is in the process of developing a similar requirement through a new designation: Registered Nurse Certified Practice - Rural and Remote RN(C). Both of these changes are intended to provide nurses with additional training to attain the competencies required for working in rural and remote locations (British Columbia Ministry of Health, 2012; Saskatchewan Registered Nurses' Association, 2012, p. 18). However, key informants report that obtaining the training will entail additional costs to Health Canada and/or nurses, and current nurses may find it challenging to access the required training.

As well, the geographic isolation of remote and isolated First Nations communities creates related issues beyond retaining nurses, such as:

• Transportation issues can affect the ability of nursing stations to transport patients to other health care facilities for urgent care. Transportation challenges include poor air transportation systems, lack of access to provincial air or land ambulance services, and/or poor weather conditions that prevent travel. The remoteness of communities also increases transportation costs for shipping supplies and equipment to remote and isolated First Nations communities.
• Connectivity and technology issues (e.g., lack of reliable and consistent Internet connectivity) can create challenges for communicating with other health care providers, accessing practice resources and clinical support tools. Lack of access to provincial health databases and lack of information sharing between health care providers can negatively affect continuity of care and, therefore, system response (e.g., due to privacy considerations, when a patient is released from hospital, nursing stations are not informed and cannot plan for case management upon the patient's return to the community).
• Accessibility issues relate to nursing stations at times experiencing difficulties in accessing a physician for urgent care consults, such as when the on-call physician does not respond in a timely manner. As well, communities can experience a general lack of physicians willing to provide services to remote and isolated First Nations communities, particularly for visiting communities to provide non-urgent care.

4.7 Performance: Issue #5 - Demonstration of Economy and Efficiency

Observations on economy and efficiency noted challenges that impacted the delivery of CCC services including: funding based on historic allocations and outdated population estimates; ongoing staffing challenges that resulted in high vacancies and staff turnover and required use of costly contract nursing agencies; unexpected overtime costs; inefficient record-keeping systems; and a lack of performance information.

The Treasury Board Policy on Evaluation (2009) defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment of economy and efficiency is based on the assumption that departments have standardized performance measurement systems and that financial systems use object costing.

Given the lack of departmental financial data that is linked to the quantity and type of outputs coupled with incomplete program-level expenditure reporting, the evaluation could not conduct an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Further, financial data encompassed more than just CCC services as program oversight costs were embedded within the broader First Nations and Inuit primary care service delivery envelope.

As a result, this evaluation provided observations on economy and efficiency based on the document review, literature review and key informant perspectives. More specifically, observations were provided on overall expenditures and program delivery efficiency. In addition, observations were made regarding the adequacy of performance measurement data.

Observations on Economy

This section provides observations on nursing costs related to CCC and includes key informant views on the sufficiency of program delivery resources.

As described in section 2.4 of the report, CCC expenditures in 2011-12 were $139.7 million in A-Base funding. This is the most representative measure of actual CCC resources during the evaluation period. In addition to the A-Base funding, the review found there was a need to annually supplement primary care nursing costs (including CCC costs) with Integrity Funding to cover funding gaps related to overtime and agency nurse costs, suggesting that the funding levels were not sufficient.^{Footnote 26} The Integrity Funding provided by Treasury Board can be used to supplement shortfalls in nursing costs. Between 2008-09 and 2011-12, $105.23 million of Integrity Funding was used with Manitoba and Ontario^{Footnote 27} accounting for the majority of these funds (44% and 38%, respectively). However, it should be noted that it is not possible to determine how much of the $105.23 million was used by CCC.

Some key informants identified overtime pay as a factor in requiring supplemental funding (i.e., use of the Integrity Funding) for CCC. Overtime pay was a result of nursing stations providing 24-hour, seven days a week, urgent care. Some regions have considered extending nursing station hours and staggering nurses' work hours to not only address overtime costs but also to better meet community needs. The "Health Service Delivery Model - Remote and Isolated" final report identified one project in Quebec successfully using this strategy (FNIHB, 2012g, p. 12); however, changing nursing hours was also viewed as creating challenges as it related to the collective agreement for nursing staff.

External key informants reported that the resources were insufficient to meet the current needs for the CCC program, either for Health Canada or band-operated nursing stations. They viewed the funding models (based primarily on historical allocations) as outdated and that current funding levels are not considered reflective of trends and changes in need at the community level (e.g., increased population and health concerns). A few key informants also observed that, in the past, additional funds were provided according to results of the Community Workload Increase System that monitored population growth and workload in First Nations and Inuit communities, but additional funding based on population growth had not occurred for a number of years.

Observations on Efficiency

This section provides a review of factors that impacted program delivery efficiency. It also includes a discussion of findings from the literature and perspectives from key informants on how to improve efficiency within the current delivery model as well as a discussion on alternative methods to deliver the program to enhance efficiency.

There were a number of strategies identified by key informants to improve program delivery efficiency. For example, many key informants said travel costs were significant, given the geographic distances of communities. One strategy for controlling education-related travel costs was to increase the use of videoconferencing and online educational formats. Other factors reported by key informants that increased costs included education and training of nurses, pharmaceuticals, shipping goods to nursing stations, staff housing, facilities, and facility security.

To improve the program delivery efficiency by which the CCC program could meet current needs of remote and isolated First Nations communities, key informants provided examples of strategies that are being used:

• Key informants from regions (including British Columbia, Alberta, and Ontario) reported one strategy used by their region to facilitate continued provision of services while decreasing reliance on agency nurses was to employ resource teams that travel into communities. The Health Services Delivery Model - Remote and Isolated Communities Project, for example, cited that Alberta used no agency nurses during the 2008-09 to 2010-11 period "due in large part to an effective staff nursing pool that meets their staffing needs" (FNIHB, 2012c, p. 67).

• The Health Services Delivery Model - Remote and Isolated Communities project final report referred to several National Nursing Innovation Strategy Program projects that obtained successful results from incorporating other health care workers, such as pharmacy clerks, midwives, and paramedics into health care teams. Some of the positive results of this approach included increased staff availability and more efficient use of staff time within their scope of practice (FNIHB, 2012g, pp. 11-12).

The literature suggested that multidisciplinary teams with flexible roles and responsibilities for team members may be more efficient than the current model in the provision of care. The literature was specific in identifying this approach for remote and isolated regions including aboriginal communities. This type of team composition would allow members to practice to the highest level of their skill set, knowledge, and credentials and may help to alleviate staff shortages and to overcome cultural barriers (Baker & Denis, 2011, p.11-12; Birks et al., 2010, p.31; Rygh & Hjortdahl, 2007, p.5).

Other potential strategies to enhance program delivery efficiency include:

• increasing the use of and access to Telehealth services which are viewed as an effective and efficient means for nursing stations to access other services for patient consultations, such as with physicians, as well as for accessing training and educational activities, and collaborating with other care-givers;
• implementing an inventory management system that would facilitate monitoring and maintenance of nursing stations' stocks of pharmaceuticals and medical supplies and could contribute to ensuring nursing stations have needed supplies on hand while at the same time reducing wastage of expired and unneeded medications; and
• implementing a quality assurance system for monitoring and maintaining nursing station medical equipment.

In addition to providing examples of strategies being used to improve program delivery efficiency, key informants were also asked for their input on alternative methods of delivery to enhance efficiency. The literature provided alternative methods as well.

The most common suggestion related to delivery models for enhancing program delivery efficiency involved more collaborative arrangements with partners and integration of services. Several surveyed stakeholders also identified the need for improved collaboration with external service providers, including hospitals and provincial health care providers. The literature also noted the benefits of collaborative arrangements for delivering primary care to rural and remote locations. For example, effective communication and agreements between various service providers that are internal and external to the community are considered essential to streamline the referral process and ensure coordinated care in rural and remote communities (Baker & Denis, 2011, p.11-12; Wakerman et al., 2009, p.89-90).

The literature also suggested that while delivery of health care through discrete services is possible in larger, more accessible communities, remote, less populated areas require an integrated and comprehensive model in order to minimize cost and maximize accessibility (FNIHB, 2012d, p.12-19; Wakerman et al., 2008, p.5). The Health Services Delivery Model - Remote and Isolated Communities Project final report recommended that Health Canada and FNIHB adopt an Interdisciplinary Expanded Care model for CCC service delivery. This model involves integration of services, such as those related to health care, disease prevention and health promotion, and social services to better serve communities and reduce inefficiencies and duplications (FNIHB, 2012g, pp. 15-18, 35).

Adequacy of Performance Measurement Data

Based on documents and data reviewed as well as input from key informant interviews, the evaluation found that no ongoing performance measurement data collection and reporting system was in place to facilitate CCC performance reporting at the community, regional, and national level. This resulted in the lack of baseline data to determine changes in program performance. This was the case, despite the existence of a logic model and a performance measurement strategy. There was also the lack of clarity or precision with performance indicators, further hindering the measurement of outcomes (e.g., measuring if an expected outcome has 'increased' or 'improved'). The lack of data collection/data usage may also inhibit analyzing and reporting on the actual level of service provided at the regional (and national level), measuring workload and resource utilization, and quantifying the impact of providing CCC services. In addition, both the 2010 Audit of Primary Care Nursing Services and recent Performance Report for Community-Based Primary Care Nursing Services identified the lack of a performance measurement system and gaps in data collection, analysis, and reporting, and that such deficiencies impede evidence-based decision making (FNIHB, 2012b, p. 19; Health Canada, 2010a, p. 16-18).

Systematic data collection was hindered by the absence of an electronic reporting system to facilitate data collection, storage, and retrieval. Key informants reported that Health Canada-operated nursing stations collected and reported to their region through paper-based monthly activity logs or month end reports. The document and data review and interviews revealed that this data was not always provided to regions, or the data provided was incomplete. Furthermore, regions did not always electronically transcribe or use the information that was provided. The paper-based data collection system challenged the compilation and analysis of data by regions. In essence, even where nursing stations had taken the time to complete and submit their reports, the information may never be used. Regions also did not use this information for reporting to National Headquarters; nor, according to some key informants, was the information requested.

Moreover, data collection for performance measurement of the CCC program was also complicated by multiple delivery agents (e.g., Health Canada and First Nations organizations) and overlapping responsibilities of CCC service providers with other program areas (e.g., Community-Based Reporting Template data included more than just the CCC program, and nurses were usually responsible for more than just the CCC program). All of these factors hindered the ability to collect and utilize performance measurement data.

Key informants indicated that band-operated nursing stations reported according to their contribution agreements, with financial information reported through financial audits and other data requirements through the Community-Based Reporting Template. Some key informants questioned the usefulness of the CCC information collected through the Community-Based Reporting Template, stating, for example, that some transferred communities are challenged in accurately reporting the information, or that the data just provided counts of specific information and did not provide useful indicators for analyzing CCC service usage.

Some regions are implementing their own systems of data collection and reporting. For example, Alberta collected and analyzed 2011 data from nursing stations through the Nursing Activity Reporting System to compile a report on client utilization. The resulting information is considered useful to plan, develop, and evaluate health care programs and services in Alberta First Nations communities, especially remote and isolated ones (Health Canada, 2011c, pp. 4-5).

Another evolution in performance measurement has been with the Community-Based Reporting Template. The format changed in 2011-12 and now requires reporting in a number of areas consistent with the CCC performance measurement indicators. While no 2011-12 Community-Based Reporting Template aggregate analysis was available for this evaluation, if the data is collected, analyzed, and reported on consistently, this information could provide valuable additional information for assessing performance based on the current logic model and performance measures.

5. Conclusions

5.1 Relevance Conclusions

Continued Need

The CCC program continues to address a demonstrable need that is responsive to the health needs of First Nations communities by addressing on-going health demands. These demands are related to higher rates of illness and changing demographics, which require CCC services in communities that would otherwise not have these services.

First Nations individuals and communities often experience higher rates of certain diseases (including chronic diseases and co-morbid conditions), substance abuse and addiction issues, mental health issues, and injuries related to acts of violence and accidents. First Nations individuals also often rank lower in social determinants of health, such as income, education, and employment, which are linked to poorer health outcomes. All of these, as well as an aging population and growing communities, are creating increased demand for CCC services. Given the geographic isolation of remote and isolated First Nations communities, members of these communities do not have the same access to health care services as other Canadians. CCC services are intended to respond to this inequity, providing remote and isolated First Nations community members access to primary care services within their communities.

Alignment with Government Priorities

The CCC program's service delivery in remote and isolated First Nations communities align with federal government priorities as articulated in Health Canada's 2012-13 Report on Plans and Priorities, federal budgets and Speeches from the Throne.

The CCC program aligns with federal roles and responsibilities as well as their jurisdictional and mandated role, as evidenced by various Acts and policies related to health and First Nations. The CCC program is directly linked to Health Canada's key program activities and strategic outcomes. Health Canada's 2012-13 Report on Plans and Priorities articulates continued financial support to primary health care services for First Nations. Federal budgets have consistently committed significant funds to initiatives that aim to improve the health outcomes of First Nations, including renewal of the Aboriginal Health Human Resources Initiative to 2015. Speeches from the Throne also demonstrate the federal government's commitment to building partnerships with First Nations.

Alignment with Federal Roles and Responsibilities

The CCC program aligns well with the role of the federal government to provide or fund health programs for First Nations through the First Nations and Inuit Health Branch of Health Canada. The Program is consistent with the Indian Health Policy and departmental mission and mandate statements.

The CCC program aligns with federal roles and responsibilities and is directly linked to Health Canada's key program activities as a sub-sub activity to Health Canada's key program activity 3.1 of the PAA as well as to Health Canada's Organizational Priority III to Strengthen First Nations and Inuit Health Programming and FNIHB's stated priorities. The CCC program is also congruent with the department's jurisdictional and mandated role as evidenced by various Acts and policies related to health and First Nations, such as the Constitution Act, 1867, the Indian Act, 1876, the Canada Health Act, 1984, the Indian Health Policy, and the Indian Health Transfer Policy.

5.2 Performance Conclusions

Expected Outcomes

The CCC program is progressing towards its intended outcomes and is responsive to the needs of First Nations individuals and communities through the provision of urgent and non-urgent health care services.

The main challenges in program delivery stemmed from staff recruitment and retention, challenges in linking with other service providers, lack of information sharing of client data and systematic tracking of both human resources and performance measurement data.

A wide variety of services are available and there is generally good access to them. Community members are generally satisfied with these services. Through the access to these services, CCC also increased the awareness of self-care and illness prevention in communities. The evaluation found increased awareness of disease prevention/health promotion by First Nations. However, the evaluation also revealed that challenges associated with the timeliness of services affected the provision and coordination of care. This was due to staffing challenges (e.g., recruitment and retention), transportation issues, lack of access to physicians and provincial health databases, and lack of information sharing of client data.

In delivering CCC services, the evaluation found that collaboration and linkages between service providers are occurring. However, improvements could be made, specifically with services related to mental health, physiotherapy, and palliative care.

Staff recruitment and retention is a major and ongoing challenge for providing CCC services, particularly for nursing. These challenges are the result of various factors (e.g., an overall nursing shortage in Canada; heavy workloads; geographic isolation; lack of the amenities and supports found in urban areas; and safety issues). Recruitment and retention challenges at nursing stations can compromise provision of services, increase workloads and stress levels of remaining nurses, and, also create a need to use costly contract nursing agencies. Health Canada has implemented various strategies to respond to recruitment and retention challenges. Other challenges relate mainly to the provision of services in remote and isolated First Nations communities, such as unreliable Internet connectivity, transportation issues, high shipping costs, access to other services, and hiring and retaining other types of health care workers.

Economy and Efficiency

There are examples of project implementation that demonstrate ways to improve CCC service efficiency that could be incorporated in future models for primary care delivery within remote and isolated First Nations communities. Improved overall performance measurement data would better support reporting requirements and the conduct of future evaluations of primary care service delivery.

The evaluation revealed that the allocation of financial resources to CCC services is primarily based on historic allocations and outdated population estimates. Regions require annual supplements to A-Base funding to meet deficits.

A significant factor challenging the efficient use of resources is the ongoing staffing challenges that resulted in high vacancies and staff turnover, and required the use of costly contract nursing agencies. Unexpected overtime costs, such as for providing non-urgent care after hours, also created inefficiencies.

More collaborative arrangements with partners and integration of services was commonly identified as a way to enhance program delivery efficiency.

6.0 Recommendations

Based on the findings and conclusions of the evaluation report, a number of recommendations have been developed. These recommendations will facilitate the further enhancement of the CCC program.

Recommendation 1: Strengthen efforts concerning nursing recruitment and retention

Staff recruitment and retention is a recognized major and ongoing challenge for providing CCC services, particularly for nursing. Health Canada National Headquarters should work with regions, remote and isolated First Nations communities, and nursing stations to strengthen ongoing efforts to not only attract but retain nurses to work in remote and isolated First Nations communities. This could include, for example, providing new CCC nurses with additional training, as required, to integrate into their positions as CCC service providers, and developing strategies to improve the work environment.

Recommendation 2: Work with regions, First Nations communities, and provincial health departments to strengthen collaboration and improve information sharing and partnerships.

Delivery of CCC services would benefit from continued development of partnerships and collaboration between Health Canada, remote and isolated First Nations communities, regional health authorities, provincial health departments, and other departments and levels of government to further integrate and streamline services and ensure comprehensive services and continuity of care to remote and isolated First Nations community members. Partnerships and collaboration would facilitate streamline the referral process and ensure coordinated care in rural and remote communities. Furthermore, Health Canada Headquarters and regions should, in particular, work with provinces to facilitate data and information sharing between provincial health services and nursing stations. The objective of this information sharing is to ensure that members of remote and isolated First Nations communities have access to the same level of continuity of care as do other Canadians.

Recommendation 3: Work with regions, First Nations communities and nursing stations to develop and implement an improved performance measurement strategy that will assist regions and National Headquarters in measuring the achievement of expected outcomes and planning for resource utilization and service requirements. This should include electronic record keeping on clients and services provided and on all personnel at the nursing stations.

A strengthened performance measurement system will assist regions and National Headquarters in measuring the achievement of expected outcomes and planning for resource utilization and service requirements. The process of developing a performance measurement strategy should include review of the logic model, expected outputs, expected outcomes, and indicators. Indicators should be measurable and clearly linked to the expected outcome. Responsibilities for data collection, maintenance and timing should also be clearly articulated. As well, the performance measurement strategy should include targets and a method for establishing baseline measures for each expected outcome. Performance measurement could also include a means of obtaining input from communities on satisfaction with CCC services received.

Implementation of a uniform electronic record keeping system at all nursing stations would facilitate collection of data that will contribute towards patient case management, as well as planning and performance measurement at the community, regional, and national level. Such a system would involve collection of relevant data that would be convenient for nursing station staff to collect and record, and for Regional Offices and National Headquarters to access, compile, and analyze for reporting purposes. Health Canada National Headquarters should work with regions and First Nations communities (including nursing staff) to gain input from the various stakeholders on all aspects of planning, developing, implementing, and maintaining and monitoring such a system. This potentially could include consultation with provinces to assess the benefits and feasibility of integration with provincial health systems. The planning and development phase could also include the feasibility of accessing and using existing software/databases.

An electronic record keeping system at nursing stations could potentially include nursing administrative records to provide Health Canada with up-to-date human resource information at nursing stations, including numbers of nurses and other health care workers at each station, vacancies, positions filled with contract agency nurses, education levels, and participation in educational activities. Development and implementation of such a system, which should include collaboration with Health Canada regional and national human resources representatives, will further facilitate human resource management and planning at the regional and national level.

Appendix 1: Details on Data Collection and Analysis Methods

Data Collection Methods

Document and data review. The document and data review involved a review of key documents and data provided primarily by Health Canada. Documents included authority providing legislation (Acts and Regulations); Health Canada's PAA and strategic outcomes related to CCC; policy documents; Speeches from the Throne; Health Canada Departmental Performance Reports and Reports on Plans and Priorities (RPP); federal budgets; Performance Measurement Strategy documents; accountability documents, such as performance reports, progress reports, annual reports, and internal audits; reporting on special studies/projects; and other relevant documents. Data included Community-Based Reporting Template data, samples of nursing station activity logs, and CCC financial expenditures.

Documents and data were reviewed to determine their usefulness in addressing the evaluation questions, and documents and data were sorted according to the questions they pertained to. With a few exceptions, the document and data review addressed the majority of the evaluation questions. Exclusion/inclusion criteria used for assessing documents and data were as follows:

Inclusion criteria

• Document/data addresses at least one evaluation question designated as using a document/data review as a data source.
• Document/data meets the above and reports on CCC services within the evaluation timeframe of April 1, 2005, to March 31, 2012. However, if any earlier documents were relevant assessments of services, they would be included to review any changes made and improvements implemented as a result of the assessments.
• Rolled-up data and documents reporting on defined performance measures. Data that is complete, aggregated, and could be provided electronically and in a timely manner, given the time restrictions of the evaluation.

Exclusion criteria

• Document/data reporting on activities prior to April 1, 2005, with the exception of those described under the inclusion criteria.
• Partial data referring to only certain aspects of a performance measure (e.g., reporting on one nursing station in one region)
• Large volumes of raw data that were not rolled up (i.e., require data entering, organizing, processing, and analyzing)
• Large volumes of recipient-written annual reports that reported on all aspects of the recipients' activities, including those outside the scope of CCC. This would have required reviewing each report for each year to find embedded relevant information.

Based on identified information gaps, additional documents and datasets from FNIHB were requested and received over several weeks (e.g., Recognizing Strengths -- Building for the Future Primary Health Care Service Delivery Models in Remote and Isolated First Nations Communities: Literature Review; Proposals and briefing notes of various nursing data projects). The total number of documents included in the final review was approximately 210, with 15 of these being data documents.

Literature Review
Using identified search terms, an initial scan of the literature was conducted in order to determine how the literature could contribute to addressing the evaluation questions. The literature review primarily addressed questions on economy and efficiency. Relevant terms were added to the list as the review progressed. Once the initial scan was completed, the scope of the review was finalized in consultation with the Evaluation Directorate. The literature review relied primarily on peer reviewed journals (scientific and academic) and relevant grey literature (newspapers and websites). Abstracts of the key articles and reports were reviewed to determine their relevance, and their bibliographies used to identify additional material for review. Of the 40 pertinent documents reviewed in greater detail, 22 documents were retained for the literature review. Documents, datasets, or pieces of literature were entered into a Zotero library for further review and analysis.  Zotera is a program that allows users to manage documents and other sources with high efficiency. The program automatically generates in-text citations and reference lists, see http://www.zotero.org/.

Key Informant Interviews
This task involved conducting telephone interviews with key stakeholders drawn from Health Canada Regional personnel for the regions delivering CCC services to remote and isolated First Nations communities, as well as the national headquarters office. Health Canada provided the list of 65 interviewees with involvement in and knowledge on the CCC services. Participants were chosen in a manner to include a representative selection of stakeholders from each region and to include regional and national representatives of management, human resources, nurse educators, Health Funding Arrangements, and finance. Regional stakeholders included Regional Directors, Regional Directors of Nursing, Regional Nursing Officers, Regional Nurse Managers, Zone Directors, Regional Nurse Educators/Practice Consultants, Human Resource Officers, Regional Financial Officers, and Health Funding Arrangement personnel. National headquarters stakeholders included representatives in the areas of finance, human resources, and nursing services.

Four sets of interview guides were drafted according to the areas of expertise of the participant groups and to address the relevant evaluation questions each group was expected to address. Guides were developed in consultation with the Evaluation Directorate and the CCC Services Evaluation Working Group (EWG). The first few interviews served as preliminary interviews for testing the interview guides; given the length of these interviews, several questions were removed and other questions shortened/streamlined in order to maintain the expected the 1.0 to 1.5 hour timeline for each interview.^{Footnote 28}

Regional Directors received a letter from Health Canada Director General's Office to inform them of the evaluation and interview process and to request they inform their staff that they may be asked to participate in an interview. Key informants were then contacted to schedule the telephone interview at a time convenient to the participant. Participants received the interview guides in advance to allow to them prepare a considered response. Interviews were digitally recorded with the participant's permission; plus, interviewees received copies of their interview notes for their records and to give them the opportunity to make revisions or additions to their notes.

Seven individuals, mainly representatives of finance or Health Funding Arrangements, indicated they had no to little involvement with CCC services and declined to participate in the interviews. A few others were not available for various reasons, such as workload, retirement, changing positions, or sick leave.

In total, 38 interviews were conducted, involving 47 participants; however, one participant indicated at the time of the scheduled interview that they had been in their position for only a short period of time and could not answer any of the questions. The remaining 37 interviews with 46 participants were from the following stakeholder groups:

Analysis: completed interview notes were imported into NVivo qualitative data analysis software for analysis. Nvivo allows the user to code content along specific categories. For the purposes of the key informant interviews, this involved coding and then organizing stakeholder responses along key evaluation issues and questions. This facilitates more effective and efficient analysis of responses for similarities and divergences of opinion.
Key informant interviews are based on open-ended questions where respondents are not generally asked yes or no questions but, rather, are asked to provide a considered response to a set of questions based on their knowledge and observations. The primary purpose of qualitative key informant interviews is to gain insight into a process or problem, and, as such, they are not conducive to counting up responses. However, to provide some context to responses, the following descriptive scale was used to report on some aspects of key informant interviews, with "a few" being approximately 10-15% or less of respondents, "some" being more than 15% to approximately 40%, "many" being more than 40% to approximately 60%, "most" being more than 60% to approximately 80%, and "almost all" being over 80%.

Stakeholder Surveys
The original intent for the survey was to use either an online format or mixed online and telephone survey. However, discussion with Health Canada regional representatives during an introductory meeting for the evaluation process revealed Internet connectivity issues for many of the remote and isolated First Nations communities. Therefore, in consultation with the Evaluation Directorate and the Evaluation Working Group (EWG), it was decided to use a telephone-based survey in order to elicit the greatest number of responses.
Three target survey groups were identified:

• Internal CCC service providers -- Registered nurses (RN), Nurse Practitioners (NP), Licensed practical nurses (LPN), Registered practical nurses (RPN), pharmacy technicians, laboratory/x-ray technicians, and public health physicians.
• External beneficiaries -- CCC Band Council chiefs/health portfolio directors as proxies for beneficiaries.
• External CCC service providers -- family practice physicians, physician specialists, air ambulance, provincial/federal hospitals, and rehabilitation settings.

However, as a result of survey pretesting (described below), and in consultation with the Evaluation Directorate, several changes were made for the survey groups representing internal and external CCC service providers.

In consultation with the Evaluation Directorate and the Evaluation Working Group, survey questionnaires were developed for each of the three stakeholder groups. Questionnaire development was also informed by information acquired through the key informant interviews, as well as by the document, literature, and database review.

Health Canada was not able to provide a complete sample of all internal CCC service providers at each nursing station, due to incomplete and/or inaccessible information on CCC nurses employed by nursing agencies or by transferred bands managing their nursing stations. Health Canada also did not have a complete list of all external CCC service providers and had no information on the external stakeholders providing services in transferred communities. Therefore, the survey sample was developed in the following way for each stakeholder group:

• Internal CCC service providers: The evaluation framework identified surveying all internal service providers at nursing stations. As noted above, Health Canada did not have a sample list to provide, and it was necessary to develop the sample list through the survey process. Health Canada provided the phone numbers of the 86 nursing stations in remote and isolated First Nations communities to call and ask to speak with the nurse manager (or nurse-in-charge). Initially, due to time restrictions for the survey process, the target was to contact every second nursing station on the list provided. However, this sample was quickly exhausted due to unavailability of the nurse manager at the time of the call, and the entire list of 86 nursing stations was used.

The remainder of the internal service provider sample was developed by asking the nurse managers contacted for names of their staff as part of their survey questionnaire. As a result of pretesting, as described below, the target sample was changed from all internal CCC service providers to only CCC nurses. Nurse managers from the 57 nursing stations contacted provided names for 144 nurses.

• External beneficiaries: Health Canada also provided names and phone numbers of 83 Bands Chiefs of remote and isolated First Nations communities with nursing stations so that the Chief or someone who was authorized to speak on behalf of the Chief could be contacted to ask them to participate in the survey. Hereafter, these respondents are referred to as band representatives.
• External CCC service providers: The evaluation framework identified surveying all external CCC service providers. However, the introductory discussions with Health Canada regional representatives revealed that regional FNIHB offices did not maintain lists of external service providers, or have access to names of external service providers used by nursing stations in transferred remote and isolated First Nations communities. Therefore, to expedite the process given the time restrictions for the evaluation, the EWG decided to request Regional Directors to identify at least four external providers for their region. However, in Saskatchewan and Quebec, which primarily have First Nations transferred communities, the nurse managers were asked to supply one external service provider contact name during their survey process. The Saskatchewan FNIHB regional office was also able to provide several contact names. With respect to the latter, after discussion with the EWG, it was determined that air ambulance stakeholders primarily provide transportation services and would not have sufficient knowledge of the CCC services in the communities to respond to the survey questions. Therefore, the survey of external service providers did not include air ambulance providers. Regional representatives provided 21 useable names of external service providers to contact for the external service providers survey.^{Footnote 29} Only two names were obtained through the nurse managers in Saskatchewan and Quebec, one of which had already been received from the region; in most cases, nurse managers either did not know the names and/or contact information of service providers or could only give the name of an organization and no specific individual to contact.

Prior to the survey, potential survey participants received a communication from their regional Health Canada office either by email or fax informing them of the evaluation and the possibility of being contacted for a short survey. The survey was administered by telephone in both French and English. Nurse survey participants were contacted between October 26 and November 15; band representatives between October 29 and November 16; and external service providers between November 5 and November 22. Surveys of nurse managers took anywhere from 20 to 40 minutes or more due to the need to gather staff information; surveys of other nurses required 10 to 20 minutes.

The surveys were conducted by telephone using CC3 (Interviewer), the industry standard computer-aided telephone interviewing (CATI) system by VOXCO.^{Footnote 30} The first few surveys completed served as a pretest of the survey questionnaire to check for clarity of questions and time requirements. As a result of the pretest, several adjustments were made to the nurse manager survey questionnaire, as well as the internal service provider target population. The original intent was to ask nurse managers to identify and gives names for all health care workers providing CCC services, including both nurses and other kinds of providers (e.g., pharmacy technicians, laboratory technicians, and health care aide). However, this took a considerable amount of nurse managers' time. Therefore, in consultation with the Evaluation Directorate, the decision was made to ask only for names of nurses that provide urgent and non-urgent primary care. This change supported the survey by optimizing the time requested of nurse managers. Plus, the fact that only CCC nurses were included ensured that the questions were directed to those health care service providers holding the most applicable information concerning CCC services. As well, as a result of pretesting, several questions that had been more open-ended were changed to closed-ended. This change further streamlined the process and reduced the time required to complete the survey. Once the surveys were finalized, they were translated into French.

Survey format and analysis: Nurses were asked to respond to the questions for the nursing station they were currently working at (i.e., where they were contacted for the survey). Band representatives were asked to respond to questions based on the nursing station or stations in their community. External service providers were asked to respond based on nursing stations where they provided CCC services.Most survey questions were constructed as closed-ended questions with a choice of responses provided that could then be quantified for findings presentations. Some questions asked respondents to rate their answers on a scale of 1 to 5. All stakeholders were also asked several open-ended questions to give them the opportunity to verbally express their varying perspectives. Verbatim responses were first reviewed to identify and develop themes/categories emerging from the data, and then coded according to these themes/categories. This facilitated quantification of responses. Finally, the survey data was analysed through SPSS (statistical analysis software) using frequency tables and descriptive statistics.

The call records for the nurse and band representative surveys are presented in Table 3. Fifty-seven nurse managers from the 86 nursing stations were contacted and surveyed, providing a response rate of 66% for the nurse manager part of the survey. In total, 93 of the 144 nurses identified by nurse managers were surveyed, for a response rate of 65% (response rate calculated on 143 eligible numbers, one number, when called, was not in service). Health Canada provided names of 83 Band Chiefs, and 26 of these band representatives were surveyed for a response rate of 31%.^{Footnote 31} These are good response rates, given the time available for conducting the surveys (approximately three weeks each), and also considering the necessity to develop the survey sample for nurses. All stakeholder groups required numerous call-backs in order to achieve the attained response rates.

The table below shows regional representation in the survey, providing response per region for nursing stations. Nurse-manager responses were high for most regions (59% to 100%), with the exception of Quebec, which had a response rate of 18% of nurse managers. The table also shows the number of nurses identified by nurse managers per region, and the nurses surveyed. Again, nurse response was high (57% to 80%) for all regions but Quebec. Considering the 57 nurse managers and 93 other nurses, a total of 150 nurses were surveyed.

The table below provides response rate per region for band representatives, based on the number of remote and isolated First Nations communities Band Chiefs whose contact information was provided by Health Canada, per region. Responses ranged from no respondents in Alberta to 45% of the band representatives in Quebec.

Five of the 22 total external service provider contacts were surveyed for a 23% response rate. Given the low number of responses, responses from these respondents were incorporated into the survey technical report findings descriptively; no survey tables were compiled for external service providers. Profiles of these five respondents include the following:

• Four said they provided services to both Health Canada- and band-operated nursing stations, and one provided services only to Health Canada-operated nursing stations.
• Three were physicians, with two indicating they provide general practitioner services and one providing specialist physician services to nursing stations. The two others were representatives of organizations providing physician and other services, with both indicating their organization provided general and specialist physician services.
• All five respondents indicated they provide a range of services to nursing stations.
  • All five respondents provide urgent and non-urgent care by phone.
  • Four of the five visit the nursing station to see patients.
  • One of the five provides both urgent and non-urgent care by Internet.
  • Four of the five provide hospital services.
  • Three of the five provide rehabilitative services.
  • Two of the five provide air ambulance services.

Appendix 2: References

• AANDC. (2011). Sustaining Momentum: The Government of Canada's Third Report in Response to the Kelowna Accord Implementation Act (2010-2011). Retrieved January 18, 2013, from http://www.aadnc-aandc.gc.ca/eng/1315228813855/1315229902908
• Baker, R., & Denis, J.-L. (2011). A comparative study of three transformative healthcare systems: Lessons for Canada. Canadian Health Services Research Foundation.
• Birks, M., Mills, J., Francis, K., Coyle, M., Davis, J., & Jones, J. (2010). Models of health service delivery in remote or isolated areas of Queensland: a multiple case study, 28(1), 25-34.
• British Columbia Ministry of Health. (2012). Resource Manual for Registered Nurses (Certified). Retrieved from http://www.health.gov.bc.ca/msp/infoprac/registered-nurses/registered-nurses-certified-manual.pdf
• British Columbia tripartite framework agreement on First Nations health governance. (2011).
• Canadian Federation of Nurses Unions. (2012). The Nursing Workforce: Canadian Federation of Nurses Unions Backgrounder. Retrieved from http://www.nursesunions.ca/sites/default/files/
  2012.backgrounder.nursing_workforce.e_0.pdf
• Canadian Institute for Health Information. (2013). Regulated Nurses: Canadian Trends, 2007 to 2011. Retrieved from https://secure.cihi.ca/estore/productSeries.htm?pc=PCC449
• Finance Canada. (2009). Budget 2009.
• Finance Canada. (2012). Jobs growth and long-term prosperity: Economic action plan 2012. Retrieved from http://www.budget.canada.ca/2012/plan/pdf/Plan2012-eng.pdf
• First Nations Information Governance Centre. (2005). First Nations Regional Longitudinal Health Survey (RHS) 2002/03. Results for Adults, Youth and Children Living in First Nations Communities. Retrieved from http://www.rhs-ers.ca/sites/default/files/ENpdf/RHS_2002/rhs2002-03-technical_report.pdf
• FNIHB. (2011). Program compendium 2011/2012.
• FNIHB. (2012a). FNI CCC&Program Oversight FINAL Activities.
• FNIHB. (2012b). Community-based primary care and nursing services performance report 2006/07 to 2009/10.
• FNIHB. (2012c). Recognizing strengths - building for the future: Primary Health Care Service Delivery models in remote and isolated First Nations communities - primary and secondary research.
• FNIHB. (2012d). Environmental scan of regional education activities & related best practices as part of the AAB primary care nursing services audit.
• FNIHB. (2012e). CCC Manitoba region: Summary of 21 communities.
• FNIHB. (2012f). Number of vacant FNIHB primary care nurse positions (FTEs) by region March 2011 to March 2012 quarterly reporting.
• FNIHB. (2012g). Recognizing strengths - building for the future: Primary Health Care Service Delivery models in remote and isolated First Nations communities - final report and recommendations.
• FNIHB. (2012h). Nursing program integrity allocations 2009-2012.
• FNIHB. (2012i). FNI CCC&Program Oversight FINAL Regions.
• FNIHB, & EPOHD. (2011). Critical incident stress management: Annual summary report, April 2010 to March 2011.
• FNIHB, unpublished report. (2010). Community-Based Reporting Template aggregated report 2009-10 (Part 6).
• FNIHB, unpublished report. (2011). Community-Based Reporting Template aggregated report 2010-11 (Part 6).
• Gao, S., Manns, B., Culleton, B., Tonelli, M., Quan, H., Crowshoe, L., ... Hemmelgarn, B. (2008). Access to health care among status Aboriginal people with chronic kidney disease. CMAJ, 179(10), 1007-1012.
• Government of Canada. (1867). The Constitution Act. Retrieved from http://laws-lois.justice.gc.ca/eng/Const/PRINT_E.pdf
• Government of Canada. (1876). Indian Act. Retrieved from http://laws-lois.justice.gc.ca/PDF/I-5.pdf
• Government of Canada. (1984). Canada Health Act. Retrieved from http://laws-lois.justice.gc.ca/PDF/C-6.pdf
• Government of Canada. (2011). Speech from the throne: 2011. Retrieved July 4, 2012, from http://www.parl.gc.ca/Parlinfo/Documents/ThroneSpeech/41-1-e.html
• Harris, S., Naqshbandi, M., Bhattacharyya, O., Hanley, A., Esler, J., & Zinman, B. (2011). Major gaps in diabetes clinical care among Canada's First Nations: Results of the Circle study. Diabetes Research and Clinical Practice, 92, 272-279.
• Health Canada. (2005). Mandate and priorities. Retrieved June 21, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/fnihb-dgspni/mandat-eng.php
• Health Canada. (2010a). Audit of primary care nursing services: Final audit report.
• Health Canada. (2010b). Clinical Practice Guidelines for Nurses in Primary Care - First Nations and Inuit Health - Health Canada. Retrieved January 21, 2013, from http://www.hc-sc.gc.ca/fniah-spnia/services/nurs-infirm/clini/index-eng.php
• Health Canada. (2011b). Program activity architecture and the performance measurement framework.
• Health Canada. (2011c). Utilization of clinical and client care services in nursing stations in Alberta.
• Health Canada. (2011d). Health Canada's northern nursing recruitment and retention strategy.
• Health Canada. (2012a). Request for proposal - Evaluation of FNIHB Clinical and Client Care (CCC) services.
• Health Canada. (2012b). First Nations and Inuit Health Fact Sheet. Health Information, Analysis and Research Division, First Nations and Inuit Health Branch.
• Health Canada. (2012c). 2012-13 report on plans and priorities. Retrieved from http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/shc/shc-eng.pdf
• Health Canada. (2012d). Aboriginal Health Human Resources Initiative. Retrieved January 18, 2013, from http://www.hc-sc.gc.ca/ahc-asc/activit/strateg/fnih-spni-eng.php
• Health Canada. (2012e). First Nations and Inuit Health Strategic Plan: A shared path to improved health.
• Health Canada. (2012f). Health Care System - Main Page - Health Canada. Retrieved December 19, 2012, from http://www.hc-sc.gc.ca/hcs-sss/index-eng.php
• Health Canada. (2012g). X-ray services in remote and isolated First Nations communities: Preliminary report.
• Kulig, J. (2005). The nature of rural & remote nursing fact sheet 2: What educational preparation do nurses need for practice in rural and remote Canada? Retrieved from http://www.carrn.com/files/factsheet2.pdf
• LTSC. (2010). Funding adequacy for the NITHA partners.
• Misener, R. M., MacLeod, M. L. P., Banks, K., Morton, A. M., Vogt, C., & Bentham, D. (2008). "There's rural, and then there's rural": Advice from nurses providing primary healthcare in northern remote communities. Nursing Leadership, 21(3), 54-63.
• NCCAH. (2011). Looking for Aboriginal health in legislation and policies, 1970 to 2008: The policy synthesis project. Retrieved from http://www.nccah-ccnsa.ca/docs/Looking%20for%20Aboriginal%20Health%20in%20Legislation%20
  and%20Policies%20-%20June%202011.pdf
• Rygh, E., & Hjortdahl, P. (2007). Continuous and integrated health care services in rural areas. A literature study. Rural and Remote Health, 7, 766.
• Saskatchewan Registered Nurses' Association. (2012, Winter). Optimizing the Role of the Registered Nurse. News Bulletin, Volume 14, Number 1, Winter 2012, p. 18.
• Shan, B., Gunraj, N., & Hux, J. (2003). Markers of Access to and Quality of Primary Care for Aboriginal People in Ontario, Canada. Research and Practice, 93(5).
• Shields, M., & Wilkins, K. (2009). Factors related to on-the-job abuse of nurses by patients. Component of Statistics Canada Catalogue no. 82-003-X.
• Statistics Canada. (n.d.). Aboriginal peoples of Canada: A demographic profile. Retrieved January 22, 2013, from http://www12.statcan.ca/english/census01/products/analytic/companion/abor/canada.cfm
• Statistics Canada. (2007). Aboriginal health and well-being. Statistics Canada.
• Stewart, N., & MacLeod, M. L. P. (2005). The nature of rural & remote nursing fact sheet 3: RNs in nurse practitioner positions in rural and remote Canada. Retrieved from http://www.carrn.com/files/factsheet3.pdf
• The First Nations Information Governance Centre. (2012). First Nations Regional Health Survey (RHS), Phase 2 (2008/10) National Report on Adults, Youth and Children Living in First Nations Communities. (Ottawa: The First Nations Information Governance Centre, June 2012).
• Treasury Board of Canada Secretariat. (2009). Directive on the Evaluation Function. Retrieved December 3, 2012, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681&section=text#appA.
• Wakerman, J., Humphreys, J., Wells, R., Kuipers, P., Entwistle, P., & Jones, J. (2008). Primary health care delivery models in rural and remote Australia - a systematic review. BioMed Central.
• Wakerman, J., Humphreys, J., Wells, R., Kuipers, P., Jones, J., Entwistle, P., & Kinsman, L. (2009). Features of effective primary health care models in rural and remote Australia: a case-study analysis. Models of Care, 191, 88-91.