Aboriginal Diabetes Initiative (ADI) Evaluation Framework

February 2002
Cat. H35-4/20-2002
ISBN: 0-662-66385-3

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Table of Contents

• 1.0 Purpose and Background Information
  • 1.1 Purpose of the Aboriginal Diabetes Initiative Evaluation
  • 1.2 Background and Description of the Aboriginal Diabetes Initiative
  • 1.3 Program Objectives
  • 1.4 Program Component Outcomes
  • 1.5 ADI Funding Levels
  • 1.6 Target Groups
  • 1.7 Program Implementation and Delivery
  • 1.8 Program Logic Model
• 2.0 Evaluation Strategy
  • 2.1 Evaluation Challenges
  • 2.2 Evaluation Tiers
  • 2.3 Evaluation Questions and Indicators
  • 2.4 Component-Level Evaluation Questions and Indicators
  • 2.5 Methodology and Data Sources
• 3.0 The Evaluation
  • 3.1 Evaluation Funding
  • 3.2 Evaluation Responsibility
  • 3.3 Changes to the ADI Evaluation Framework

This Evaluation Framework was drafted by the Accountability and Evaluation Working Group of the Aboriginal Diabetes Initiative. The group consisted of:

• Dr. Gillian Bailey, First Nations and Inuit Health Branch (FNIHB), Health Canada
• Michelle Callaway, FNIHB, Health Canada
• Kidane Gebremariam, FNIHB, Health Canada
• Barbara LeBrun, Population and Public Health Branch, Health Canada
• Joanne Lucarz-Simpson, FNIHB, Health Canada
• Alastair MacPhee, Congress of Aboriginal Peoples
• Onalee Randell, Inuit Tapiriiksat Kanatami (ITK)
• Satish Seetharam, FNIHB, Health Canada
• Anita Stevens, Assembly of First Nations
• Susan Taylor-Clapp, FNIHB, Health Canada
• Maureen Thompson, FNIHB, Health Canada
• Krista Van Aert, FNIHB, Health Canada
• Francine Vincent, First Nations of Quebec and Labrador Health and Social Services Commission
• Joanette Wallace, FNIHB, Health Canada
• Joan Wentworth, Battleford Tribal Council

For all enquires regarding this document, please contact:

Satish Seetharam
Program Evaluation Manager
Community Health Programs Directorate
Jeanne Mance Bldg., Room 2007
Tunney's Pasture Ottawa,
Ontario,
K1A 0L3
Phone (613) 952-4230
Fax (613) 954-8107
Email: satish_seetharam@hc-sc.gc.ca

1.0 Purpose and Background Information

Yearly, it is estimated that diabetes costs Canadians $9 billion, is directly responsible for 5,500 deaths and contributes to 25,000 potential person years of life lost. Incidence increases by 60,000 new cases every year.

1.1 Purpose of the Aboriginal Diabetes Initiative Evaluation

Overview of the Canadian Diabetes Strategy

Diabetes is a chronic disease where the body fails to produce insulin(type 1) or cannot properly use insulin (type 2), a hormone essential for normal body metabolism. Approximately 90 percent of all diabetes cases among Canadians are type 2, which usually occurs after age 40. Seniors, Aboriginal people, baby boomers and other high-risk groups (those at greatest risk of weight gain, those with a sedentary lifestyle) are therefore key targets for the Canadian Diabetes Strategy (CDS). State of the art knowledge indicates that programs which address key modifiable risk factors of obesity and inactivity should help prevent/delay the onset of type 2 and its complications.

Yearly, it is estimated that diabetes costs Canadians $9 billion, is directly responsible for 5,500 deaths and contributes to 25,000 potential person years of life lost. Incidence increases by 60,000 new cases every year. It is estimated that over two million Canadians have diabetes, and an estimated one-third of the cases are undiagnosed.

Diabetes is three to five times more prevalent in Aboriginal populations than in the general population. Among the Inuit, the rates of diabetes are lower than in the general population. Type 2 diabetes accounts for almost all cases of diabetes amongst Aboriginal people, and generally occurs at a much younger age: it has appeared in Aboriginal children as young as five. Long duration of high blood sugar levels is correlated with complications, hence there is great concern that this early onset of diabetes will lead to early onset of complications.

The 1999 federal budget committed the federal government to the development of the Canadian Diabetes Strategy - a strategy of $115 million over five years, which includes a major Aboriginal component. (The CDS was formerly known as the Canadian Diabetes Prevention and Control Strategy, initially set at $55 million over three years). This commitment evolved from the 1997 Speech from the Throne that committed to addressing the rapid increase in diabetes in Aboriginal communities, the 1998 announcement of Gathering Strength - Canada's Aboriginal Action Plan, and increasing pressure for a Canada-wide diabetes strategy from national groups.

The Government of Canada is engaging all stakeholders in activities to address diabetes and its complications. In consultations with provincial and territorial governments, Aboriginal organizations, and national diabetes and nonprofit groups, consensus was reached on four strategic components and priority areas for action: Prevention and Promotion; Aboriginal Diabetes Initiative; National Diabetes Surveillance System; and National Coordination. These four elements recognize that:

• type 2 diabetes is thought to be mainly preventable;
• Aboriginal populations are particularly at risk;
• a greater knowledge base is required to promote informed decision-making; and
• pooling efforts, reaching consensus, complementing and building on existing resources will engender the greatest benefits.

Liaison will continue throughout the duration of the Strategy. Advisory Committees will actively engage all stakeholders, where appropriate, to maximize the investment of the federal government in diabetes and also to look toward the future.

The overall objective of the CDS is to set the stage for significantly reducing the incidence and prevalence of diabetes and diabetes- related complications in Canada. The CDS will address the needs of Canadians by focusing on assisting those populations which, according to state-of-the-art knowledge, are most at risk for developing diabetes and its complications: Aboriginal people, seniors, people who are overweight, people with sedentary lifestyles and Canadians already living with the disease.

The CDS is premised on the recognition that all levels of govern- ment, as well as non-governmental organizations (NGOs), Aboriginal organizations, communities and individuals have important roles to play, and contributions to make in addressing diabetes in Canada. The CDS therefore seeks to develop and facilitate partnerships with all key stakeholders. The CDS design involves branches of Health Canada working together to set the stage for achievement of positive health outcomes that will contribute to the overall mandate and mission of Health Canada: To maintain and improve the health of Canadians. Consultations and ongoing dialogue with the provinces and the territories have singled out health promotion and type 2 diabetes prevention as the actions likely to reap the greatest benefits in addressing diabetes, especially for those at risk of developing type 2 diabetes.

Evaluation of the Aboriginal Diabetes Initiative

The Aboriginal Diabetes Initiative (ADI) requires an ongoing monitoring and evaluation strategy to assist with the management of activities and provide ongoing information on the progress toward national goals. It will also provide information on intermediate impacts and outcomes resulting from implementation of the program.

The Evaluation Framework is designed and developed for the formal evaluation of the Aboriginal Diabetes Initiative. This framework is designed to identify the specific components of the ADI as approved by Treasury Board. For each component, the long-term objectives and key indicators are presented. A Logic Model provides the major elements of the evaluation framework including what is to be evaluated and the expected activities and outputs.

The evaluation methodology described in this report identifies existing information sources, new (proposed) information collection mechanisms (consultations/key informant interviews, surveys, special studies, etc.) and the independent evaluations of the three ADI components, in order to evaluate the effectiveness of activities and projects in contributing to the common goals.

The Evaluation Framework states what is to be evaluated (the objectives and activities/outputs of the three program components), what sources of information will be used, and what impacts and effects the activities demonstrate. The evaluation framework begins with the Aboriginal Diabetes Initiative, followed by each of the three components of the ADI:

1. care and treatment;
2. prevention and promotion; and,
3. lifestyle support.

This framework report is developed to complement and expand upon those developed for the overall Canadian Diabetes Strategy and First Nations and Inuit Health Branch (FNIHB) evaluation frameworks.

1.2 Background and Description of the Aboriginal Diabetes Initiative (ADI)

General Background Information

The ADI is a response to a critical need for action. It strives to create a holistic environment for good health, lifestyle changes, community linkages and partnerships. There is a strong need for collaboration and partnership on many levels, including care and treatment providers, community members, leaders, FNIHB regions, NGOs and others, in order to develop adequate and effective Care and Treatment, Prevention and Promotion, and Lifestyle Support programs and services, which are culturally appropriate and also community-based and delivered.

Under the umbrella of the CDS, and in keeping with Health Canada's unique responsibilities, the goal of the ADI is to begin to increase awareness of type 2 diabetes and reduce the incidence of its complications in Aboriginal people by implementing culturally sensitive, holistic, and accessible programs. The ADI will do this by providing access to direct care and treatment and lifestyle support programs and services for First Nations living on reserves and Inuit living in Inuit communities. It will also provide an overall culturally appropriate approach to prevention and health promotion programs for all Aboriginal peoples, including Métis, urban Inuit and off-reserve populations.

The ADI is one of four components of the CDS, the Government of Canada's response to Diabetes, which involves many stakeholders and partners. It also has a strong link with the First Nations and Inuit Home and Community Care (FNIHCC) Program. The ADI is a 5 year program, at the end of which Health Canada must report to Treasury Board. An on-going and outcome based evaluation is necessary for the ADI, in accordance with the Treasury Board submission.

Program Description

The ADI will begin to address the epidemic of diabetes among Aboriginal people by focusing its efforts in three main areas:

• Care and Treatment;
• Prevention and Promotion; and,  
• Lifestyle Support.

The ADI is comprised of two programs:

1. First Nations On-Reserve and Inuit in Inuit Communities Program (FNOIIC); and,
   
   
2. Métis, Off-Reserve Aboriginal and Urban Inuit Prevention and Promotion Program (MOAUIPP).

Details of these programs are presented in the two logic models on pages 14 and 15.

The ADI for First Nations On-Reserve and Inuit in Inuit Communities makes up 75% of the program, and it focuses on all three major program components (care and treatment, prevention and promotion, and lifestyle support). These three major components will be delivered:

• through the FNIHB regional processes;
• through the funding formula;
• according to a general set of guidelines; and,
• in partnership with communities.

The Métis, Off-Reserve Aboriginal and Urban Inuit Prevention and Promotion Program (MOAUIPP) focuses on primary prevention and health promotion. It will:

• be delivered through a headquarters request for proposal process;
• be based on a funding formula;
• follow program guidelines; and
• encourage partnerships, wherever possible.

Program Components

1. Care and Treatment - In keeping with Health Canada's unique responsibility to ensure access to direct care and treatment services to First Nations on-reserve and Inuit living in Inuit communities, the care and treatment element will be specific to these groups. (Provinces and territories will continue to provide care and treatment services to Métis, off-reserve Aboriginal and urban Inuit, as is their jurisdiction.)
   
   Care and Treatment will comprise one part of the ADI, and will be delivered through contribution agreements, administered through FNIHB regions. Programs (such as screening programs, foot and eye clinics, regular examinations, tests and treatment programs) will serve to meet the care and treatment needs of First Nations and Inuit already diagnosed with diabetes, and to prevent further complications from developing. Programs will be developed and delivered in a culturally sensitive and holistic manner by the communities themselves. Activities will be community-based, to ensure that First Nations and Inuit are able to build the necessary capacity to manage sustainable and culturally appropriate diabetes programs.
   
   
2. Prevention and Promotion - Prevention and promotion programs will target all Aboriginal people: Métis, on-and-off-reserve First Nations and Inuit. Programs will be culturally appropriate, and will be administered through contribution agreements and other arrangements. Consultations have indicated that primary prevention is a major concern of the Inuit who do not want to see the extremely high rates of diabetes experienced by First Nations replicated in their communities. First Nations have also indicated the importance of ensuring that youth and those most at risk are educated on diabetes risk factors.
   
   National organizations such as the Assembly of First Nations (AFN), the Inuit Tapiriiksat Kanatami (ITK), the Native Women's Association of Canada (NWAC), the Métis National Council (MNC), the Congress of Aboriginal Peoples (CAP), and the National Aboriginal Diabetes Association (NADA) are participating in implementation planning and providing input into program delivery to their constituencies. Prevention and promotion activities will include best practice sites, train the trainer programs, awareness programs, school-based programs (on-reserve and in Inuit communities) and an Aboriginal resource centre/clearinghouse.
   
   
3. Lifestyle Support - Lifestyle support services will be provided to First Nations and Inuit communities to help those living with diabetes and their families to cope with its devastating effects. Lifestyle support services for those living with diabetes are an important element of the actions required to address diabetes at the community level.
   
   Community support groups and systems which help enhance quality of life will be established and programs that help support self-management by promoting positive lifestyle choices will be provided and funded. Programs will be implemented at a community level, and will involve people living with diabetes, their families, community leaders and community centres. Links to NGOs such as the Canadian National Institute for the Blind (CNIB), NADA and the Canadian Diabetes Association (CDA) will be encouraged.

1.3 Program Objectives

Working in partnership with Aboriginal committees, Health Canada's investments are intended to achieve the following objectives for the two ADI programs:

First Nations On-Reserve and Inuit in Inuit Communities

• Raise awareness of diabetes, its risk factors, and the value of healthy lifestyle practices.
• Support the development of a culturally appropriate approach to care and treatment, diabetes prevention and health promotion programs, and lifestyle support programs.
• Build capacity, linkages and infrastructure for all components of the ADI in First Nation and Inuit communities.
• Promote effective self-management of diabetes.
• Coordinate with other community-based programming, specifically the First Nations and Inuit Home and Community Care program.

Métis, Off-Reserve Aboriginal and Urban Inuit: Prevention and Promotion

• Raise awareness of diabetes, its risk factors, and the value of healthy lifestyle practices.
• Promote Aboriginal/Inuit ownership of diabetes primary prevention and health promotion programs.
• Ensure the fair and equitable allocation of available resources among Métis, off-reserve Aboriginal people and urban Inuit.
• Ensure that programming is delivered as equitably as possible across the country.
• Promote innovative approaches to diabetes primary prevention and health promotion projects.

1.4 Program Component Outcomes

The most important program outcomes associated with the three program components are presented in the following table.

Care and Treatment Outcomes

• Improved access to care and treatment for First Nations and Inuit with diabetes
• Stronger partnerships with First Nations and Inuit diabetes care providers  
• Increased capacity of First Nations and Inuit communities to manage effective diabetes programs
• Improved clinical outcomes including decreased rates and severity of complications and hospitalization

Prevention and Promotion Outcomes

• Overall improvement in diabetes awareness: risk factors, signs and symptoms, benefits of healthy lifestyle choices
• Implementation of effective diabetes prevention programs and projects
• Production and distribution of resource materials appropriate to First Nations, Métis and Inuit cultures and languages
• Improved early detection and reporting of diabetes

Lifestyle Support Outcomes

• Effective community-level programs in place to provide support and services to people with diabetes, their families and care providers
• Strengthened community-based organizations to support diabetes programs and services
• Improvement in quality of life and self-sufficiency for First Nations and Inuit with diabetes

1.5 ADI Funding Levels

The following table presents the five-year total and annual funding levels for the Aboriginal Diabetes Initiative.

1.6 Target Groups

The ADI aims to increase awareness of diabetes, and reduce the incidence and prevalence of diabetes related complications among Aboriginal people including the following groups:

• First Nations and Inuit of any age:
• Those who live on a First Nations reserve, or in an Inuit community, or in a First Nations community in the Yukon, Northwest Territories or Nunavut;
• Métis living in Métis communities, rural or urban centres;
• First Nations people living off-reserve in either rural or urban centres;
• Inuit living outside Inuit communities or in urban areas; and,
• Non-Status Indians.

1.7 Program Implementation and Delivery

The ADI is a five-year process. A National ADI Steering Committee was formed to guide the implementation and delivery of the ADI, and includes representatives from the major Aboriginal representative organizations, Health Canada, and the National Aboriginal Diabetes Association.

• In year one of the ADI (1999/2000), the Aboriginal Diabetes Initiative Steering Committee held implementation planning meetings with representatives from Aboriginal organizations, communities, regional and headquarters staff early in the winter of 1999/2000. These national and regional meetings confirmed the program needs and priorities, identified opportunities, and led to a full implementation framework. Additionally, funding was provided to existing pilot projects that are already established (that is, projects previously funded by National Health and Research Development Program (NHRDP), and given bridge funding by Medical Services Branch (MSB), Health Canada in 1998 to prevent the loss of capacity and momentum), to the National Aboriginal Diabetes Association, and to regionally-based implementation planning.
  
  
• Year two of the ADI (2000/2001) focused on implementing the full range of care and treatment, prevention and promotion and lifestyle support programs on-reserve and in Inuit communities across the country through the FNOIIC process; and creating a Request for Proposal process which resulted in the implementation of 16 primary prevention and health promotion programs under the MOAUIPP program.
  
  
• In years three and four (2001/2002 and 2002/2003), many programs created in year two will be continued, complete with accountability and reporting frameworks. Additional programming will be added as the funding level increases to its maximum.
  
  
• In year five (2003/2004), programs with accountability and reporting frameworks will be continued. Analysis of the gaps in programming, and ongoing program needs will be undertaken and an evaluation of outcomes (immediate outcomes) will also be conducted.

There will be two levels of ADI evaluations. Level I focuses on ongoing performance measurement. It will include monitoring, gap analysis and project evaluation for projects conducted in years three through five (2001/2002 through 2003/2004). Level II will be a final summative evaluation focusing on final program outcomes and impacts and the effectiveness of the ADI in reaching program goals and objectives. The results of the summative evaluation will be reported to the Minister of Health.

There is a need to build linkages with the Provinces, Territories, Regions, Communities, and other stakeholders. The ADI also has a linkage with the FNIHCC program, since much of the direct diabetes care and treatment services to those First Nations and Inuit living with diabetes will be delivered by FNIHCC workers in the community. There are also linkages with other programs such as Aboriginal Head Start (AHS), the Canada Prenatal Nutrition Program (CPNP), and the First Nations and Inuit Health Information System (FNIHIS).

1.8 Program Logic Model

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Click here to see the Program Logic Model (PDF version).

2.0 Evaluation Strategy

Program evaluation is a tool for good management practices that is used on an ongoing basis.

Program evaluation is a tool for good management practices that is used on an ongoing basis. In most cases, program evaluation is done at the end of a program's life cycle or according to a predetermined time frame. However, evaluation concepts should be incorporated on an ongoing basis at all stages of program delivery, especially at the front end, to articulate linkages between activities and outcomes. Though final evaluation of various components may be conducted, it is extremely important that timely, comprehensive, and strategic evaluation studies of projects and activities within each component are conducted on an ongoing basis to feed into the final evaluations of the components and the overall initiative.

As presented in Section 1, there will be two levels of evaluation. A process evaluation will focus on ongoing program monitoring (Level 1). It will include monitoring, gap analysis and project evaluation for the three years of full program implementation. Process evaluation is on-going, and complements and strengthens the summative (outcome) evaluation. An outcome/summative evaluation (Level 2) will focus on final program outcomes and impacts and the effectiveness of the ADI in reaching program goals and objectives.

Activities during the first year of bridge financing (1999-2000) will be covered during the first year of ongoing performance measurement (2000-2001).

2.1 Evaluation Challenges

The ADI Evaluation is an ongoing process covering a five-year time period and involving many stakeholders and Aboriginal organizations and communities from across the country. Following are some of the challenges to a successful ADI evaluation.

• Multiple players - Health Canada is only one of many players whose activities and policies influence the health of Canadians.
  
  
• Limited mandate and resources - Health Canada has a limited mandate and resources to act on a wide spectrum of activities elated to diabetes prevention, management and control.
  
  
• Long-term co-ordination of efforts - Reaching these targets and objectives requires long-term, co-ordinated efforts from all players across all jurisdictions in Canada.
  
  
• Limited time to measure ADI objectives - The time period of ADI, from 1999-2000 to 2003-2004, will not be sufficient to demonstrate the overall objectives. The evaluation will need to address the progress made by ADI as well as shorter-term or immediate outcomes.
  
  
• Provincial and territorial health reforms - Provincial and territorial health reform initiatives will have an impact on the diabetes care and treatment of Aboriginal people.
  
  
• Jurisdictional issues for program delivery - The policies and programs of other federal departments, provincial and territorial governments, and Aboriginal structures will have a direct impact on the health status of Aboriginal people. This issue may be particularly important in Nunavut and the Northwest Territories where there is no program delivery directly to Inuit except through government.
  
  
• Determinants of health - There are a variety of factors, known as the determinants of health, which influence the health of] Canadians.
  
  
• Establishing causality - In most instances, given the variety of factors and agents influencing change, it will be difficult to establish direct causal linkages between the ADI and changes in the health status of Aboriginal Canadians.
  
  
• Multiple jurisdictions - There are many jurisdictions involved in the delivery of health care including provinces, territories, Health Canada, Aboriginal structures and non-governmental agencies. Achieving key results will require long-term efforts from all partners across all jurisdictions (local, regional and national), going beyond a five-year period.
  
  
• Linkages with other First Nations and Inuit Home and community programs - ADI provides funding for specific types of projects. At any given point in time, many community programs are administered simultaneously: e.g. First Nations and Inuit Home and Community Care (FNIHCC) and Aboriginal Head Start (AHS). In many cases, the programs are delivered collaboratively. Success or failure of the ADI could depend on how other programs are delivered in the community.
  
  
• ADI and FNIHCC have different funding levels - In some communities, ADI and FNIHCC resources are integrated to achieve better results. ADI provides funding for a defined period of time under its 5 year mandate while the FNIHCC program has ongoing funding. This difference could complicate the integration of activities and the development of delivery mechanisms.
  
  
• Tracking when funding is distributed - Many of the ADI projects will be sponsored and delivered at several levels including national, regional, Tribal Council and community levels. Tracking the timing of the funding distribution by organizations operating under different financial reporting practices will require reliable reporting mechanisms and ongoing monitoring.
  
  
• Sharing information among program partners and stakeholders - Sharing, retrieving and reporting on information from provinces, regions, communities and other stakeholders may be difficult. The large number of program participants will complicate communications. Managing the overlap and linkages of information from programs with different agendas and mandates, such as the FNIHCC program and The National Diabetes Surveillance System (NDSS), will present significant challenges. There may also be reluctance among delivery partners to establish new and additional communications protocols when existing ones are working properly.

2.2 Evaluation Tiers

The evaluation will be conducted at four tiers or levels of aggregation.

• Communities;
• Band Council / Tribal Council;
• First Nations and Inuit Health Branch (FNIHB) Regional; and,
• National.

Evaluation data can be summarized to support analysis for each of these four tiers. For a program of the size and complexity of the ADI, specific data sources will be more useful for different evaluation tiers. For example, the NDSS will provide very useful data at the national and regional levels but the NDSS data may not support analysis at the Band/Tribal Council and community levels of analysis. On the other hand, consultations, reviews or case studies conducted locally may provide valuable results for community level evaluation but be insufficient to support analysis at the regional or national levels.

The tables presented in Section 2.3 and 2.4 present a preliminary review of the correspondence between different evaluation questions and methods and the four evaluation tiers. These relationships will have to be revisited after the evaluation methodology has been designed and the evaluator has had a chance to assess the quality of the data collected through the different methods.

2.3 Evaluation Questions and Indicators

Consistent with the approach of evaluating the CDS presented in the Treasury Board submission (CDS Evaluation Framework), the progress toward meeting the goals of the ADI will be measured according to the following criteria:

• Relevance - whether program components are still needed;
• Progress/Success - the extent to which activities and outputs have led to the specific outcomes identified in the program plan;
• Implementation & Management - the extent to which implementation and management were appropriate and enabled the achievement of the identified outcomes; and,
• Cost-effectiveness - the costs associated with each outcome.

The general ADI evaluation questions that correspond to these four criteria, along with some key indicators, are presented in the following table. Preliminary lists of data sources and data collection methodologies as well as evaluation tiers are also presented. Similar tables for the three program components follow in Section 2.4.

Program-Level Evaluation Questions

1. Relevance
   
   
   • To what extent has the ADI been relevant to increasing diabetes awareness, and reducing the incidence and prevalence of diabetes-related complications among First Nations, Métis and Inuit populations in Canada?

   Indicators

   • Incidence and prevalence rates of diabetes and its complications - Morbidity and mortality rates among people living with diabetes

   Methodology/Data Sources

   • NDSS
   • First Nations and Inuit Health Information System (FNIHIS)
   • First Nations, Métis and Inuit community organizations and care providers

   Evaluation Tiers

   • National
   • Regional
     
     
2. Progress / Success
   • To what extent have ADI activities contributed to outcomes?
   • Improved access to care and treatment
   • Increased awareness and prevention of diabetes
   • Community programs to improve quality of life and self-sufficiency for people with diabetes

   Indicators

   • Improved clinical outcomes
   • Higher rates of awareness of diabetes risk factors and prevention
   • Improvements in quality of life and self-sufficiency for First Nations and Inuit with diabetes

   Methodology/Data Sources

   • NDSS/FNIHIS
   • Aboriginal community organizations and care providers
   • Surveys of Aboriginal populations
   • Consultations with people living with diabetes

   Evaluation Tiers

   • National
   • Regional
   • Band Council/ Tribal Council
   • Community
     
     
3. Implementation and Management
   
   
   • To what extent are the implementation and management of the ADI appropriate to achieve the program objectives?

   Indicators

   • Program/project funding reflects the ADI objectives and priorities
   • Funding allocation according to program plan
   • Collaborative arrangements with Aboriginal care providers

   Methodology/Data Sources

   • Program data review
   • Case study/review of selected projects
   • Consultations with program managers, project sponsors, and Aboriginal organizations

   Evaluation Tiers

   • National
   • Regional
     
     
4. Cost-effectiveness
   
   
   • To what extent has the ADI used the most appropriate, efficient and cost-effective methods to achieve program objectives?

   Indicators

   • Overall ADI cost-effectiveness - Cost-effectiveness of different program options and project types
   • Appropriateness of diabetes initiatives to Aboriginal peoples and communities

   Methodology/Data Sources

   • Program administrative data review
   • Case study/review of selected projects
   • Cost-benefit analysis of ADI project types

   Evaluation Tiers

   • National
   • Regional

2.4 Component-Level Evaluation Questions and Indicators

Care and Treatment

Key Evaluation Question

• Has ADI improved access to care and treatment for First Nations and Inuit with diabetes?

Indicators

• Increased number of on-reserve and Inuit communities served by diabetes care and treatment programs
• Increased proportions of on-reserve and Inuit populations with access to care and treatment
• Improved screening and early detection of diabetes

Methodology/Data Sources

• NDSS/FNIHIS
• First Nations and Inuit community organizations and care providers

Evaluation Tiers

• National
• Regional
• Band Council/ Tribal Council
• Community

Key Evaluation Question

• Has ADI increased the capacity of First Nations and Inuit communities to manage effective and sustainable diabetes programs?

Indicators

• Stronger partnerships with First Nations and Inuit organizations providing care and treatment
• Greater participation of First Nations and Inuit organizations in care and treatment programs
• Improved capacity to manage diabetes programs

Methodology/Data Sources

• Consultations with Aboriginal organizations and care providers
• Consultations with program managers, health care providers
• Review of project files

Evaluation Tiers

• National
• Regional
• Band Council/ Tribal Council
• Community

Key Evaluation Question

• Did ADI promote effective management of diabetes?

Indicators

• Partnerships in delivery of programs
• Timely delivery of programs and services

Methodology/Data Sources

• Program data review
• Consultations with program managers, project sponsors, and Aboriginal organizations

Evaluation Tiers

• National
• Regional
• Band Council/ Tribal Council
• Community

Key Evaluation Question

• Has the ADI built capacity, linkages and infrastructure for all components of the ADI in First Nation and Inuit communities?

Indicators

• Increase in Aboriginal organizations capable of delivering diabetes programs and services
• Partnerships in program delivery

Methodology/Data Sources

• Program data review
• Consultations with program managers, project sponsors, and Aboriginal organizations
• Provincial and Territorial governments

Evaluation Tiers

• National
• Regional
• Band Council/ Tribal Council
• Community

Key Evaluation Question

• Has the ADI coordinated with other community-based programming, specifically the First Nations and Inuit Home and Community Care program?

Indicators

• Partnerships in delivery of programs

Methodology/Data Sources

• Program data review
• Consultations with ADI and other program managers, project sponsors, and Aboriginal organizations

Evaluation Tiers

• National
• Regional

Prevention and Promotion

Key Evaluation Question

• Has ADI led to an overall improvement in awareness of diabetes, its risk factors, and the value of healthy lifestyle practices?

Indicators

• Increased awareness among community organizations
• Increased awareness among First Nations, Métis and Inuit populations
• Awareness among high-risk subgroups

Methodology/Data Sources

• NDSS/FNIHIS
• Consultations with First Nations, Métis and Inuit community organizations and care providers
• Surveys of Aboriginal populations

Evaluation Tiers

• National
• Regional
• Community

Key Evaluation Question

• Has ADI provided for implementation of effective diabetes prevention programs for all Aboriginal people?

Indicators

• Implementation of diabetes awareness and prevention programs and projects
• Increased awareness of diabetes, risk factors, and benefits of healthy lifestyle practices
• Reduced incidence of diabetes

Methodology/Data Sources

• NDSS/FNIHIS
• Program data review
• Consultations with Aboriginal community organizations and care providers
• Consultations with program managers

Evaluation Tiers

• National
• Regional
• Band Council / Tribal Council
• Community

Key Evaluation Question

• Has ADI supported the development of a culturally appropriate approach to prevention and health promotion programs?

Indicators

• Support of First Nations, Métis and Inuit organizations
• Development and use of resource materials appropriate to Aboriginal languages and cultures

Methodology/Data Sources

• Review of program data
• Case study/Review of selected projects
• Consultations with project sponsors and Aboriginal organizations

Evaluation Tiers

• National
• Regional
• Band Council / Tribal Council
• Community

Key Evaluation Question

• Has ADI promoted Aboriginal/Inuit ownership of diabetes primary prevention and health promotion programs?

Indicators

• Partnerships in delivery of programs
• First Nations, Métis and Inuit sponsorship of ADI projects

Methodology/Data Sources

• Review of program data
• Case study/Review of selected projects
• Consultations with project sponsors and Aboriginal organizations

Evaluation Tiers

• National
• Regional

Key Evaluation Question

• Has the allocation of available ADI resources been fair and equitable among Métis, off-reserve Aboriginal people and urban Inuit?

Indicators

• Percentage allocation of program funds among target groups

Methodology/Data Sources

• Review of program data

Evaluation Tiers

• National
• Regional

Key Evaluation Question

• Has ADI promoted innovative approaches to diabetes primary prevention and health promotion projects?

Indicators

• Peer ratings of ADI projects
• Introduction of new programs for First Nations, Métis and Inuit people

Methodology/Data Sources

• Review of program data
• Case study/Review of selected projects
• Consultations with program managers, project sponsors and Aboriginal organizations

Evaluation Tiers

• National
• Regional
• Band Council / Tribal Council
• Community

Lifestyle Support

Key Evaluation Question

• Have effective and relevant diabetes support programs and services been put in place?

Indicators

• Timely implementation of ADI-sponsored projects
• Peer ratings of effectiveness and relevance of programs and services introduced

Methodology/Data Sources

• Review of program data
• Consultations with program managers, project sponsors, Aboriginal and Inuit organizations

Evaluation Tiers

• National
• Regional
• Band Council / Tribal Council
• Community

Key Evaluation Question

• Is there an improvement in the quality of life for First Nations and Inuit with diabetes because of ADI?

Indicators

• Increased access to support services for individuals with diabetes, their families and care providers
• Support groups created in communities

Methodology/Data Sources

• NDSS/FNIHIS
• Consultations with Aboriginal and Inuit organizations
• Survey and consultations with individuals with diabetes and care providers

Evaluation Tiers

• National
• Regional
• Band Council / Tribal Council
• Community

Key Evaluation Question

• To what extent has ADI promoted self-sufficiency and management of diabetes for First Nations and Inuit?

Indicators

• Increased awareness of diabetes management practices
• Increased access to support services
• Increased availability and access to equipment and supplies needed for self-management

Methodology/Data Sources

• Review of program data
• Consultations with Aboriginal and Inuit organizations
• Survey and consultations with individuals with diabetes and care providers

Evaluation Tiers

• National
• Regional

2.5 Methodology and Data Sources

Wherever possible, the methods and data sources used to evaluate the ADI will be similar to those used for the evaluation of the overall CDS. These include the NDSS, the primary data source for monitoring and evaluating the CDS. Other sources developed for the CDS that could be used for the ADI evaluation (perhaps in modified form) include the template for minimum data collection from funded community projects, and the MOAUIPP project database.

Following are some of the other data sources and methods that could be used for the ADI evaluation. This list is preliminary and presents possible sources only. A more precise specification of data sources and methods will be made during the evaluation design when more information about ADI projects and participants is available.

1. Data Collection Methodologies

   Administrative program data review - The large array of data collecting during the process of administering the program will be an important source of information for the evaluation. Some of these data sources are as follows:

   • ADI project data including applications, workplans, and progress reports (where required and/or available);
   • activity reports;
   • funding and financial reports;
   • reports from related programs such as the NDSS and FNIHCC; and,
   • annual reports from CDS.

   Project document review - Every project will have some documentation in addition to the initial application. Such documentation could include communications materials (e.g. brochures, posters, videos), training guides, planning documents, meeting agendas or notes, correspondence, and summaries or reports of various types. Asking sponsors to provide key documents produced during their project activities will assist Health Canada with ADI monitoring and evaluation. Sampling projects and requesting more detailed documentation about the activities and outcomes can also provide valuable information for a project review.

   Key informant interviews - Consultations with stakeholders, program partners, project sponsors and other relevant individuals and organizations can be conducted through key informant interviews. Key informant interviews are usually conducted with a single individual and are confidential. They can also be conducted with small groups. Both in-person and telephone interviews can be used depending on the subject and length of time required for the interview and the preferences of the respondent. Practical considerations such as travel costs are also very important.

   Expert Interviews - Consultations with several types of experts may be useful to program monitoring and evaluation. Medical personnel with expertise in diabetes, community care experts, social workers and counsellors, nutrition and recreation experts, and communications and marketing experts could make useful contributions to the evaluation. An understanding of and sensitivity to the First Nations, Métis and Inuit populations served by the ADI would be essential for experts consulted for the evaluation.

   Surveys of populations served by ADI - Surveys of larger numbers of program clients (i.e. representative probability samples) can help to provide a broad overview of program outcomes and impacts. Such surveys typically collect relatively small amounts of quantitative information from large numbers of people. The advantage is that the information collected can be used to make statistical inferences (within identifiable error parameters) to the Aboriginal populations served by the program. Surveys of Aboriginal populations present particular challenges in terms of interview methods, access to representative samples of study populations (sample bias), and logistics. Careful planning and the support of the groups being surveyed are essential to success.

   Project reviews and evaluations - Some organizations sponsoring ADI projects may have their own processes for reviewing or evaluating activities. Methods could include case studies, self-evaluations, and community consultations. Such information could be incorporated into ADI monitoring and evaluation where organizations are willing to share their results.

   Case studies/reviews of projects - Case studies or other types of more detailed reviews could be conducted for selected projects. Case studies, which typically involve between two and five days of a researcher's time, would involve in-depth examinations of all aspects of particular projects. Case studies would provide detailed evidence about the activities of a project and its outcomes and impacts. A good case study will also provide insights into the reasons for success or failure.

2. Methods of Analysis

   Performance monitoring - Performance monitoring can be conducted for ADI projects for which progress data is available. Progress data collection should include measures that correspond to the indicators presented in this evaluation framework.

   Cost-benefit analysis - At the project level, cost-benefit analysis would require detailed data that could be collected through a case study or project audit. Cost-benefit analysis can be conducted after sufficient time has passed to allow for meaningful measurement of outcomes and benefits. Cost-benefit analysis at higher levels of aggregation could be conducted once adequate project-level data is available.

   Analysis of linkages - An analysis of linkages with other key programs could be conducted at an early stage in the process. Key stakeholders should be consulted for an early review of the extent to which ADI is being integrated successfully with other programs.

   Gap analysis - A gap analysis can be conducted to make sure that workplans and activities conform to ADI objectives. Gap analysis involves the systematic comparison between planned activities and actual with the stated objectives of the ADI. These comparisons can be made at each level of analysis. Gap analysis can be used to make sure that the program is on track and to identify any required mid-program changes.

3.0 The Evaluation

3.1 Evaluation Funding

Evaluation funding will be at two levels, with funding commensurate to the areas of responsibility. National Office will provide technical and financial support to the Regions, with the amounts to be discussed and reviewed on an ongoing basis.

The Community Health Programs Directorate of the First Nations and Inuit Health Branch of Health Canada will be responsible for the ADI evaluation

3.2 Evaluation Responsibility

The Community Health Programs Directorate of the First Nations and Inuit Health Branch of Health Canada will be responsible for the ADI evaluation, including the ongoing performance measurement over the next three years and the final summative evaluation in 2004.

National Office will be responsible for the evaluation framework and design, and technical and financial support to Regions (as required). Regions will be responsible for regional level projects and establishing connections with Tribal Council and community level projects.

Evaluation Responsibilities

ADI Evaluation Framework
National Office

Gap Analysis
National Office

Canadian Diabetes Strategy (CDS) Annual Reports
National Office

ADI Annual Reports
National Office

Evaluation - Ongoing Performance Measurement
National Office, Regional Offices

Evaluation - Final Summative Evaluation Study
National Office, Regional Offices

3.3 Changes to the ADI Evaluation Framework

This evaluation framework, prepared in July 2001 (reviewed and edited in November), reflects the design of the ADI and incorporates the two programs and three components. It is likely that further changes in the upcoming years could affect the contents of this Framework. The Director General, Community Health Programs, will be responsible for accommodating those changes and for notifying all stakeholders including Treasury Board.