Evaluation of the Drug Strategy Community Initiatives Fund 2008-2009 to 2012-2013

Prepared by Office of Evaluation Health Canada and the Public Health Agency of Canada

July 2014

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes for Youth (Effectiveness)
  • 4.5 Performance: Issue #4 - Achievement of Expected Outcomes for Communities (Effectiveness)
  • 4.6 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Conclusions - Relevance
  • 5.2 Conclusions - Performance
• 6.0 Recommendations
• Appendix 1 - Evaluation Description
• Appendix 2 - Summary of Findings
• List of Tables
  • Table 1: Age groups served by the DSCIF projects
  • Table 2: Geographic Area Served by Projects
  • Table 3: Intermediaries engaged in DSCIF projects
  • Table 4: Limitations and Mitigation Strategies
  • Table 5: Resources produced by DSCIF Projects
  • Table 6: DSCIF Planned Versus Actual Contribution Funding
  • Table 7: DSCIF Planned versus Actual O&M and Salary

Executive Summary

This evaluation covered the Drug Strategy Community Initiatives Fund (DSCIF) program for the period from 2008-09 to 2012-13. The evaluation was undertaken to fulfill the requirements of the Financial Administration Act and the Treasury Board of Canada Policy on Evaluation (2009) to conduct an evaluation of all ongoing grant and contribution programs every five years.

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the DSCIF program. The evaluation examined activities undertaken at the Program level, as well as the activities of projects funded as a result of the 2007 and 2009 Call for Proposal (CFP) processes. It excluded an assessment of the performance of projects funded through the 2013 CFPs as they only received approval at the end of fiscal year 2013-14.

Program Description

The DSCIF program is an ongoing community-based contribution program created in 2004 under Canada's Drug Strategy (CDS) and realigned in 2007 to support the Prevention Action Plan of the federal government's National Anti-Drug Strategy (NADS). Health Canada is the lead department for the Prevention Action Plan. The objectives of DSCIF are to facilitate the development of local, provincial, territorial, national and community-based solutions to illicit drug use among youth and to promote public awareness of illicit drug^{Footnote i} use among youth. Overall, the DSCIF program had a budget of $56.6M over five years. Approximately $50 million was contribution funding to community-based organizations and other stakeholders across Canada to carry out front-line health promotion and prevention activities to prevent illicit drug use among youth ages 10 to 24 years who, for various reasons, are at risk of using illicit drugs.

Conclusions - Relevance

All the lines of evidence confirmed the ongoing relevance of the DSCIF.

Continued Need

There is an ongoing need for community-based health promotion and prevention efforts to build capacity and to address gaps and emerging issues associated with illicit drug use. National surveys indicate that the conditions that led to the realignment of DSCIF under the NADS in 2007 still exist. Rates of illicit drug use among youth 15 to 24 years of age remain higher than among adults. Certain subpopulations of youth (Aboriginal, street-involved/homeless youth and lesbian, bi-sexual, gay and transgendered (LGBT) youth) have even higher rates of illicit drug use. The literature indicates that prevention efforts can mitigate a range of legal, social and economic impacts of illicit drug use. The DSCIF provides an important source of funds to support collective action at the community level, in a way that is appropriate for the unique circumstances of each community.

Alignment with Government Priorities

The DSCIF program's health promotion and prevention activities support the federal government's priority areas of illicit drugs and youth. Health Canada is the lead on the NADS Prevention Action Plan. The DSCIF, as a key component of the Prevention Action Plan, therefore aligns with federal priorities for illicit drug use as articulated in the NADS.

The objectives of the DSCIF program to raise awareness and understanding of the harmful social and health effects of illicit drug use also supports Health Canada's Strategic Outcome #2 -- health risks and benefits associated with food, products, substances, and environmental factors are appropriately managed and communicated to Canadians.

There are no clear, planned formal mechanisms in place to ensure the appropriate internal and external stakeholders are engaged at key decision points in the program lifecycle. Stakeholder engagement should be planned and deliberate so that the purpose and intent of engagement are obvious to all involved.The policy function is currently located in the Healthy Environments and Consumer Safety Branch (HECSB) and the management and administration of the DSCIF contribution agreements is located in the Strategic Policy Branch (SPB). Multiple reorganizations, staff turnover and reductions have contributed to decreased collaboration between the two areas at key decision points (e.g., policy decisions, decisions on DSCIF funding priorities and CFP processes). The evaluation found that the transition from a regional to national delivery model has also impacted on Health Canada's level of engagement with external stakeholders in identifying gaps and determining funding priorities.

Alignment with Federal Roles and Responsibilities

Education and prevention are typically the responsibilities of the provinces and territories (P/Ts). Funding community-based programming is generally not a federal role.However, the concerted effort of all levels of government is necessary to address the complexity and many challenges associated with illicit drug use, as well as the differing level of capacity to address illicit drug use among youth across the country.

The P/Ts generally address substance abuse more broadly to include both licit and illicit substances. The federal government has played a role at the broad policy level through NADS to address illicit drug use. Health Canada supports this role as the lead on the Prevention Action Plan of the NADS. It fulfills the federal commitment to implement the Prevention Action Plan by funding innovative projects and mobilizing a wide range of community-based organizations to carry out health promotion and prevention activities that focus on illicit drugs while also supporting the role of the P/Ts. However, as the environment has recently changed for this Program, an opportunity exists to examine the direction and scope of the new Program going forward to more fully align with the federal role.

Most P/Ts and project stakeholders supported a role for Health Canada as a funder but also noted that Health Canada could have a stronger and more influential role as a funder of innovative practices, in knowledge translation and exchange, and in facilitating collaboration and coordination to address illicit drug use among youth.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

To assess the achievement of outcomes, the evaluation relied on outcome data collected by the Program, and supplemented this information with key informant interviews and case studies.

DSCIF program produced the expected outputs and made progress in achieving most of its immediate and some intermediate outcomes. Reporting focussed on the lifecycle of the agreements and therefore longer term outcomes were not captured as part of the reporting tools. For some projects, this was a lost opportunity to collect information on longer term outcomes, such as, behaviour change. However, a few projects reported on progress toward achieving longer term outcomes. Also, given the small scope of the DSCIF program, there was no expectation that the Program would have national level prevalence impacts.

The evaluation confirmed that youth who participated in the DSCIF funded projects increased their awareness and understanding of healthy lifestyle choices, and illicit drugs and their negative consequences and improved their capacity (knowledge and skills) to avoid illicit drugs. All funded organizations were able to provide strong evidence of engagement of community partners and networks in efforts to prevent illicit drug use.

Funding recipients produced knowledge products and resources and increased the access to and awareness of these knowledge products and resources within their community and networks. There was also evidence that progress was made with respect to the community uptake of these knowledge products and resources.

The evaluation identified a gap in terms of knowledge translation and pan-Canadian knowledge exchange. The logic model identifies knowledge exchange as a program-level activity. Although the Program has summarized progress and evaluation reports, and produced a lessons learned report, Health Canada's role in knowledge translation (analysis and synthesis of the performance data from the funded projects) to identify promising practices, to inform program decision making or in the pan-Canadian dissemination of the knowledge products and resources (linked to promising practices) produced by the funded projects was limited. Some passive knowledge diffusion took place at the project level through the funded organizations. However, smaller organizations, in particular, lacked the capacity for knowledge exchange beyond their immediate community and networks.

The DSCIF program has taken a number of steps to ensure the availability of performance data at the program level. The Program has the opportunity to leverage the performance data it has collected and to use it strategically to inform program priority setting and funding decisions, and to adjust DSCIF objectives and program design. Furthermore, the Program could contribute to the knowledge base on drug prevention programming for youth by further analyszingthe project performance data it has collectd to identify promising practices as well as promising knowledge products and resources produced by projects. Disseminating this information more broadly would enable stakeholders to act on and use the information, thereby extending the impact of the Program. P/T and project stakeholders expect Health Canada, or another national organization, to have a role in knowledge translation and dissemination.

Demonstration of Economy and Efficiency

Overall, the contributions to community-based organizations were well managed in terms of monitoring progress and ensuring compliance with contribution agreements. The Program has made efforts to operate more efficiently, however the evaluation identified opportunities for additional efficiencies by streamlining the CFP and performance reporting processes. The evaluation also identified opportunities to improve collaboration and coordination between the policy and program delivery organizations within Health Canada which would help increase efficiency in the approach to managing funding agreements.

The evaluation noted that many of the funding recipients interviewed found that reporting requirements could be burdensome and that the time invested in meeting these requirements detracted from time for program delivery. While performance data was available to inform the evaluation, reporting activities represented a significant investment of National Office resources. For the next round of funding, the Program has streamlined monitoring and reporting requirements using a risk-based approach to reporting. However, in moving forward, the Program should assess its performance reporting requirements to ensure that they are sufficient to produce quality performance data, but not excessive for recipients.

Recommendations

The evaluation identified the following two recommendations.

Recommendation 1

Significant changes to the program infrastructure, including recent Government of Canada announcements on the inclusion of prescription drug abuse (as an area of focus for DSCIF), the proposed merger of DSCIF and the Drug Treatment Funding Program (DTFP), the impact of the transition from regional and national program delivery on the level of service that the Program can provide funding recipients and the expectations of stakeholders with respect to knowledge translation and exchange, suggest the need for the HECSB and the SPB to engage in a policy and strategic planning exercise to define the parameters of the new substance abuse program going forward.

Health Canada stakeholders should engage in a policy and strategic planning discussion that will lead to a decision on the direction and scope of the new Program going forward. This could include identifying:

• the delivery model for the new Program within the broader Health Canada context of controlled substances;
• Health Canada's role in knowledge translation and exchange and in the dissemination of project knowledge products and resources; and
• the roles and responsibilities of Health Canada stakeholders (HECSB policy and research and surveillance, and the SPB Drugs Program), and collaborative mechanisms to engage stakeholders in setting departmental policy direction on illicit substances and making programmatic decisions.

Recommendation 2

The Program has accumulated a wealth of project performance data that has not yet been leveraged to the fullest extent to identify information and promising practices that can inform Health Canada and stakeholder policy and program decisions.

The Program should:

1. leverage its investment in performance measurement by further analyzing data collected from the projects and by using lessons learned strategically to inform program priority setting and funding decisions and to adjust the DSCIF objectives and program design.
2. disseminate lessons learned and project developed knowledge products and resources, linked to promising practices, to inform the activities of stakeholders.

Management Response and Action Plan

1.0 Evaluation Purpose

The purpose of the evaluation of the DSCIF program was to assess the relevance and performance (effectiveness, efficiency and economy) of the Program for the period 2008-09 to 2012-13. The results will also inform the implementation of current and future activities of Health Canada drug prevention programming.

The evaluation was required by the Financial Administration Act (1985) and the Treasury Board of Canada Policy on Evaluation (2009).

2.0 Program Description

2.1 Program Context

The DSCIF is an on-going contribution program created in 2004 under Canada's Drug Strategy (CDS). CDS was announced by the Government of Canada in May 2003 following calls for a comprehensive renewed drug strategy from the Auditor General of Canada (December 2001), the Senate Special Committee on Illegal Drugs (September 2002) and the House of Commons Special Committee on Non-Medical Use of Drugs (December 2002).^{Footnote 1,Footnote 2,Footnote 3} CDS encompassed core federal activities of prevention, treatment, enforcement and harm reduction to address a broad range of substances, including the use of illegal drugs and the abuse of alcohol and psychoactive pharmaceuticals. Under the CDS, the DSCIF focussed its prevention efforts on a broad range of substances, both legal and illegal, and included a harm reduction approach to substance abuse.

The Government of Canada announced the National Anti-Drug Strategy (NADS) in 2007 to improve Canada's response to illicit drug use. The NADS focuses explicitly on illicit drug issues and includes federal programming initiated under CDS as well as new initiatives. The goal of NADS is to improve the health and safety of Canadians through coordinated efforts that support its three action plans: preventing illicit drug use (with a focus on youth), treating illicit drug dependency, and combating illicit drug production and distribution. The NADS is a horizontal initiative, led by the Department of Justice Canada in collaboration with 11 partner federal departments and agencies.

In 2007, the DSCIF was refocused to align with the NADS focus on illicit drugs and funding was reoriented from CDS to the Prevention Action Plan of the NADS. The DSCIF's Terms and Conditions were renewed in 2010 to reflect the new focus of the Program.

Health Canada has the lead for the Prevention Action Plan of the NADS. As a partner department under the NADS, the DSCIF is the primary mechanism available to Health Canada to support community-based organizations in preventing illicit drug use among youth.

The objectives of DSCIF are to facilitate the development of local, national, P/T and community-based solutions to substance abuse problems, and promote public awareness of substance abuse issues.^{Footnote 4} These objectives align with the objectives of the Prevention Action Plan of the NADS:

• prevent youth from using illicit drugs by enhancing their awareness and understanding of harmful social and health effects of illicit drug use; and
• develop and implement community-based interventions and initiatives to prevent illicit drug use.^{Footnote 5}

To achieve these objectives, and to address the complexities and interrelationships of the many challenges associated with illicit drug use as well as the varying needs and priorities across the country, the DSCIF provides contribution funding to community-based organizations across Canada to carry out front-line health promotion and prevention activities to improve the level of awareness and knowledge of illicit drug use, improve attitudes and behaviour, increase resiliency and enhance the capacity of youth to make better choices, adopt healthy behaviours and ultimately avoid illicit drug use.

The DSCIF program also funds national organizations such as the Canadian Centre on Substance Abuse (CCSA) to address illicit drug use among youth. For example, a contribution to the CCSA funded the development of a drug prevention strategy for Canada's youth. Other national funding agreements include the Council on Drug Abuse, the Schizophrenia Society of Canada, and the Students Commission of Canada.

2.2 Program Profile

The DSCIF program provided approximately $50 million in contribution funding over five years to local, P/T, national and community-based organizations for illicit drug use among youth. The DSCIF program has issued three Calls for Proposals (CFP) since 2007 and has funded a total of 136 projects.

2.2.1 Program Target Population

The target population of the DSCIF are youth ages 10 to 24. The DSCIF target population encompasses youth who for various reasons may be at risk of using illicit drugs. Many factors are associated with this risk, such as the youth's age and stage of development, living circumstances, relationships with family and school, peer associations and the availability and access to illicit drugs.

The 2007 CFP targeted the general youth population by focusing on illicit drugs most likely to be "tried" by this group, and addressing the contexts/situations that often give rise to their drug use.^{Footnote 6} The 2007 CFP was the first to reflect the reorientation of the DSCIF to the new NADS focus on illicit drugs and sought submissions from a variety of organizations, such as not-for-profit organizations; educational institutions; other levels of government; Metis, Inuit and off-reserve First Nations organizations; business associations; and community coalitions.

The 2009 CFP targeted the general youth population and also identified as priorities specific subpopulations of youth who demonstrated risk factors associated with illicit drug use. These subpopulations included street involved youth or youth at risk of becoming street involved, youth in care (e.g., child welfare system), LGBT and two-spirit youth, and Aboriginal youth (off-reserve).^{Footnote 7}

The priorities for the 2013 CFP focused on specific youth populations known to have an increased risk of using illicit drugs compared with the general population and sought projects to prevent illicit drug use through behavioural changes in the following priority populations:

• youth in life transition stages (e.g., moving from elementary school to high school, high school to college/CEGEP/university, and high school/college/CEGEP/university to the work force);
• youth disadvantaged by their living conditions(e.g., parents abuse drugs, adolescents with mental health and substance use disorders, and other high risk youth); and
• youth living in rural/remote communities.

The Program reached the target population of youth between the ages of 10 and 24 years of age. Table 1 shows the age distribution targeted by the projects funded by the 2007 and 2009 CFPs. Most projects targeted more than one age group. The majority of projects targeted the 13 to 18 age group.^{Footnote ii}

Projects also targeted urban, rural or remote/northern communities or some combination of these three. As shown in Table 2, urban communities were served by most projects.

2.2.2 Program Beneficiaries

The beneficiaries of the DSCIF health promotion and prevention activities are:

• young people contemplating or experimenting with illicit drugs, including targeted at risk or vulnerable populations;
• parents of young people, who will benefit from enhanced knowledge about how to prevent and address their children's drug use; and
• community intermediaries (educators, health, social service, professionals, police, researchers and related communities of practice), who will benefit from access to enhanced knowledge about how to address and respond to illicit drug use affecting their communities.

Ultimately, Canadians will benefit from any reduction in the use of illicit drugs. Issues associated with illicit drug use directly or indirectly affect all Canadians because of their potential to negatively impact personal and community health and safety, increase crime, including organized crime, and create additional costs for our justice, health, and social delivery systems.

2.2.3 Program Reach

The DSCIF program's reach extends to communities. It engages a range of intermediaries and stakeholders to address the issues associated with illicit drug use among the youth population.

The primary categories of intermediaries for the 2007 and 2009 CFPs were educators, social service providers, parents and families. Table 3 shows the range of intermediaries engaged in DSCIF funded activities for these two CFPs.

2.2.4 Program Delivery

The DSCIF program funding guidelines indicate that the Program's eligible recipients include all levels of government, Aboriginal communities, non-governmental organizations (NGOs), Canadian institutions including universities, boards of education and other centres of education, Métis, Inuit and off-reserve First Nations not-for-profit organizations, business sector organizations and professional associations with an interest in addressing the issue of illicit drug use among youth.

As a result of this broad range of funding recipients, there is a considerable degree of variability in the type of health promotion and prevention activities carried out by the funded organizations.

2.2.5 Program Governance

The Department of Justice Canada has the overall lead for the NADS. Health Canada is the lead department for the NADS Prevention Action Plan. The Healthy Environments and Consumer Safety Branch (HECSB) within Health Canada is the Office of Primary Interest (OPI) and the main liaison between Health Canada and the Department of Justice Canada for the NADS and as such, is responsible for providing all Health portfolio, including DSCIF, performance data to the Department of Justice. During the period covered by the evaluation, the DSCIF manager participated in the NADS Prevention and Treatment Working Group and led the Prevention and Treatement sub-working group.

The DSCIF program is currently situated within the Drugs Program of the SPB. However, during the period covered by the evaluation the Program experienced several organizational changes. Prior to 2009, DSCIF was managed by the HECBS, Drug Strategy and Controlled Substances Program. However, with the creation of the Regions and Programs Bureau (RAPB) in 2009, the DSCIF program moved from the HECSB to the RAPB. However, the drugs policy function stayed with the HECSB. As a result of Budget decisions in 2012, the Drugs Program, including the DSCIF, moved from the RAPB to the SPB. The SPB assumed responsibility for the coordination and implementation of the DSCIF, and the administration of the DSCIF contribution agreements. In this role, the SPB is responsible for ensuring that funding is allocated and spent in accordance with established criteria and guidelines, that government procedures are adhered to, and that reporting and accountability standards are met.

For much of the period of the evaluation, contribution agreements were managed and administered by national and regional offices located in Halifax, Montreal, Ottawa, Toronto, Regina, Edmonton, Vancouver and Whitehorse. During 2012-13 the DSCIF transitioned from a regionally delivered program to a nationally delivered program. As a result, all contribution agreements with community-based organizations are now administered by the national office (which includes national staff located in British Columbia and Quebec).

Health Canada engages partners, including P/T governments and other key stakeholders in setting objectives and establishing priorities for the DSCIF. Health Canada maintains linkages with P/T governments on substance abuse issues primarily through the Federal /Provincial/ Territorial (F/P/T) Liaison Committee on Problematic Substance Use, but also through the F/P/T Ministers Responsible for Health and the F/P/T Committee on Substance Use and Abuse.

2.3 Program Logic Model and Narrative

2.3.1 Description of the Logic Model^{Footnote iii}

The logic model shows the activities, outputs and immediate, intermediate, long-term and ultimate outcomes of the program which will contribute to a reduced demand for illegal drugs among youth.

Activities, Outputs and Outcomes:

Activities:

The logic model identifies two sets of activities:

• project design and funding, including program promotion and stakeholder engagement; and
• project-level activities, including the planning, organization and delivery of health promotion

Outputs:

The outputs of project design and funding activities are tools and resources for managing the program and the contribution funding process.

The outputs of project-level activities are health promotion and prevention projects and community-oriented collaborations, partnerships and knowledge.

These activities and outputs are intended to reach the target population, youth ages 10-24, and stakeholders and intermediaries.

Immediate Outcomes:

The planning, organization and delivery of health promotion and prevention projects and community-oriented collaborations, partnerships and knowledge are expected to contribute to the following three immediate outcomes:

• The target population (youth) increases its awareness/understanding of healthy lifestyle choices & of illicit drugs & their negative consequences.
• Communities increase their awareness/knowledge of Health Promotion and Prevention resources to prevent illicit drug use among youth.
• Communities have increased access to Health Promotion and Prevention resources to prevent illicit drug use among youth.

Intermediate Outcomes:

The three immediate outcomes are expected to contribute to the following three intermediate outcomes:

• Youth acquire/improve their capacity (knowledge & skills) to avoid illicit drug use.
• Communities increase their uptake of Health Promotion and Prevention knowledge and resources to prevent illicit drug use among youth.
• Communities increase the engagement of community structures and networks in Health Promotion and Prevention efforts to prevent illicit drug use among youth.

Long-Term Outcome:

The immediate and intermediate outcomes are expected to lead to the following two long term outcomes:

• Youth reduce their risk-taking behaviors associated with illicit drug use.
• Communities improve their practices which, in turn, increase the effectiveness of Health Promotion and Prevention activities addressing illicit drug use among youth. Improved community practices also contribute to reducing risk-taking behaviours among youth.

The immediate and intermediate and long term outcomes are expected to lead to the ultimate outcome of a reduced demand for illicit drugs among youth.

2.3.2 Project activities and outputs

The ultimate outcome of DSCIF funded activities is to reduce the demand for illegal drugs. The activity areas, outputs, immediate, intermediate and long term outcomes to achieve this outcome were outlined in a Drug Strategy and Community Initiatives Fund Performance Measurement Strategy developed in 2010.^{Footnote 8}

The DSCIF program consists of two principal sets of activities:

• Program design and funding included program promotion and engagement with stakeholders, partnership development and knowledge exchange to facilitate the implementation of the DSCIF and achievement of program results, and support for projects, including funding as well as technical support that contributes to program results.
  • Program design and funding activities produced the following outputs:
    • tools and resources to manage the Program and its contribution funding process;
    • knowledge exchange sessions and evaluation support; and
    • partnerships.
• Project implementation and delivery includes the activities undertaken by funding recipients that are related to the planning, organization and delivery of heath promotion and prevention projects in accordance with the DSCIF funding guidelines. In addition to health promotion and prevention activities, the DSCIF project leads also participate in program-level activities such as knowledge exchange and evaluation.

As the literature indicates, multiple factors are associated with the likelihood of illicit drug use among youth. The risk of drug use involves the relationship among the number and type of risk factors and protective factors. Risk factors include youth's age, stage of development, living circumstances, relationships with family and school, peer associations and social context, periods of transition (changes in physical environment, social situation, and family situation) and the availability and access to drugs within the community. Protective factors, such as the ability to cope with stressful or adverse living circumstances, availability and access to family and community supports, may reduce the probability that those at risk become involved in illicit drug use.

Project implementation and delivery activities produced the following types of outputs:

• information, training or practice tools or resources (such as publications, multi-media products and toolkits);
• knowledge exchange mechanisms (such as conferences, workshops and symposia);
• skills transfer mechanisms (such as training workshops and seminars, training or practice tools);
• planning tools and infrastructure development processes (such as strategies, frameworks, networks and consortia); and
• community-oriented collaborations, partnerships and knowledge that influenced or directly strengthened efforts to prevent illicit drug use among youth.

2.3.3 Expected Outcomes

The implementation of the activities identified above corresponds with specific immediate, intermediate and longer-term outcomes for both youth and their communities. As the DSCIF target population encompasses populations at different ages, stages and levels of risk and covers a range of illicit drugs, the degree and nature of change may vary depending on the specific characteristics of the population targeted and type/nature of illicit drug use targeted. However, while the projects varied considerably, they share a common set of outcomes.

Expected Outcomes for Youth

The outcomes for youth are based on the assumption that the progression from awareness and understanding to knowledge and skills will lead to behaviour change. Projects funded through the DSCIF focussed on one or more of these elements, depending on the type of prevention or health promotion activity and the characteristics of the population being targeted.

• The following was the expected immediate outcome for youth:
  • Increased awareness/understanding of healthy lifestyle choices and of illicit drugs and their negative consequences. This outcome is important because increasing awareness and understanding of healthy lifestyle choices is expected to help prevent youth from using illicit drugs. Moreover, increased awareness and understanding of illicit drugs and their negative consequences (e.g., on personal health, relationships with family, school, work, and the legal consequences of using) will help youth to acquire factually grounded views about the harmful effects and consequences of using illicit drugs. This in turn is expected to influence their views about illicit drugs and help to inform any personal decisions regarding illicit drug use.
• The achievement of this immediate outcome was expected to lead to the following intermediate outcome:
  • Acquired/improved capacity (knowledge and skills development) to avoid illicit drug use. This outcome builds on the awareness and understanding that is achieved at the immediate level by focusing more specifically on developing or improving the targeted population's capacity (knowledge and skills) to avoid illicit drug use. Improved knowledge about helpful services and supports in the community and improved skills, including positive coping skills, and strategies to avoid or resist peer pressure to use, are expected to lead to improved decision-making and a decreased likelihood of substance use.
• The achievement of the intermediate outcome was expected to lead to the following long-term outcome for youth:
  • Reduced risk-taking behaviours associated with illicit drug use. This outcome is based on the premise that if youth have the capacity and skills to make good decisions, they will, in turn, change their behaviours with respect to drug use as well as other types of risky behaviours associated with drug use. For example, the DSCIF program may help to prevent youth from using illicit drugs at an early age or from using illicit drugs that may jeopardize their health and well-being or contribute to social and legal consequences.

Expected Outcomes for Communities

DSCIF is a community-based program with the objective of strengthening community responses to illicit drug issues. The theory of change underlying the expected community outcomes is based on the premise that increased access and awareness/knowledge of health promotion and prevention resources is expected to lead to increased use of these resources and improvements in community practice which will contribute to the reduction of the risk taking behaviours associated with illicit drug use among youth in these communities.

• The following are the expected immediate outcomes for the community:
  • Increased access to health promotion and prevention resources to prevent illicit drug use among youth. DSCIF program level knowledge exchange/learning sessions and dissemination activities are expected to increase the availability of project resources to communities. As well, by facilitating the sharing of resources across projects, the DSCIF program will help to increase access to resources that prevent illicit drug use among youth and increase awareness and understanding across projects and communities.
  • Increased awareness/knowledge of health promotion and prevention resources to prevent illicit drug use among youth. As the primary generators of DSCIF resources, project activities include undertaking or participating in knowledge dissemination and exchange activities. The types of resources produced may include resources to increase public awareness and understanding and improve program delivery practices. Empirical and practice-based knowledge that contributes to practice improvements may also be generated through project-specific research and or evaluation activity.

These outcomes are tied to the expected results as stated in the DSCIF Terms and Conditions of "increased availability of community based promotion and prevention initiatives to address substance use and abuse" and "greater awareness and availability of effective models of intervention."

• Increased awareness and access to health promotion and prevention resources is expected to contribute to the achievement of the following intermediate outcomes for communities:
  • Increased uptake of health promotion and prevention knowledge and resources to prevent illicit drug use among youth. Increased access to and awareness of health promotion and prevention knowledge and resources are expected to lead to increased application of this knowledge and resources and better prepare communities to address illicit drug use among youth.
  • Increased engagement of community structures and networks in Health Promotion and Prevention efforts to prevent illicit drug use among youth. Partnerships and community-based activities are essential to address the complexities and interrelationships and the many challenges associated with illicit drug use, as well as the varying needs and priorities across the country. Engaging new and existing community structures and networks in health promotion and prevention is expected to help entrench and support illicit drug prevention in the community.
• Together, the uptake and application of health promotion knowledge and resources and the engagement of the community is expected to contribute to the following long-term outcome:
  • Improvements to community practice that increase the effectiveness of health promotion and prevention activities addressing illicit drug use among youth. This outcome is based on the expectation that health promotion and prevention knowledge and resources will be applied to improve community practice and ultimately will reduce illicit drug use.

The achievement of youth and community outcomes were ultimately expected to contribute to a reduced demand for illicit drugs among youth.

The connection between the DSCIF program's activities and the expected outcomes is described in Section 2.3.1. The evaluation assessed the degree to which the defined outputs and outcomes were achieved over the evaluation timeframe.

2.4 Program Alignment and Resources

The DSCIF program is part of Health Canada's Program Alignment Architecture (PAA) under the Program Substance Use and Abuse (2.5) and the Sub-Program Controlled Substances (2.5.2).

The Program contributes to the achievement of Health Canada's Strategic Outcome 2.0 - Health risks and benefits associated with food, products, substances, and environmental factors are appropriately managed and communicated to Canadians.^{Footnote 9}

Overall, the Program had a budget of $56.6M over five years. Contribution funding represented $50 million. Within the broader scope of government funding for illicit drugs, between 2007-2008 and 2010-2011 funding for DSCIF made up 11.6% of the overall NADS planned budget of $405.1, and 12.8% of the actual spent funding.^{Footnote 10}

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The Public Health Agency and Health Canada Five-Year Evaluation Plan 2014-2015 to 2018-2019 identified this evaluation to be of small materiality and low risk.^{Footnote 11} Therefore, the evaluation was subject to a limited evaluation design.

The evaluation covered the period from April 1, 2008 to March 31, 2013, and encompassed all DSCIF activities currently managed within the SPB. These activities included program promotion, stakeholder engagement, partnership development and knowledge exchange. The evaluation also included an assessment of the performance of projects which received contribution funding to plan, organize, deliver and report on their health promotion and prevention activities. As projects for the 2013 CFP were approved in the last quarter of 2013-14, and were just being implemented at the time of reporting, an assessment of their performance was out of scope for this evaluation.

The evaluation issues were aligned with the Treasury Board of Canada Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 1. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, to identify and to develop lessons learned and to determine if there were any unintended consequences.

The Program focused its activities, and therefore its project level performance data collection, on three outcomes. For youth, the focus of data collection was on the immediate outcome to increase awareness/understanding of healthy lifestyle choices and illicit drugs and their negative consequences, and on the intermediate outcome to acquire/improve capacity (knowledge and skills) to avoid illicit drug use. For communities, the focus of performance measurement was on the intermediate outcome to increase engagement of community structures and networks in health promotion and prevention efforts to prevent illicit drug use among youth. As a result the evaluation has focused on these outcomes.

Performance data for the remaining outcomes was reported in project progress and evaluation reports. However, as each project developed their own evaluation approaches independently, reporting for these outcomes was not systematic. As a result, it is difficult to aggregate data across the projects to make an overall assessment as to the level of achievement for those DSCIF outcomes.

The Treasury Board of Canada Policy on Evaluation (2009) also guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the Evaluation Framework document^{Footnote 12} which detailed the evaluation strategy for the DSCIF program and provided consistency in the collection of data to support the evaluation.

Data for the evaluation were collected using various methods, which were: literature review, document and file review, financial data review, performance data review, key informant interviews (both internal and external), and case studies. More specific details on the data collection and analysis methods are provided in Appendix 1. Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. Table 4 outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

There is an on-going need to address illicit drug use by youth, in particular vulnerable segments of the youth population, to address gaps and emerging issues and to build capacity

There is ample evidence in the literature that the circumstances, such as the high rates of illicit drug use among youth, which led to the reorientation of the DSCIF under NADS in 2007, continue to exist.

National statistics from the 2012 Canadian Alcohol and Drug Use Monitoring Survey (CADUMS) and the Youth Smoking Survey (YSS) showed that although rates of illicit drug use have declined, the rates of illicit drug use among youth 15 to 24 years has remained higher than the rates reported by adults:

• According to 2012 CADUMS data, the prevalence of past year cannabis use by youth 15 to 24 years of age has declined since 2004 to 20.3% from 37.0% but was still almost two and a half times the rate of 8.4% for adults (those over 25 years). The prevalence among males remained twice as high as that of females (13.7% versus 7.0%, respectively).^{Footnote 13} The average age of initiation of the use of cannabis was 16.1 years.^{Footnote 14}
• The national 2010-11 YSS found that past year cannabis use was reported by 21% of students in grades 7 to 12 (typically ages 12 to 18 years).^{Footnote 15}
• According to data from a CCSA report on Student Alcohol and Drug Use, which examined the cumulative results of all available jurisdictional student surveys (ages 12 to 18), reported past year cannabis use increases from 3% to 8% of students in Grade 7 to 30 to 53% by grade 12.^{Footnote 16}
• Furthermore, based on children ages 11, 13, 15 who report having used marihuana in the last 12 months, Canadian youth are the top users of marihuana (28%) in the developed world according to a 2013 UNICEF Office of Research report.^{Footnote 17}

The CADUMS also reported that the prevalence of past-year use of at least one of five drugs excluding cannabis (cocaine, speed, ecstasy, hallucinogens (excluding salvia) or heroin) to be almost five times higher among youth aged 15 to 24 (6.3%) than reported by adults (1.2%).^{Footnote 18} The Youth Smoking Survey found that other than cannabis, the illicit drugs most commonly used by the youth in grades 7to 12 were ecstasy (4.5%), hallucinogens (4.4%) and cocaine (2.3%).^{Footnote 19}

The need to address the specific circumstances of sub-populations of youth is also supported by the literature that shows a higher risk of illicit drug use among vulnerable sub-populations of youth, including Aboriginal, street involved/homeless and sexual minority youth.

• Drug use is consistently reported to be higher among Aboriginal populations than the general population. According to the Northwest Territories Addictions Report (2010), the proportion of the Aboriginal population's cannabis use in the past 12 months was twice as high as the cannabis use among non-Aboriginals (approximately 25-30% versus 10-15%). The same study found that about 40-45% of youth (aged 15 to 24) living in the Northwest Territories used marihuana in the past year, which is higher than the national average of 20.3% reported by CADUMS 2012.^{Footnote 20} Another study, reported in the Canadian Medical Association Journal, of 605 Aboriginal youth in two urban centres, showed high rates of injection drug use. Young Aboriginal women were found to be twice as likely to inject drugs as men. The authors concluded that their findings highlight the need for culturally appropriate prevention and cessation policies and programs for this at-risk population.^{Footnote 21}
• Drug use has also been identified as a problem among street-involved youth. For example, a high rate of drug use among street youth was reported in a 2011 City of Ottawa study of street youth. The study found that 91% reported using non-injection drugs in the past year. 70% reported using non-injection drugs other than marihuana or non-medicinal prescription drugs, compared with 11% of Ottawa grade 9 to 12 students surveyed by the Ontario Student Drug Use and Health Survey (OSDUHS).^{Footnote 22} These rates are very similar to a 2006 study of street youth by the Public Health Agency of Canada which found that 94% of street youth between the ages of 15 and 24 years reported non-injection drug use (most commonly cannabis). More than one in five (23%) in the same age range reported injection drug use in their lifetime - most commonly cocaine (29%) and morphine (28%).^{Footnote 23} This same report noted that street youth are 11 times more likely to die of drug overdose and suicide.^{Footnote 24}
• High rates of drug use among the lesbian, gay, bi-sexual and transgendered (LGBT) youth population is also frequently reported in the literature. The Council on Drug Abuse (CODA) has reported that LGBT youth experience higher rates of cigarette, alcohol and marihuana use, as well as other illicit drugs including cocaine, methamphetamines and injection drugs.

Among the issues related to drug use and youth emerging in the literature, and also identified by some P/T and project interview respondents, were non-medical use of pharmaceuticals, drug-impaired driving and concurrent disorders such as mental illness and drug use:

• There is also a growing body of evidence that youth are abusing prescription drugs. Almost 1 in 20 (4.9%) youth indicated that they had abused a psychoactive pharmaceutical in 2012.^{Footnote 25} Rates among urban Aboriginal youth were even higher. Some 18.4% of Inuit youth aged 12 to 17, 11% of Aboriginal youth and 8.8% of Métis youth living in urban Canada, as compared with 5.6% of non-Aboriginal youth, self-report abuse of prescription drugs including sedatives, stimulants and pain relievers.^{Footnote 26} Prescription drug abuse has been raised as an area needing more research and prevention efforts. In 2013, the National Advisory Committee on Prescription Drug Misuse produced an overview of the scope of the prescription drug crisis in Canada and produced a roadmap for reducing the harms associated with these drugs which includes prevention.^{Footnote 27}
• Driving after drug use is also emerging as an issue among youth. For example, Ontario data for 2013 indicated that 9.7% of drivers in grade 10 through 12 (ages 16 to 18) with a G-class licence reported driving after using cannabis at least once during the past 12 months.^{Footnote 28} Similarly, a British Columbia survey reported that 4.8% of drivers age 16 to 18 tested positive for drugs. Cannabis was the most common drug type.^{Footnote 29}
• Finally, increasing evidence suggests those who use cannabis are more likely to develop dependence, use other illicit drugs, and develop psychotic symptoms, psychosis and emotional problems such as depression.^{Footnote 30,Footnote 31} Approximately half of all mental health problems begin by age 14. At the same time, alcohol and cannabis experimentation increases significantly between the ages of 12 and 18. The research suggests that the association between cannabis use and depression, conduct problems, tobacco smoking, excessive drinking and use of illicit drugs shows a harmful pattern of comorbidity that may lead ultimately to further negative outcomes. As such, early detection and intervention are crucial to preventing mental health and substance abuse issues from continuing into adulthood.^{Footnote 32}

There is a continued need for a community-based health promotion and prevention approach to address illicit drug use among youth

There was general consensus in the Canadian and international literature, and among stakeholders interviewed, of the need for ongoing prevention of youth substance abuse. The literature review confirmed that the community-based approach that has been taken by the DSCIF program is similar to approaches being taken to address drug use in the United States (U.S.) and internationally. Evidence-informed, collaborative approaches to prevention that are community-based, which engage key institutions such as schools and families, and engage communities, are recognized as ways to enhance and sustain positive youth outcomes. In Europe, for example, most of the prevention activity happens in the schools although it is also implemented in other settings as well.^{Footnote 33}

Although the literature review was unable to find recent information on the national distribution of substance abuse prevention programs for youth, the literature and stakeholders interviewed suggested that substance abuse prevention and intervention services for youth lack sufficient funding and that capacity varies across and within jurisdictions. For instance, the literature identified differences and gaps in the distribution of services between large urban, rural and remote settings. Funding tends to be concentrated in urban centres, along with the more highly trained staff. As a result, people living in large urban centres tend to be better served than those in small urban, rural and remote communities.^{Footnote 34}

The literature review also suggested specific areas to focus prevention efforts. For example, some reports suggest that given its status as the most commonly used drug, cannabis is an important issue for prevention work with young people.^{Footnote 35,Footnote 36} Another area is raising awareness about the harms associated with drug use. The prevalence of self-reported harms due to their own drug use was four times higher among youth aged 15 to 24 years (5.5%) than adults aged 25 years and older (1.4%).^{Footnote 37} Perceptions of risk and awareness of the harms and consequences of drug use have been found to influence drug use. This points to a need for effective prevention interventions that increase levels of awareness and knowledge about the harms associated with illicit drug use, and to shift attitudes and behaviours toward healthier directions.

There was also consensus among stakeholders surveyed for the 2013 evaluation of the NADS on the need for drug prevention programs for youth. Stakeholders familiar with the Prevention Action Plan were asked to rate the need for programming that raises awareness of the harmful effects of illicit drug use, on a scale of 1 to 5 where 1 is no need at all, 3 is somewhat of a need and 5 is a major need. They provided an average rating of 4.9 (n=13) noting that there is a particular need to increase awareness about marihuana, drugs and driving, and skills to avoid drug use. These stakeholders explained that there is a need to have prevention programming for not only at-risk youth, but for communities in rural areas and areas where there is higher prevalence of use. As well, external stakeholders (e.g., national, provincial, municipal representatives and academics/experts, n=9) provided an average rating of 4.4 and explained that there is a need for strong, consistent messaging with respect to illicit drug use.^{Footnote 38}

Similarly, funding recipients and the majority of P/T respondents interviewed for the evaluation agreed that the objectives of DSCIF are still relevant and that there is a continued need for community-based prevention programs. P/Ts provide prevention programming; however P/T strategies were developed independently, so priorities, policies, funding and delivery models vary significantly. Reinforcing the need to support communities, interview respondents pointed to the varying levels of capacity for age appropriate services for young people and their families to address illicit drug use across Canadian jurisdictions and within jurisdictions (e.g., rural and northern areas).^{Footnote 39} Recipient respondents stressed the importance of DSCIF funding for their projects. All respondents indicated that without DSCIF funding they would not have been able to undertake the project.

The level of demand for DSCIF funding exceeded the availability of funds, providing further evidence that there is a continued need for funding to address illicit drug use among youth. There were 158 proposals received for the 2013 Call for Proposal (CFP); 34 proposals were approved. The 2009 CFP received 183 proposals and 37 were approved and the 2007 CFP received 209 proposals and 65 projects were approved.

Finally, further support for drug prevention programs for youth comes from the literature which suggests that prevention programs that target the behaviours that lead to illicit drug use can potentially forestall the development of these behaviours and provides an opportunity to not only have a positive influence on the future development of youth as individuals, but also to reduce the impact of substance abuse on society as a whole.^{Footnote 40} Prevention has been shown to reduce upstream cost associated with a range of legal, social and health costs to individuals and to society incurred by drug use and dependence.^{Footnote 41}

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Health promotion and prevention activities directed at illicit drug use among youth remain relevant in relation to Government of Canada priorities and Health Canada's strategic outcomes and objectives.

The document review demonstrated that the objectives and expected outcomes of the DSCIF supported Government of Canada priorities, as well as the achievement of Health Canada's Strategic Outcome #2: Health risks and benefits associated with food, products, substances, and environmental factors are appropriately managed and communicated to Canadians.^{Footnote 42}

The program also aligns with departmental priorities and objectives related to drugs, youth and at-risk populations.

4.2.1 Government of Canada

The federal government has identified illicit drug use and youth as priorities. A commitment to address illicit drug use and its consequences, particularly among youth, was identified as a priority in the 2007 Speech from the Throne in which the federal government indicated its intention to invest in preventing illicit drug use among youth through its NADS − "It will help families and local communities in steering vulnerable youth away from a life of drugs and crime, and the Anti-Drug Strategy will help to treat those suffering from drug addiction."^{Footnote 43} The Government of Canada's continued commitment to addressing drug use is reflected by the expansion of NADS in Budget 2014 to also address the growing problem of prescription drug abuse.^{Footnote 44}

The DSCIF was realigned in 2007 to support the objectives of the federal government's NADS Prevention Action Plan: to prevent youth from using illicit drugs by enhancing their awareness and understanding of the harmful social and health effects of illicit drug use and to develop and implement community-based interventions and initiatives to prevent illicit drug use.^{Footnote 45}

Further establishing its relevance to the NADS, the DSCIF program shares a number of immediate, intermediate and longer term outcomes with the NADS, including:

• increased awareness and understanding of illicit drugs and their negative consequences;
• enhanced knowledge in communities to address illicit drug use and its negative consequences;
• enhanced capacity among targeted populations to make informed decisions about illicit drug use;
• strengthened community responses to illicit drug issues in targeted areas; and
• reduced risk-taking behaviours associated with illicit drug use among youth.

4.2.2 Health Canada

The document review found that the DSCIF program aligns with Health Canada's Strategic Outcome #2, and with departmental priorities and objectives. The objectives of the DSCIF program to raise awareness and understanding among Canadians of the harmful social and health effects of illicit drug use supports the achievement of Health Canada's Strategic Outcome #2 which focuses on ensuring that health risks, including those associated with substances are appropriately managed and communicated to Canadians.

Community-based programming is a dominant theme that has seen continued commitment as reflected by recent public statements by federal Ministers. For example:

• At the National Addictions Awareness Week - November 18-24, 2013: "The Government of Canada is [also] continuing its efforts under the National Anti-Drug Strategy ... The Drug Strategy Community Initiatives Fund supports non-governmental and community organizations, as well as municipal, provincial and territorial governments to help prevent drug use among youth. Activities also promote public awareness of substance abuse issues through health promotion and prevention projects."^{Footnote 46}
• At the International Day against Drug Abuse and Illicit Trafficking - June 26, 2013: "Our Government is continuing its efforts under the National Anti-Drug Strategy (NADS)... Our Government is working closely with other levels of government, community groups, non-governmental organizations and international partners to ensure the Strategy helps make Canada's communities safer and healthier."^{Footnote 47}

Furthermore, the DSCIF program supports Health Canada in achieving its mandate to Canadians, to maintain and improve their health and contribute to strengthening Canada's record as a country with one of the healthiest populations in the world.^{Footnote 48} It supports five of six Health Canada's objectives:^{Footnote 49}

• DSCIF`s focus on health promotion and prevention activities to prevent illicit drug use are aligned with Health Canada objectives to:
  • prevent and reduce the risks to individual health and the overall environment;
  • promote healthier lifestyles; and
  • integrate renewal of the health care system with longer term plans in the areas of prevention, health promotion and protection.
• The DSCIF`s activities to address illicit drug use among vulnerable sub-populations of youth contributed to the objective to reduce health inequalities in Canadian society.
• The DSCIF`s production of drug prevention tools and resources, and awareness and capacity building activities directed at youth and communities contributed to the objective to provide health information to help Canadians make informed decisions.

Finally, the objectives of the DSCIF program, also directly align with the objective of the Substance Use and Abuse Program which is to manage risks to the health of Canadians associated with the use of tobacco products, and the illicit use of controlled substances and precursor chemicals.

4.2.3 Alignment with P/T Priorities

The DSCIF program aligns with the policy direction of the NADS with a focus on illicit drugs. P/T strategies to address substance abuse typically address a broad spectrum of substance use and are not exclusively focussed on illicit drugs. Almost all P/T interview respondents reported the use of alcohol as a more significant problem in their province. Some respondents noted that alcohol is generally a precursor to drug use and suggested that DSCIF would be more relevant and better serve Canadian youth if its scope extended beyond illicit drugs to include substances more generally, including alcohol. As noted by one respondent, "There is a need for programs with a broad focus that address alcohol, drug use and mental health; alcohol should be more front and centre. National statistics show it is the drug of choice."

The formative evaluation of DSCIF Call for Proposals (2010) and project progress reports also identified recipient concerns about the focus on illicit drugs. The evaluation report noted that many project recipients were critical of the focus on illicit drugs and considered the focus to be too narrow, in particular in relation to alcohol and/or prescription drugs.^{Footnote 50}

Evidence from project summaries also identified the challenges faced by funding recipients as a result of the reorientation of DSCIF to align with the NADS. Some projects reported that to receive DSCIF funding they had to reorient their health promotion and prevention efforts from an approach that had addressed a broader range of substances, both legal and illegal, and/or which used a harm reduction approach to working with youth.

4.2.4 Policy direction and priority setting for DSCIF

There are opportunities to improve internal processes for collaboration on policy direction and priority-setting for the DSCIF program

The evaluation team was unable to determine what collaborative processes exist to ensure that the appropriate departmental and external stakeholders are engaged at key milestones in the program lifecycle, including in policy discussions about program direction, when setting priorities for the CFPs, in the CFP review process and in discussions on lessons learned.

DSCIF staff identified the NADS as setting the overall policy direction for the DSCIF and identified the HECSB as the OPI with respect to the NADS. HECSB staff noted the importance of input from program staff when determining substance abuse policy direction and setting research priorities. However, it was unclear if any formal mechanisms exist to ensure that both policy and program stakeholders are routinely engaged in establishing the departmental policy focus for illicit drug use prevention.

Staff attributed this to multiple reorganizations, and staff turnover and reductions. In 2009 the DSCIF program moved out of the HECSB to the RAPB. The policy function remained with the HECSB and the management and administration of the contribution agreements moved to the RAPB. In 2012, the management and administration of the contribution agreements moved to the SPB. Staff from both Branches indicated that the reorganizations eroded routine communications mechanisms resulting in less collaboration between the two areas.

The need for better collaboration and communications among internal policy and program stakeholders also became apparent in discussions with staff about the process for setting priorities for the 2013 CFP. Again, roles and responsibilities and processes for engaging internal stakeholders at various key points in the program lifecycle, including in setting the CFP priorities, were unclear. The extent to which the DSCIF program fed lessons learned from project implementation into the policy process was also not clear. No routine formal mechanisms appear to be in place for these exchanges.

The extent to which external stakeholders, in particular the P/Ts, were engaged in setting the priorities for the 2013 CFP was also less well-defined, compared to the 2007 and 2009 CFP processes. For the 2007 CFP, the Program had taken a structured and planned consultation approach with regional stakeholders to establish priorities for the regions. Evidence from the document review and key informant interviews with staff and P/Ts indicated that under the regional delivery model, regional staff held comprehensive stakeholder consultations to assess and document regional needs and gaps. As a result, three regions (Alberta, Atlantic, Quebec) formulated regional priorities, within the broader priority areas. A less formal process occurred for the 2009 CFP. Program managers conducted an informal gap analysis by reviewing the projects funded through the 2007 CFP and identifying sectors and geographic areas that were missing, to target in the 2009 CFP.

Evidence of similar consultations with P/Ts for the 2013 CFP priorities was not found. P/T representatives indicated that while they were involved in reviewing proposals, they were not involved with setting the funding priorities for the 2013 CFP. DSCIF staff indicated that P/T involvement in the 2013 priority setting was through the F/P/T Liaison Committee on Problematic Substance Use. However, the evaluation team heard from staff and also from a member of the Committee that this committee is focused primarily on treatment issues and has been much less engaged with issues related to prevention.

DSCIF staff also noted that the transition from a regional to a national delivery model, accompanied by staff reductions, impacted on the level of engagement with funding recipients and P/Ts. Under the regional delivery model, former regional staff reported that they had well established relationships and networks with their regional stakeholders and engaged P/Ts and regional stakeholders in identifying gaps and setting priorities. The document review and interviews with funding recipients confirmed that recipients also received considerable support from regional staff during the implementation of their projects. Some National Office staff indicated that under the national delivery model, they have experienced challenges establishing and maintaining relationships with stakeholders and noted that developing such relationships takes time and can be more challenging to do under the national delivery model.

Communications with stakeholders is nevertheless an important component in assessing regional needs and gaps. The 2010 DSCIF Terms and Conditions emphasizes that ongoing collaboration with non-government organizations and partners at the federal, provincial and territorial levels helps to establish priorities and to ensure all parties invest their resources where they have the most impact, as well as to ensure regional needs are met within the context of the overall DSCIF objectives.^{Footnote 51}

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

There is still a role for the federal government and Health Canada in preventing illicit drug use among youth

The federal role is grounded in its authorities under the Constitution Act (1867) and in section 4 of the Department of Health Act (1996). The Department of Health Act offers a broad public health mandate and sets out the Minister's powers, duties and functions, including:

• the promotion and preservation of the physical, mental and social well-being of the people of Canada;
• the protection of the people of Canada against risks to health and the spreading of diseases;
• investigation and research into public health, including the monitoring of diseases;
• the collection, analysis, interpretation, publication and distribution of information relating to public health; and
• the cooperation with provincial authorities with a view to the coordination of efforts.^{Footnote 52}

However, under the Constitution Act, education and prevention are primarily the responsibility of P/T governments.

The federal government has established a role in addressing illicit drug issues at the broad policy level through the NADS and, more specifically, has defined a role for Health Canada as the lead for the Prevention Action Plan of the NADS. As a partner department under the NADS, the DSCIF is the primary mechanism available to Health Canada to support community-based organizations in preventing illicit drug use among youth.

Departmental documents indicate that Health Canada has assumed various roles to achieve its mandate to help Canadians to maintain and improve their health, including regulator, service provider, catalyst for innovation, funder and information provider in Canada's health system.^{Footnote 53} With respect to the DSCIF, Health Canada's role has primarily been funder. DSCIF has supported the role of the P/Ts using the policy lever of contribution funding whereby the federal government has provided funding support to community-based and national organizations to pursue the policy objectives of the NADS.^{Footnote 54} Grants and contribution programs are widely recognized as important instruments for achieving federal government results and delivering on its responsibilities to Canadians.^{Footnote 55,Footnote 56} The contribution funding mechanism has enabled Health Canada to engage communities in preventing illicit drug use by mobilizing a wide range of community-based organizations to carry out health promotion and prevention activities that contribute to the Government of Canada and departmental objective to prevent illicit drug use among youth.

The document review and key informant interviews also provided support for a federal and Health Canada role in addressing illicit drug use among youth. The evaluation of the NADS found agreement among the majority of P/T respondents that the federal government should have a role in the prevention of illicit drug use among youth, since the sheer magnitude of the illicit drug problem demands sizeable resources and because the problem is not contained within local, provincial/territorial, or regional boundaries.^{Footnote 57}

Analysis of feedback from P/T representatives interviewed for this evaluation supported the findings of the NADS evaluation. The majority of P/T respondents see a role for the federal government in providing funding support particularly to jurisdictions and organizations that lack the capacity or resources for prevention activities.

In terms of the role that stakeholders expect Health Canada to play in the prevention of illicit drug use among youth, the majority of P/T interview respondents expressed the view that Health Canada could assume a greater role in knowledge translation and exchange through the pan-Canadian dissemination of knowledge (synthesis and analysis of lessons learned, evidence based and promising practices) and products and resources (tools, guidelines, standards etc.) and by encouraging and facilitating collaboration and coordination among stakeholders. These expectations align with Health Canada's role as an information provider in Canada's health system.

Another theme that emerged through internal and external interviews, and related to knowledge translation and exchange, was the view that Health Canada should fund innovative projects and disseminate findings and lessons learned from these projects so that others can use the information and not have to reinvent the wheel. This view aligns with Health Canada's role as a catalyst for innovation. To date, the focus of DSCIF funding has not been exclusively on innovative practices although the CFP criteria do not exclude the funding of innovative projects. Funding criteria therefore have been based on community needs and the needs of vulnerable populations rather than on innovation.

A key strength of the DSCIF program is its flexibility in terms of being able to fund a wide range of community-based projects. The funding of community-based health promotion and prevention are not typically the responsibility of the federal government. However, to address the complexity and many challenges associated with illicit drug use and the differing levels of capacity across the country to address the need requires a concerted effort by all levels of government. DSCIF funding to a wide range of community-based organizations is the primary mechanism whereby the federal government can advance its policy objectives and ensure a focus on the prevention of illicit drug use among youth while supporting the role of the P/Ts. It also provides funding to build awareness and capacity at the community level and the opportunity to support innovative projects, the results of which can be applied more broadly. However, as the environment has recently changed for the DSCIF, an opportunity exists to examine the direction and scope of the new Program going forward to more fully align with the federal role.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes for Youth (Effectiveness)

The expected ultimate outcomes for youth (i.e., reduced demand for illicit drugs) is based on the premise of a progression of change starting with increased awareness, knowledge and understanding of illicit drugs and their negative consequences, to acquiring the capacity and skills to avoid illicit drugs and, finally, behavioural change in the form of reduced risk taking behaviours.

4.4.1 To what extent have the immediate outcomes for youth been achieved?

The DSCIF contributed to an increased awareness and understanding, amongst the targeted youth, of healthy lifestyle choices and of illicit drugs and their negative consequences

Youth awareness was the primary focus for 39 of the 103 DSCIF projects funded through the 2007 and 2009 CFPs, with 36 projects in total reporting on this outcome. These projects tended to target the general youth population and were concerned about increasing awareness of illicit drugs, their use and their harmful effects. Awareness of the negative consequences of illicit drug use on personal health, relationships with family, school and work, as well as the legal consequences of using, were other areas that projects addressed. This outcome was based on the assumption that increasing awareness and understanding of illicit drugs and their negative consequences would help youth to acquire informed views of the harmful effects of using illicit drugs.

A variety of approaches were employed by projects to achieve their goals. The main approaches were peer education, mentoring, school curriculum supplementation and the use of social marketing or social media tools and techniques. In one classroom-based project focussing on drug impaired driving, students developed the products to share with others in their class. This use of peer education reinforced the messaging and was felt by the students to be more informative. Other projects targeted certain subpopulations of youth and used train the trainer approaches to enable youth leadership, peer-to-peer teaching and coaching and promotion of civic engagement.

From the 2007 CFP, 27 project leads reported on increased awareness and understanding. About two thirds perceived that their projects had contributed toward increasing awareness and understanding of healthy lifestyle choices (63%) and of illicit drugs and their effects (67%). In one project that focussed on both parents and youth, there was a substantial increase in awareness of parents on signs of drug use and risk factors that contribute to drug use. At baseline, 27% of parents were aware or very aware of the signs of drug use and this increased to 82% at the three month follow-up. Awareness of the risk factors that contributed to drug use increased from 28% at baseline to 83% three months post-intervention. Another project reported that youth visiting their website identified a change in awareness and understanding. The most frequent changes were increased awareness regarding short and long term effects of drug use, how to deal with drug and alcohol addictions and when to get help, and the different names by which drugs are known.

Nine of the eleven 2009 CFP projects that focused on this outcome administered the Health Canada Survey: Young People and Illegal Drugs 1 to youth before and after they participated in the intervention. This survey assessed the youth's awareness and understanding of healthy lifestyle choices and of illicit drugs and their negative consequences.

Overall, youth awareness of where to get reliable information on drugs increased after the project interventions. Initial levels were quite high (over 66%) but the increase to 77% post intervention was still statistically significant. As well, there was a statistically significant positive change in the level of awareness of healthy lifestyle choices. Youth were able to identify the importance of various healthy choices, such as eating healthy, getting enough sleep and avoiding drugs.

Projects also had an impact on the overall awareness and understanding of illicit drugs. Youth generally felt that they had a high or very high level of awareness of illicit drugs and their consequences pre-intervention (61%). Post project intervention, there was a statistically significant increase to 65%. This result was mainly attributable to the 13 to 15 age group. When analyzed by gender, only males had a significant increase in level of awareness of illegal drugs and their consequences.

When knowledge about the effects of specific illegal drugs was looked at, it was found that at baseline youth reported knowing most about the effects of cannabis, followed by cocaine, and crack. Youth knowledge about the effects of specific drugs increased significantly (between 6.5% and 14.2 %, depending on the drug) post project intervention for all but two drugs, GHB (ecstasy) and Ketamine.

Youth awareness and understanding of the effects of illicit drug use on personal health, relationships, school or work performance, crime and relationships was not changed by the projects as it was already high. In general, over 85% of the respondents could see that drug use had a medium to large effect on health and relationships. Awareness and understanding of the impacts of drug use on community was lower (74%), but there was a increase after project programming to 78%. Youth were more readily able to identify personal behaviour choices that would be impacted by drug use after the project interventions.

The survey also examined the awareness and understanding of factors that influence decisions to use or not use drugs and potential problems that can affect people who use illegal drugs. Post project results show that youth significantly increased their awareness of these factors. The importance of family and community connections, including school, was seen as a protective factor. Increasing protective factors to help youth avoid illicit drug use is seen as an important element in the program theory of change. If youth are provided with the necessary resilience and skills there will be a reduction in risk taking behaviours associated with illicit drug use. Connections within the community are key resilience factors. In one of the case study projects, participants noted that as a result of participating in the project they felt more engaged with their school and were more comfortable seeking help from school officials. These participants also reported that they were now more likely to resist peer pressure and avoid situations where they may be exposed to illicit drugs.

4.4.2 To what extent have the intermediate outcomes for youth been achieved?

Youth who participated in DSCIF projects were found to have acquired or improved their capacity (knowledge and skills) to avoid illicit drug use

There were 49 projects funded through the 2007 and 2009 CFPs with a primary focus to help youth acquire or improve their capacity to avoid illicit drug use. Forty-two projects reported on this outcome.

The most common approach to achieve this outcome was to use a peer leadership model. Projects selected youth with leadership skills and trained them to be peer teachers of other youth at risk of harmful substance use. Several projects used similar approaches targeted at specific subpopulations of vulnerable youth such as girls and young women, aboriginal youth in both rural and urban settings, homeless and street involved youth, youth living in large social housing projects, youth with concurrent disorders (addiction and mental health) and youth who identified as LGBT.

The expected outcomes of project activities were acquisition of knowledge and skills to avoid drug use, including coping, avoidance and resistance strategies and skills. Another important component of building capacity was building participant resilience, including self-efficacy and awareness of access to supports in the community.

Project leads for 29 projects from the 2007 CFPs reported on one or more of the indicators related to this outcome:

• 69% perceived an increased level of knowledge about how to avoid illicit drugs among youth participating in their programs. The positive effects they reported were increased confidence, awareness of triggers to substance use, development of skills to respond to peer pressure, increased ability to communicate, greater comfort to refuse drugs and leadership skills. A few projects reported no significant change in knowledge or skills.
• 83% perceived positive changes in the level and nature of coping, avoidance and resistance skills such as, overall increased coping skills, new skills in relationship building, development of supportive interactions, positive attitude shifts, increased skills in facilitation techniques, greater ability to engage in prevention and educational discussions, and improvements in family change variables, family functioning, parenting communications and organization.
• 90.5% perceived an increase in the level and nature of participant resilience, including self-efficacy and access to support. Project leads reported that participants improved their relationships, gained confidence, received positive community support, increased their access to resources and increased their awareness of where to go for help, and enhanced their social skills.
• 78.3% perceived a change in intention to use, including first use, frequency and nature of use. Project leads reported qualitative statements of resolve made by participants not to use or to reduce use of illicit drugs.

Project leads from the 2009 CFP who had as their primary outcome to acquire/improve capacity (knowledge and skills) to avoid illicit drug use administered the Health Canada Survey: Young People and Illegal Drugs 2 to youth before and after the intervention. Three indicators were used to measure this outcome: level and nature of knowledge about how to avoid illicit drug use, level and nature of skills - coping, avoidance and resistance and level and nature of participant resilience -, self efficacy and access to support.

Project participants were asked about the likelihood that they would use one of several different strategies to avoid drug use in certain situations and about their ability to avoid/resist drug use under specific situations. Survey results showed statistically significant positive findings from baseline to post project intervention suggesting an increase in the likelihood that youth who participated in the projects will use various strategies to avoid or resist drug use. Becoming educated about drugs showed the greatest change, from 79% at baseline to 88% post intervention.

Project participants were asked a number of questions around the intention to use marihuana and illicit drugs other than marihuana. Survey results showed a statistically significant decrease from baseline to post project intervention in the following:

• likelihood to try or regularly use marihuana in the next 12 months. For example the number of respondents who indicated that they were very unlikely or unlikely to regularly use marihuana increased from 84.4% at baseline to 92.7% post intervention.
• likelihood to try or regularly use other illicit drugs in the next 12 months. At baseline, a large number of respondents already responded that they were very unlikely to regularly use other illegal drugs (92.7%). The percentage post-intervention increased marginally to 94.1%.

Age and gender differences were found for the intention to try or regularly use marihuana. Survey results showed the following statistically significant results:

• decrease in the intention to try marihuana for the 13 to 15 and 16 to 18 age groups but no significant change for the 10 to 12 and the 19 years and older age group.
• decrease in the intention to regularly use marihuana among 10 to 12, 13 to 15 and 16 to 18 age groups. There were an insufficient number of respondents in the 19 and older age group.
• decrease in the intention to try marihuana for females but not for males
• decrease in the intention to regularly use marihuana among males but not females. The intention to use marihuana was already low at baseline for females.

These survey results showing age and gender differences, along with CADUMS data which showed the average age of onset of marihuana use to be 16.1 years^{Footnote 58}, provide useful information for future planning and delivery of prevention programs for youth. For instance, these findings underscore the importance of timing in preventing drug use and suggest that different prevention goals may be more appropriately targeted at those in junior high or their early high school years instead of in their final year of high school.

Youth were asked about their sources for information about illegal drugs, the people with whom they were comfortable talking about drugs, and the extent to which they have positive relationships. They were also asked a series of questions to measure self-esteem, leadership skills, and communication skills. The survey results showed:

• The most frequently cited sources of information at baseline were the internet, parents, friends and school. Post intervention, other family member increased by 10 percentage points, and school increased by 5.5 percentage points. Brochures and posters also increased by almost 10 percentage points, perhaps as a result of many projects producing their own promotion and educational material.
• There was no significant change between baseline and post project intervention scores in the following areas: developing positive relationships, self-esteem, leadership skills or communications skills.

In one of the case studies the youth participants practiced various ways to avoid or resist illicit drugs. The 4 Ds (Define, Decide, Do or Depart) of Friendly Refusal were taught, and reinforced through skits and presentations to younger students. Participants were able to provide examples of how they used these techniques to avoid situations where illicit drugs may be present and to resist pressure to try illicit drugs.

4.4.3 To what extent have the longer term outcomes for youth been achieved?

While the document review found specific examples reported by projects of reduced risk taking behaviour among youth, the overall extent to which the DSCIF program has contributed to this outcome cannot be assessed at this time

This outcome is based on the premise that improving the target population's knowledge of how to avoid drug use and providing them with the capacity and skills for avoiding drug use, including coping, avoidance and resistance strategies and skills, will lead to improved decision-making and a decreased likelihood of illicit drug use.

The evaluation cannot make an overall assessment of the achievement of this outcome because the focus of performance data collection was on the immediate and intermediate outcomes. Data from project progress and evaluation reports did provide some examples of behaviour change; however, results cannot be generalized.

Positive changes associated with risk taking behaviour among youth participating in DSCIF projects include:

• A reduction in frequency of use of illicit drugs. One project found that all of their youth participants decreased or ceased using cannabis, as a result of involvement with the project.
• Improvements in class attendance and involvement during the project. As some projects were school based, participants increased school attendance to continue involvement with the project activities.

4.5 Performance: Issue #4 - Achievement of Expected Outcomes for Communities (Effectiveness)

The DSCIF program's outcomes for communities are generally aligned with the following stages along the knowledge translation continuum:

• access to health promotion and prevention resources;
• awareness and knowledge of health promotion and prevention resources;
• uptake of health promotion and prevention resources;
• engagement of community structures and networks; and
• improvements to community practices.

4.5.1 To what extent have the immediate outcomes for communities been achieved?

Two related outcomes focus on immediate changes within the community as a result of knowledge translation and exchange activities: access to health promotion and prevention resources to prevent illicit drug use among youth and awareness/knowledge of health promotion and prevention resources to prevent illicit drug use among youth.

There is evidence that funded projects increased access to health promotion and prevention resources to prevent illicit drug use among youth within their communities

Projects developed a variety of health promotion and prevention resources. All projects produced one or more products and/or resources to increase public awareness and understanding of illicit drugs or to improve program delivery. As illustrated in Table 5, among the products produced by the projects were publications (e.g., pamphlets, brochures, fact sheets, posters, booklets and reports, promotional materials), websites, multi-media products (e.g., CDs, DVDs, videos), and toolkits.

In addition projects produced planning tools/infrastructure development (e.g., strategies, frameworks, networks and consortia) or organized knowledge exchange mechanisms such as conferences, workshops or symposia, or skills transfer mechanisms including training workshops, seminars, and curriculum resources.

While projects disseminated their resources to stakeholders within their communities and networks, further dissemination of knowledge and resources regionally or nationally did not happen for most projects

As generators of prevention and health promotion knowledge products and resources, the DSCIF funded projects were expected to increase access to and awareness/knowledge of their health promotion and prevention resources through dissemination activities. While projects reported on their dissemination activities, the extent to which the DSCIF projects increased awareness/knowledge of DSCIF resources was difficult to determine.

The 2007 and 2009 CFPs included a requirement that project proposals describe the plan for sharing information and materials resulting from the project.^{Footnote 59} Evidence from the document review and verified through interviews with project representatives indicated that DSCIF projects proactively disseminated and shared the products, resources and learnings with local partners and through their networks. Materials developed by DSCIF projects have been used in classroom settings, and by youth workers. In some instances, resources have been incorporated into strategic plans and community programming.

With the exception of national organizations and some of the larger organizations, evidence indicated that funding recipients generally did not disseminate their DSCIF products and resources more broadly (i.e., outside their communities or networks). Ten of the funded projects did participate in a NADS knowledge event in 2012. However, this event has not been repeated. Thirty-two projects reported having a website where they posted materials. Many of these websites were not maintained after DSCIF funding ended. There also was not consistent collection of web statistics that would allow a measure of the reach of these resources.

The prevention standards developed by the Canadian Centre for Substance Abuse (CCSA) are an exception. In the first month after release in 2010, the Community Based Standards document was downloaded 478 times (392 English downloads and 86 French downloads). As well, 150 hard copies of the document were requested.

Interviews demonstrated that project representatives were generally unaware of the resources produced by other DSCIF projects, although some were aware of other DSCIF funded projects in their province. This awareness was a result of participation in regional 'showcases' organized by regional staff under the regional program delivery model. These showcases took place at the beginning of the project funding and brought project leads together to discuss their projects. The showcases were viewed positively by funding recipients as an opportunity for information sharing and awareness raising. Informal information sharing at the regional level also facilitated the development of networks within the region and increased awareness and access to health promotion and prevention resources. The lack of knowledge about resources produced by other DSCIF funded projects may have lead to the production of similar resources, and a duplication of efforts on the parts of projects. Increased dissemination may have prevented the 'reinvention of the wheel' and allowed projects to spend their resources on other activities.

Many P/T respondents were also unaware of the projects funded through the DSCIF operating in their province as well as the resources produced by these projects. Again, the CCSA's Community Based Standards was an exception, as some of the P/T representatives were familiar with this resource. In addition, these Standards were a foundational piece used by the United Nations Office on Drugs and Crime to develop the International Standards, released in March 2013.

Health Canada's role in knowledge translation and exchange is not well defined

At the Program level, the DSCIF logic model identified knowledge exchange as one of the Program level activities. Some knowledge exchange activities have occurred at the Program level. For example, a presentation on findings and lessons learned was made at the 2013 Issues of Substance Conference. DSCIF staff have also engaged in knowledge sharing regarding their approach to performance data collection. Presentations on the approach to collecting consistent and comparable outcome data at the project level have been made in a variety of external settings: other government department information sharing on performance measurement, Performance and Planning Exchange (PPX) conference, and the Canadian Evaluation Society 2014 Annual Conference, as well as within Health Canada.

However, the document review and interviews with project representatives revealed that the Program has assumed a limited role in knowledge translation (analysis or synthesis) and pan-Canadian dissemination of DSCIF knowledge products and resources to interested stakeholders. For example, project representatives and P/T respondents were interested in the results and lessons learned from DSCIF projects. Although the Program has been proactive in sharing the lessons learned and the report on project outcome results with newly funded projects, many completed projects were unaware of the results of the outcome surveys which their participants had completed as part of the project's evaluation requirements.^{Footnote iv}

Knowledge exchange and the importance of drawing a link between project results and knowledge transfer in order to contribute to broader research, policy, and program development was also identified as a weakness in the evaluation of the NADS, as well as in Health Canada's internal lessons learned report. The NADS evaluation concluded that efforts should be made to coordinate and strengthen knowledge transfer activities across all partner departments. The evaluation noted the importance of recognizing that the eventual impact of projects is dependent on the ability to transfer that knowledge to other parties and for them to act on it.^{Footnote 60} The evaluation recommended development of a mechanism for disseminating knowledge developed through the prevention and treatment components of the NADS.^{Footnote 61} As a result of the 2012 NADS evaluation, DSCIF and other programs under the Prevention and Treatment Action Plans have made efforts to coordinate and strengthen knowledge transfer activities. A Knowledge Exchange Working Group has been developed, and a Knowledge Development translation and exchange strategy has been drafted.

Similarly, an internal Health Canada lessons learned report (2013) concluded that there was a need for a knowledge translation and exchange role for the DSCIF program. The report concluded that "DSCIF should play a greater role in knowledge translation and exchange. DSCIF should embed knowledge translation and exchange into their overarching program design. This would include developing a knowledge strategy and allocating sufficient resources to complete related tasks."^{Footnote 62}

The DSCIF could have a broader impact if the Program strengthened its role as an information provider, by engaging in knowledge translation (analyzing and synthesizing project information and performance results) and exchange activities and facilitating pan-Canadian access to the knowledge products and resources produced by DSCIF funded projects. Relying exclusively on projects to disseminate the knowledge products and resources that they produce limits the reach of the knowledge products and resources. Small organizations have limited capacity to disseminate knowledge and resources broadly. Assuming a more significant role in knowledge translation and Pan-Canadian knowledge exchange to make the knowledge products and resources generated through DSCIF projects more accessible, would extend the Program reach and influence and enable stakeholders to act on the information.

However, the Program first needs to define its role in knowledge translation and exchange, as it is currently not clear. Interviews with P/T and project representatives stressed that knowledge exchange on promising evidence based practices was important, and many identified this as a key role that could be played by Health Canada. Others, however, suggested that Health Canada may lack the capacity to take on this role and another organization would be better suited to assume the role.

Within Health Canada, staff also had differing views as to Health Canada's role regarding knowledge translation and exchange. Although many of the Program staff interviewed recognized the need to have a role in synthesizing findings and identifying promising evidence-based practices, they indicated that currently there is limited capacity within the Program to embark on these activities. Some staff felt that projects had the responsibility to disseminate their resources and information. The document review confirmed this view, as funding recipients were asked to report their dissemination plans in their progress reporting. Guidance on dissemination plans did not instruct projects to attempt national level knowledge exchange.

4.5.2 To what extent have the intermediate outcomes for communities been achieved?

At the community level, there is evidence of uptake of health promotion and prevention resources generated through DSCIF projects

The Program logic model focuses on the uptake of health promotion and prevention knowledge and resources at the community level. The increased application of health promotion and prevention knowledge/resources is expected to better prepare communities to address illicit drug use among youth.

The document review of project progress and evaluation reports found evidence that, at the community level, uptake of resources as a result of some DSCIF projects is occurring. For example, the document review found that information and resources produced by DSCIF projects are being used in classrooms and, more broadly, schools and communities outside the school that produced the resources, used in youth workers' daily work, incorporated into municipal and organizations' strategic plans and program planning and integrated into organizations' ongoing work.
The following are specific examples of uptake of health promotion and prevention knowledge and resources from the document review:

• One project reported that following a presentation of their health promotion and illicit drug use prevention program for young women, a local women's centre indicated interest in implementing the project within their organization.
• Another project reported an increased demand for their project's activities following dissemination of promotional activities and implementation of their communications plan.
• In one school based project, 37% of teachers identified increased awareness of drug information and resources that are available to youth as a key learning from the professional development session.

Other projects provided evidence that the resources they developed through DSCIF funding, or knowledge acquired through participation in a DSCIF project, were being used. For example:

• One project reported that school community councils and minor sport organizations were using its resources. Another project indicated that their products are in high demand and very valuable to front-line staff and their clients, especially with translated versions in many languages.
• A survey following up on the implementation of the CCSA standards indicated 39% of survey respondents and 51% of those aware of the Standards have begun using them in their work.
• In one school based project, 83% of teachers said they were ready to implement the project in their classrooms, 2-3 weeks after training; 33% said they had already implemented the project and 33% were ready to do so.

At the Program level, there is an opportunity to leverage the substantial performance data that has been collected to generate knowledge that can inform strategic planning, policy and program development, identify promising practices and guide community based programming

As noted above, the DSCIF program logic model also identifies knowledge exchange as a Program level activity, independent of the activities of the funded projects in this area. However, knowledge exchange requires analysis and knowledge translation to distil what is relevant to policy and program decision makers and to the prevention community. There is also an expectation that the Program will apply the knowledge gained through project performance reporting and evaluation activities to inform Program policy and program decisions as well provide information that will inform and support the prevention efforts of its stakeholders.

The Program has collected a wealth of project level performance and evaluation data over the last five years. This has included bi-annually reporting against indicators, project progress and evaluation reports and the collection of outcome data. The Program did use these findings and lessons learned to focus the priorities for the third round of funding on projects that focused on capacity building and behaviour changes amongst youth. Aside from this example, it is unclear how project level performance data is being used to inform strategic decisions about the Program (e.g., program objectives, program design improvements and to identify future research). This may be due, in part, to the fact that some of the final progress reports and evaluations for the 2009 CFP projects and the outcome report only became available in the fall of 2013. Staff indicated that they had not yet had an opportunity to analyze the full data set.

The Program has an opportunity to leverage the performance information it has collected from projects, along with the more recently completed (November 2013) project outcome report, to support strategic decisions about the future direction of the DSCIF and to contribute to the evidence base on community based illicit drug prevention programming.

A key achievement of the DSCIF was the extent to which community partners and networks were engaged in efforts to prevent illicit drug use among youth in their communities

Projects that had as their goal to increase community engagement focussed their activities on increasing collaboration between community-based groups and agencies, networks of people and resources, social inclusion, communication and skill building. Engaging new and existing community structures and networks in health promotion and prevention efforts was expected to entrench and support illicit drug prevention in the community. Community structures and networks were also involved in strategy and resource creation and dissemination.

A wide range of community partners were identified through the document review including community-based agencies and organizations (e.g., addictions and mental health services, YMCA, youth centres), schools and school boards, universities, non-governmental organizations, other levels of government, health authorities, RCMP and police, clergy, parent-school committees, youth, business owners, and Aboriginal organizations.

All lines of evidence including the Project Template Summary results from the 2007 CFP projects, survey responses from 2009 CFP projects, the document review, case studies and interviews with project representatives indicated that the DSCIF achieved this outcome. Community engagement was an integral aspect of the DSCIF program and working collaboratively with existing and new partners appeared to be inherent to the implementation of DSCIF projects. The evaluation found that in some communities, the DSCIF projects were instrumental in starting the dialog among community organizations, parents and youth about the issue of drug use among youth.

The majority of project leads (96.7%) from the 2007 CFP who reported on community engagement (n=32) provided positive evidence that both new and existing community partners were engaged in their project.

Similar results were achieved with the administration of the Community Capacity Building Tool pre and post intervention by the 2009 CFP projects.^{Footnote 63} 12 surveys were completed. There was a statistically significant change from baseline to post project on all of the following dimensions of community capacity:

• participation by the target population, community members, and other stakeholders;
• developing and nurturing both formal and informal local leaders during the project;
• engaging smaller or less formal community groups and committees;
• receipt of external supports from, for example government departments and regional health authorities;
• extent to which the project has explored the root causes and involved the target population in exploring root causes and finding solutions;
• capacity development (skills, knowledge and learning) as the project develops;
• linking the project with individuals and others; and
• sense of community.

Interviews with project representatives and the case studies also found that establishing partnerships was critical to the success of the project. The DSCIF provided an important source of funds to support collective action at the community level, in a way that was appropriate for the unique circumstances of each community. 100% of project representatives interviewed spoke to the importance of community partnerships and were able to describe a wide range of partnerships associated with their projects. Many indicated that the partnerships that were developed as a result of their DSCIF-funded project have continued beyond the DSCIF funding period.

The types of benefits derived from community engagement and the contributions of partners included knowledge/expertise, coordination and linkages with networks of existing services, in-kind resources (e.g., time, facilities, services such as translation, web design, and social media), and support for delivery, communications, strategic planning and involvement in strategy and resource creation and dissemination.

The following are some concrete examples, identified through the document review, of the benefits derived from a range of different types of community partnerships:

• One project reported new relationships with three new public health units, with interest in maintaining the program, and six schools from five different school boards.
• Several projects collaborated with local universities to harness their resources and partnerships to build community capacity.
• A project that created new community structures and networks in the form of project advisory committees in several communities, comprised of representatives from most of the key community sectors having an interest in youth and illicit drug issues, found that these committees not only provided guidance for project activities, but also operated as forums for the exchange of information and ideas on services, resources and programs available to, and/or needed within the area. Another project that created and maintained an Advisory Committee found that the Committee played an instrumental role in each step of the resource development.

A case study of a municipal-level project found active involvement in the development of its drug strategy by over 30 members from diverse types of organizations that included justice, enforcement, political, public health, primary care health, substance use treatment, housing, Aboriginal governance, private business, and self-help and faith-based. The strategy also involved citizens of all ages, including youth and seniors. The project reported that representatives from these diverse organizations were consistently engaged in the creation and consequent implementation of the drug strategy. Prior to the DSCIF initiative, there was no structure in place that incorporated organizations from diverse sectors at a leadership level.

There were several lessons learned about partnerships. Establishing partnerships early in the process, as well as maintaining good relationships through good communication and flexibility throughout the project contributed to success. Meaningful involvement of youth as partners at all stages of the project emerged as an important lesson learned for many of the projects. Youth engagement was also raised as a key success factor by most of the case study projects. Some projects had to learn these lessons as the project evolved. There is an opportunity for the DSCIF to share these lessons learned about youth engagement with future projects so that they are able to build on this knowledge.

4.5.3 To what extent have the longer term outcomes for communities been achieved?

While there are examples of improvements to community practice; the extent to which the Program has contributed to the achievement of this outcome is not clear given the available evidence.

This outcome builds on the expectation that the uptake of health promotion and prevention knowledge will lead to the application of knowledge and lead to improved community practices.

As is the case with the longer term outcome for youth, there is limited data available to make an overall assessment on the achievement of this longer-term outcome. However, concrete examples are available from the document review:

• A project that partnered with Black community organizations resulted in the project being able to increase the services offered to families, build networks with other social institutions like school boards, and obtain strategic planning, knowledge and expertise through collaborative and consultative endeavours.
• Another project reported that strengthened partnerships with police services and addictions services have led to the availability of in-school addictions services and the appointment of a Community-School Police Liaison Officer. In addition, the engagement of Elders with important cultural messages regarding healthy lifestyles and the need to avoid alcohol and illicit drugs enhanced the messages being delivered by the Program.

4.6 Performance: Issue #5 - Demonstration of Economy and Efficiency

The DSCIF program has made efforts to operate more efficiently and economically but there are further opportunities for administrative efficiencies.

The demonstration of efficiency and economy is defined by the Treasury Board of Canada Policy on Evaluation (2009) as an assessment of program resource utilization in relation to the production of outputs and progress toward expected outcomes. This evaluation focused on an operational efficiency analysis based on findings from the key informant interviews, literature review, and available financial data.

DSCIF had an overall budget of $56.6M including a $50M contribution budget. Table 6 shows the planned (Treasury Board allocation) versus actual spending for DSCIF's contribution funding. The Program indicated that the majority of the unspent contribution budget in 2008-09 was reprofiled to 2010-11 (60%). The Program noted that although lapses occurred in the first year, primarily due to delays in project approvals, funds were re-profiled to future years and there have been no significant lapses since 2009-2010.

Actual operating expenditures for DSCIF were below planned amounts in every year except 2010-11, as seen in Table 7. This was also the year with the highest contribution funding. The actual administrative ratio over the five years covered by the evaluation was 12%; this is lower than the administrative ratio (13%) from the original Treasury Board allocation. The administrative ratio is calculated as the percentage of total funds that is used for administrative purposes (i.e., O&M and salary spending). The Program achieved administrative efficiencies by supporting multi-year funding of projects, and employing existing administrative processes (such as amendments to low risk projects) to continue project work and effectively utilize all available funds.

There have been changes within the program area that have contributed to operational efficiency. In 2012-13, all program delivery was moved to the national office. This reduction of regional staff and operations provided more streamlined internal financial monitoring. The Program also adjusted to an 80% reduction in their operations and management budget. This impacted on the level of support to and oversight of funded projects as site visits were no longer feasible.

The CFP process was seen as an area that could be more efficient, by both the funded projects and by Program staff. DCSIF uses a traditional call for proposal process with a set window of opportunity for projects to apply within, as well as a set time frame for project implementation to commence. Projects found the timing of the actual calls, the length of time before approval, and the short time between approval and project start-up caused inefficiencies. The burden on staff to review proposals also caused some delays. The Program did shorten the average time from launch of the call for proposals to project start-up between the 2007 and 2009 funding periods; there are still areas for time savings.

Many other Grant and Contribution Programs are examining alternative methods for project submissions. The letter of intent process, which allows projects to submit a shorter general proposal for review, prior to the submission of a full proposal, is one approach. For the 2013 CFP, program staff felt that this would result in additional burden on the Program and would not be feasible within the current parameters of the DSCIF. The letter of intent process can benefit projects by allowing them more time to develop a proposal and ensure that all partners are on board.

The DSCIF program has made strides towards reducing the burden of reporting requirements on funding recipients. It has introduced a risk-based approach to reporting, including reducing reporting requirements funded under the 2013 CFP by eliminating the need to complete an interim project evaluation and final project report. There were issues regarding the formats required for reporting early in the funding process, but the technical difficulties have been mainly resolved. The Program is continuing to work with initiatives within the department on grants and contributions electronic reporting. As well, a risk based approach to reporting requirements and the elimination of the mid-project evaluation report have reduced the overall burden. Projects also have the opportunity to incorporate the data required for the common outcome reporting tools into their project level evaluations. This may reduce duplication of effort within the projects and further decrease the perceived burden of reporting.

Prevention based activities have been shown to have a significant positive effect on society and the economy. While it was not possible to examine the societal savings associated with DSCIF within the scope of this evaluation, American evidence^{Footnote 64} has shown that for every dollar spent on drug use prevention there is an associated savings of at least $15. The school based approach has shown clear cost savings. For example, one Washington State school life skills training program that targeted the initiation of risky behaviours leading to substance abuse showed a benefit-cost ratio of 37:1^{Footnote 65}

Many of the completed projects indicated that some or all of their activities would continue, beyond the DSCIF funding period. Of the 61 projects providing this information, 88% were planning on continuing some or all of the activities.The remaining projects reported that they would not be able to sustain their activities once their DSCIF funding ended. The sustainability of project activities demonstrates the on-going benefit to their communities and the overall support for drug prevention initiatives. Those projects that were fully continuing were able to integrate their activities into the ongoing work of their host or partner organizations. In one instance, project partners combined their resources to continue supporting a staff position previously funded through the DSCIF. In another of the case studies, the project was able to show the value of their activities and secure on-going funding from an external source after the DSCIF funding contribution ended.

As well, the majority of project representatives interviewed reported that they received in-kind support for their project activities beyond the DSCIF contribution amounts. The majority of in-kind resources came from project partners, and was in the form of expertise, connections to networks, and resources (such as facility space and administrative support). The file review corroborated that projects leveraged their relationships with stakeholders and partners. Some projects were able to identify external funding sources that would not have been possible without the initial funding from DSCIF. For these projects, the external funding was approximately 10% of the DSCIF amounts. However this is not a full representation of the amount of leveraged resources as reporting requirements that were in place did not require funding recipients to report on the dollar value of financial and in-kind resources.

The DSCIF focus on community level prevention activities could have led to duplication of efforts, as there are other funders involved at the community level. However, no duplication with other programs was identified. The health promotion and prevention community approach is supported and complemented by the work of other NADS partners, including the National Crime Prevention Centre (NCPC). NCPC supports targeted, evidence-based national and community-based crime prevention projects that aim to prevent and reduce drug abuse and drug-related crime among at-risk populations and communities. Target populations include at-risk children aged 7-12 who use substances, youth aged 13-17 who use substances and are at risk of displaying delinquent behaviour, juvenile and adult offenders no longer in correctional supervision who are addicted to substances, and Aboriginal people who are addicted to substances. Provincial and territorial governments' efforts towards preventing illicit drug use among youth tend to be part of broader strategies that address substance abuse as a whole. This also complements the DSCIF focus on illicit drugs within at-risk populations.

In many jurisdictions across Canada there is a focus on the continuum of care that links prevention, treatment and aftercare. Currently, discussions are occurring within Health Canada about merging the DSCIF and the DTFP. This proposed merger would align with the continuum of care approach of the provinces. As well, DSCIF has broadened its scope to fund projects that address prescription drug abuse. This broader focus also aligns with provincial direction. The merger of two funding programs could reduce costs by eliminating any overlapping administration associated with separate funds. With a consolidated fund, projects can address both prevention and treatment and allow for flexibility in meeting the needs of at-risk populations.

4.6.1 Observations on the Adequacy and Use of Performance Measurement Data

The DSCIF has been collecting a wealth of performance data from the funded projects. The efforts made in performance measurement by the Program have yielded solid evidence on the impacts being made by the funded projects.

Projects completed progress reports, mid-term and final project evaluations and participated in common data collection within one of three outcome areas. The implementation of a common tool to collect outcome indicators was a positive step. This common outcome data provided key evidence in this evaluation, and will continue to support Program activities. There were some issues with the implementation of the approach as tools and support were not in place at the start of the funding. As well, some projects felt that the tools were not appropriate for their target population or participants. This was especially true for Aboriginal projects. The Program is continuing to address these concerns by allowing modifications of questionnaires and providing further training on the evaluation process for projects.

The project level reporting on outcomes has led to inefficiencies. Projects were expected to evaluate their projects and provide information on all the outcomes identified in the logic model, as well as participate in the outcome surveys which focussed on three of the main outcome areas. Although projects had the opportunity to incorporate the data required for the common outcome reporting tools into their project level evaluations, not all did. As the DSCIF program staff will now be responsible for the outcome evaluation (it was previously managed by a contractor), there should be attention paid to opportunities to reduce burden on both project and Program staff, while ensuring that sufficient performance and outcome data is available for future program and evaluation requirements.

As well, many project representatives interviewed were not familiar with the final results from the common outcome evaluation. The Program had shared results with newly funded projects, but not those projects that were no longer receiving funding. Project representatives interviewed would have appreciated receiving feedback from their outcome survey results so that they could learn from the results and identify areas for improvement.

The Program also consolidated and reviewed performance information from the other data sources (overview, progress and evaluation reports), to report bi-annually against indicators and to prepare internal documents on lessons learned and project achievements. Combining these with the common outcome information has allowed for the inclusion of evidence based information in parliamentary reports and has supported the required NADS reporting. This information has also been shared with new funding recipients and at conferences.

There are some areas where performance measurement could be enhanced to support assessment of efficiency and economy, specifically leveraged funds and financial value of in-kind support.

5.0 Conclusions

5.1 Conclusions - Relevance

All the lines of evidence confirmed the ongoing relevance of the DSCIF.

5.1.1 Continued Need

There is an ongoing need for community-based health promotion and prevention efforts to build capacity and to address gaps and emerging issues associated with illicit drug use. National surveys indicate that the conditions that led to the realignment of DSCIF under the NADS in 2007 still exist. Rates of illicit drug use among youth 15 to 24 years of age remain higher than among adults. Certain subpopulations of youth (Aboriginal, street-involved/homeless youth and lesbian, bi-sexual, gay and transgendered (LGBT) youth) have even higher rates of illicit drug use. The literature indicates that prevention efforts can mitigate a range of legal, social and economic impacts of illicit drug use. The DSCIF provides an important source of funds to support collective action at the community level, in a way that is appropriate for the unique circumstances of each community.

5.1.2 Alignment with Government Priorities

The DSCIF program's health promotion and prevention activities support the federal government's priority areas of illicit drugs and youth. Health Canada is the lead on the NADS Prevention Action Plan. The DSCIF, as a key component of the Prevention Action Plan, therefore aligns with federal priorities for illicit drug use as articulated in the NADS.

The objectives of the DSCIF program to raise awareness and understanding of the harmful social and health effects of illicit drug use also supports Health Canada's Strategic Outcome #2 −health risks and benefits associated with food, products, substances, and environmental factors are appropriately managed and communicated to Canadians.

There are no clear, planned formal mechanisms in place to ensure the appropriate internal and external stakeholders are engaged at key decision points in the program lifecycle. Stakeholder engagement should be planned and deliberate so that the purpose and intent of engagement are obvious to all involved. The policy function is currently located in the Healthy Environments and Consumer Safety Branch (HECSB) and the management and administration of the DSCIF contribution agreements is located in the Strategic Policy Branch (SPB). Multiple reorganizations, staff turnover and reductions have contributed to decreased collaboration between the two areas at key decision points (e.g., policy decisions, decisions on DSCIF funding priorities and CFP processes). The evaluation found that the transition from a regional to national delivery model has also impacted on Health Canada's level of engagement with external stakeholders in identifying gaps and determining funding priorities.

5.1.3 Alignment with Federal Roles and Responsibilities

Education and prevention are typically the responsibilities of the provinces and territories (P/Ts). Funding community-based programming is generally not a federal role. However, the concerted effort of all levels of government is necessary to address the complexity and many challenges associated with illicit drug use, as well as the differing level of capacity to address illicit drug use among youth across the country.

The P/Ts generally address substance abuse more broadly to include both licit and illicit substances. The federal government has played a role at the broad policy level through NADS to address illicit drug use. Health Canada supports this role as the lead on the Prevention Action Plan of the NADS. It fulfills the federal commitment to implement the Prevention Action Plan by funding innovative projects and mobilizing a wide range of community-based organizations to carry out health promotion and prevention activities that focus on illicit drugs while also supporting the role of the P/Ts. However, as the environment has recently changed for this Program, an opportunity exists to examine the direction and scope of the new Program going forward to more fully align with the federal role.

Most P/Ts and project stakeholders supported a role for Health Canada as a funder but also noted that Health Canada could have a stronger and more influential role as a funder of innovative practices, in knowledge translation and exchange, and in facilitating collaboration and coordination to address illicit drug use among youth.

5.2 Conclusions - Performance

5.2.1 Achievement of Expected Outcomes (Effectiveness)

To assess the achievement of outcomes, the evaluation relied on outcome data collected by the Program, and supplemented this information with key informant interviews and case studies.

The DSCIF program produced the expected outputs and made progress in achieving most of its immediate and some intermediate outcomes. Reporting focussed on the lifecycle of the agreements and therefore longer term outcomes were not captured as part of the reporting tools. For some projects, this was a lost opportunity to collect information on longer term outcomes, such as behaviour change. However, a few projects reported on progress toward achieving longer term outcomes. Also,given the small scope of the DSCIF program, there was no expectation that the Program would have national level prevalence impacts.

The evaluation confirmed that youth who participated in the DSCIF funded projects increased their awareness and understanding of healthy lifestyle choices, and illicit drugs and their negative consequences and improved their capacity (knowledge and skills) to avoid illicit drugs. All funded organizations were able to provide strong evidence of engagement of community partners and networks in efforts to prevent illicit drug use.

Funding recipients produced knowledge products and resources and increased the access to and awareness of these knowledge products and resources within their community and networks. There was also evidence that progress was made with respect to the community uptake of these knowledge products and resources.

The evaluation identified a gap in terms of knowledge translation and pan-Canadian knowledge exchange. The logic model also identifies knowledge exchange as a program-level activity. Although the Program has summarized progress and evaluation reports, and produced a lessons learned report, Health Canada's role in knowledge translation (analysis and synthesis of the performance data from the funded projects) to identify promising practices, to inform program decision making or in the pan-Canadian dissemination of the knowledge products and resources (linked to promising practices) produced by the funded projects was limited. Some passive knowledge diffusion took place at the project level through the funded organizations. However, smaller organizations, in particular, lacked the capacity for knowledge exchange beyond their immediate community and networks.

The DSCIF program has taken steps to ensure the availability of performance data at the program level. The Program has the opportunity to leverage the performance data it has collected and to use it strategically to inform program priority setting and funding decisions, and to adjust DSCIF objectives and program design. Furthermore, the Program could contribute to the knowledge base on drug prevention programming for youth by further analysing the project performance data it has collected to identify promising practices as well as promising knowledge products and resources produced by projects. Disseminating this information more broadly would enable stakeholders to act on and use the information, thereby extending the impact of the Program. P/T and project stakeholders expect Health Canada, or another national organization, to have a role in knowledge translation and dissemination.

5.2.2 Demonstration of Economy and Efficiency

Overall, the contributions to community-based organizations were well managed in terms of monitoring progress and ensuring compliance with contribution agreements. The Program has made efforts to operate more efficiently, however the evaluation identified opportunities for additional efficiencies by streamlining the CFP and performance reporting processes. The evaluation also identified opportunities to improve collaboration and coordination between the policy and program delivery organizations within Health Canada which would help increase efficiency in the approach to managing funding agreements.

The evaluation noted that many of the funding recipients interviewed found that reporting requirements could be burdensome and that the time invested in meeting these requirements detracted from time for program delivery. While performance data was available to inform the evaluation, reporting activities represented a significant investment of National Office resources. For the next round of funding,the Program has streamlined monitoring and reporting requirements using a risk-based approach to reporting. However, in moving forward, the Program should assess its performance reporting requirements to ensure that they are sufficient to produce quality performance data, but not excessive for recipients.

6.0 Recommendations

The evaluation identified the following two recommendations.

Recommendation 1

Significant changes to the program infrastructure, including recent Government of Canada announcements on the inclusion of prescription drug abuse (as an area of focus for DSCIF), the proposed merger of DSCIF and the Drug Treatment Funding Program (DTFP), the impact of the transition from regional and national program delivery on the level of service that the Program can provide funding recipients and the expectations of stakeholders with respect to knowledge translation and exchange, suggest the need for the HECSB and the SPB to engage in a policy and strategic planning exercise to define the parameters of the new substance abuse program going forward.

Health Canada stakeholders should engage in a policy and strategic planning discussion that will lead to a decision on the direction and scope of the new Program going forward This could include identifying:

• the delivery model for the new Program within the broader Health Canada context of controlled substances;
• Health Canada's role in knowledge translation and exchange and in the dissemination of project knowledge products and resources; and
• the roles and responsibilities of Health Canada stakeholders (HECSB policy and research and surveillance, and the SPB Drugs Program), and collaborative mechanisms to engage stakeholders in setting departmental policy direction on illicit substances and making programmatic decisions.

Recommendation 2

The Program has accumulated a wealth of project performance data that has not yet been leveraged to the fullest extent to identify information and promising practices that can inform Health Canada and stakeholder policy and program decisions.

The Program should:

1. leverage its investment in performance measurement by further analyzing data collected from the projects and by using lessons learned strategically to inform program priority setting and funding decisions and to adjust the DSCIF objectives and program design.
2. disseminate lessons learned and project developed knowledge products and resources, linked to promising practices, to inform the activities of stakeholders.

Appendix 1 - Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of Health Canada's Agency's Drug Strategy Community Initiatives Fund program between April 2008 and March 2013, including its objectives and activities: The assessment of performance covered Program level activities, including program promotion, stakeholder engagement, partnership development and knowledge exchange. The evaluation also included an assessment of the performance of projects which received contribution funding to plan, organize, deliver and report on their health promotion and prevention activities. The assessment of performance focused on immediate and intermediate outcomes. The assessment of longer term outcomes was more limited as the focus of funding and reporting was on the immediate and intermediate outcomes.

The evaluation did not include an assessment of the performance of projects funded through the 2013 CFP, as these projects were approved in the last quarter of 2013-14, and were just being implemented at the time of reporting.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the Program and guided the evaluation process.

Core Evaluation Issues and Questions

Evaluators collected and analyzed data from multiple sources, including:

Literature review

The search for Canadian and international literature focussed on published, peer-reviewed literature and concentrated on literature published from January 2007 until October 2013 and online written materials available on selected federal, national, provincial/territorial and international websites. The focus was on identifying material relevant to understanding the problem of illicit drug use among youth ages 10-24, trends, state-of-the-art thinking, guidance and evidence on effective health promotion and prevention.

Financial data review

This review looked at of financial data from 2008-09 to 2012-13, including budgeted and actual expenditures.

Document review

Program documents reviewed included CFP documentation and program authorities. Data was analysed with NVIVO.

Performance data

The review of data on implementation of Program activities and funded project between 2008-09 and 2012-13, included a review of Program synopsis of project progress and evaluation reports, Program performance reports to meet NADS reporting requirements, Program level reports, including a project overview report, project outcome report, case studies and a lessons learned report.

Key informant interviews

Approximately 40 nterviews were conducted with both internal and external key stakeholders, including Program staff and management (n=15); representatives from the P/Ts (Directors, Managers and Analysts of provincial substance abuse/addictions programs (n=7)), project leads for funded projects (n=9) the Department of Justice Canada (n=2) and leads of the projects selected for the case studies (n=4). Interviews were transcribed and were analysed with NVIVO.

Case studies

Four case studies supplemented the case studies conducted by the Program. Projects were selected based on recommendations from key informant interviews. The projects covered the various regions as well as a national project. The projects were from both cohort 1 and 2, and represent all of the clusters. All projects were multiple years, and the funding amounts varied. Documents reviewed included the initial proposal, ongoing monitoring reports, final evaluation reports, cluster evaluation instruments and other available documentation provided by the projects. Interviews were conducted with the project lead of the funded organization. Interviews addressed the sustainability of the project and knowledge translation, as well as project outcomes and the overall funding process.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of data to illustrate key findings; statistical analysis of quantitative data from databases; thematic analysis of qualitative data; and comparative analysis of data from disparate sources to validate summary findings.

Appendix 2 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance

A summary of Performance ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

References

• Auditor General of Canada. (2001). 2001 December Report of the Auditor General of Canada. Chapter 11--Illicit Drugs--The Federal Government's Role.
• Beasley, Erin et. al. (2012). Alcohol and Drug Use Among Drivers Following the Introduction of Immediate Roadside Prohibitions in British Columbia: Findings from the 2012 Roadside Survey.
• Canadian Centre on Substance Abuse. (2007). Substance Abuse in Canada: Youth in Focus.
• Canadian Centre on Substance Abuse. (2011). Student Alcohol and Drug Use: Report in Short.
• Canadian Centre on Substance Abuse. (2012). National Treatment Indicators Report.
• Canadian Centre on Substance Abuse. (2013). A Case for Investing in Youth Substance Abuse Prevention, citing Lee, S. Drake, E. Pennucci, A., Miller M. and Anderson, L (2012). Return on investment: Evidence-based options to improve statewide outcomes. Olympia: Washington State Institute for Public Policy, April 2012.
• Canadian Centre on Substance Abuse. (2014). Cannabis Use and Risky Behaviours and Harms: A Comparison of Urban and Rural Populations in Canada. Ottawa: Canadian Centre on Substance Abuse.
• Centre for Addiction and Mental Health. (2013). Drug Use Among Ontario Students 1977-2013.
• City of Ottawa. (2011). Enhanced Street Youth Surveillance in Ottawa 2011.
• Department of Justice Canada. (2012). National Anti-Drug Strategy Evaluation. Ottawa. Department of Justice Canada.
• European Monitoring Centre for Drugs and Addictions. (2013). European Drug Report 2013: Trends and developments.
• Government of Canada. (1985). Canada Health Act, R.S.C., 1985, c. C-6.
• Government of Canada. (2002). House of Commons Special Committee on Non-Medical Use of Drugs.
• Government of Canada. (2002). Report of the Senate Special Committee on Illegal Drugs.
• Government of Canada. (December 2006). Report From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs.
• Government of Canada. (October 2007). Speech from the Throne to Open the Second Session of the Thirty Ninth Parliament of Canada.
• Government of Canada. (February 11, 2014). The Road to Balance: Creating Jobs and Opportunities.
• Hall, WD and Pacula RL. (2003). Cannabis use and dependence: Public health and public policy. Cambridge: Cambridge University Press; 2003. 298 pp
• Health Canada. (2007). Drug Strategy Community Initiatives Fund Guide for Applicants 2008-2009. Ottawa: Health Canada.
• Health Canada. (2007). Drug Strategy Community Initiatives Fund Guide for Applicants 2008-2009.
• Health Canada. (2009). Drug Strategy Community Initiatives Fund (DSCIF) - At a Glance.
• Health Canada. (2010). Terms and Conditions for the Drug Strategy Community Initiatives Fund Contribution Program.
• Health Canada. (November 2010). Drug Strategy and Community Initiatives Fund (DSCIF) Performance Measurement Strategy.
• Health Canada. (2010). Formative Evaluation of the DSCIF 2008 and 2009 Calls for Proposals.
• Health Canada. (2011). About Mission, Values, Activities.
• Health Canada. (2012). Supplementary Tables: Youth Smoking Survey 2010-2011.
• Health Canada. (2013). Canadian Alcohol and Drug Use Monitoring Survey Detailed Tables.
• Health Canada. (2013). Canadian Alcohol and Drug Use Monitoring Survey Summary of Results for 2012.
• Health Canada. (2013). National Addictions Awareness Week - November 18-24, 2013.
• Health Canada (2013). Report on Plans and Priorities 2014-2015.
• Health Canada. (2014). DSCIF Lessons Learned.
• Health Canada. (2014). Health Care System.
• Health Canada. (2013). Report on Plans and Priorities 2014-2015.
• Jackman, M. (2000). Constitutional Authority over Health in Canada. Health Law Journal, 8, 95-117.
• Lee, S., Drake, E., Pennucci, A., Miller, M. & Anderson, L. (2012). Return on investment: Evidence-based options to improve statewide outcomes. Olympia: Washington State Institute for Public Policy, April 2012 (Document No. 12-04-120).
• Mental Health Commission of Canada. (2014). Issues of Substance.
• MacLellan-Wright, Mary Francis, et. al. (2007). The development of measures of community capacity for community-based funding programs in Canada. Heath Promotion International. Volume 22, Issue 4. Pp. 299-306
• Miller, Cari L. (2011). The Cedar Project: risk factors for transition to injection drug use among young, urban Aboriginal people. CMAJ. vol. 183 no. 10.
• National Advisory Committee on Prescription Drug Misuse. (2013). First Do No Harm: Responding to Canada's Prescription Drug Crisis. Ottawa: Canadian Centre on Substance Abuse.
• Northwest Territories Health and Social Services. (December 2010). Prevalence of alcohol, illicit drug, tobacco use and gambling in the Northwest Territories. Retrieved March 26, 2012.
• Public Health Agency of Canada. (2006). Enhanced Street Youth Surveillance (E-SYS) in Canada. Ottawa, Ontario: Centre for Communicable Diseases and Infection Control, Public Health Agency of Canada.
• Public Health Agency of Canada. (2006). Street Youth in Canada. Findings from Enhanced Surveillance of Canadian Street Youth, 1999-2003.
• Public Health Agency and Health Canada Evaluation Directorate. (2013). Drug Strategy Community Initiatives Fund Evaluation Framework.
• Public Health Agency and Health Canada. (2013). Public Health Agency and Health Canada Five-Year Evaluation Plan 2014-2015 to 2018-2019.
• Rey, Joseph M. (2002). Mental health of teenagers who use cannabis. The British Journal of Psychiatry. 180: 216-221
• Treasury Board Secretariat. (October 2008). TB Policy on Transfer Payments.
• UNICEF Office of Research. (2013). Innocenti Report Card 1. Child well-being in rich countries A comparative overview. Office of Research, Florence.
• Webster, Paul C. (2012). Prescription drug abuse rising among Aboriginal youths. CMAJ. September 4, 2012 vol. 184 no. 12.

Endnotes

List of Acronyms

ADM

Assistant Deputy Minister

CADUMS

Canadian Alcohol and Drug Use Monitoring Survey

CCSA

Canadian Centre for Substance Abuse

CDS

Canada's Drug Strategy

CFP

Call for proposals

CODA

Council on Drug Abuse

CPAB

Cominucations and Public Affairs Branch

DSCIF

Drug Strategy Community Initiatives Fund

DTFP

Drug Treatment Funding Program

DG

Director General

F/P/T

Federal/Provincial/Territorial

HECSB

Healthy Environments, Consumer Safety Branch

HPSI

Health Programs and Strategic Initiatives

HQ

Headquarters

LGBT

Lesbian, Gay, Bi-sexual and Transgendered

NADS

National Anti-Drug Strategy

NGO

Non-Governmental Organizations

OPI

Office of Primary Interest

OSDUHS

Ontario Student Drug Use and Health Survey

PAA

Program Alignment Architecture

PHAC

Public Health Agency of Canada

P/T

Provincial/Territorial

RAPB

Regions and Programs Bureau

RPP

Report on Plans and Priorities

SPB

Strategic Policy Branch

TBS

Treasury Board of Canada Secretariat

U.S.

United States

YSS

Youth Smoking Survey

Footnotes