Evaluation of the Human Drugs Program 1999-2000 to 2011-2012

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

May 2014

Executive Summary
Management Response and Action Plan
1.0 Introduction
- 1.1 Organization of the report
2.0 Profile of the Human Drugs Program
3.0 Evaluation methodology
- 3.1 Evaluation design and methods
- 3.2 Limitations of the methodology and mitigation strategies
4.0 Findings - relevance
5.0 Findings: governance and implementation
6.0 Lessons Learned
7.0 Findings - efficiency and economy
8.0 Conclusions and recommendations
Appendix A - Evaluation Matrix
Appendix B - List of References
Appendix C - Supplementary Data Tables

Executive Summary

The Human Drugs Program (HDP) is managed through the Health Products and Food Branch (HPFB) of Health Canada. The HDP encompasses human drugs (also known as pharmaceutical drugs), which includes prescription and non-prescription drugs intended for human use, excluding biologic-related drugs, vaccines, blood and blood products, cells, tissues, and organs, which fall within the scope of HPFB's Biologics Program.

The main program partners involved in delivering the HDP are the Therapeutic Products Directorate (TPD), the Marketed Health Products Directorate (MHPD), the HPFB Inspectorate (the Inspectorate), and the Regions and Programs Bureau (RAPB), as well as the Policy, Planning and International Affairs Directorate (PPIAD) and the Resource Management and Operations Directorate (RMOD). Program activities include creating and maintaining the regulatory framework, interacting and communicating with partners and stakeholders, performing risk-benefit assessments, conducting post-market monitoring and surveillance, and compliance and enforcement of the regulatory framework.

The evaluation of the HDP is part of Health Canada's Five-Year Evaluation Plan. In accordance with the Treasury Board Policy on Evaluation (TBS, 2009), the evaluation assessed the relevance and performance (effectiveness, efficiency, and economy) of Health Canada's activities under the program. The evaluation covered the period from 1999 to the end of calendar year 2012, with an emphasis on the last five years, although the report reflects program activities up to the end of June 2013. The results of the evaluation will inform the implementation of current and future program activities.

An independent evaluation consulting firm conducted the evaluation on behalf of Health Canada. The evaluation drew on several lines of evidence, including literature review, document review, review of administrative data, case studies, surveys of manufacturers and other stakeholders, and interviews with external key informants. In addition, throughout the process, the evaluation consulted with program representatives by means of conference calls and email to request additional information and clarify issues. This report presents the evaluation findings, draws conclusions, and makes recommendations.

Findings

Relevance

The evaluation confirmed an ongoing need for government oversight of human drugs in order to protect the health and safety of Canadians. As use of these products grows due to population growth and aging, as well as marketing by industry, more Canadians will be exposed to the risks, as well as the benefits, of these products. Moreover, trends such as the emergence of combination products and globalization of the supply chain are creating uncertainties that further support the need for government intervention to protect the health and safety of Canadians. Such a role, furthermore, is consistent with federal and Health Canada roles and responsibilities, as described in federal statutes and regulations, and aligns directly with Health Canada's strategic outcome to inform and protect Canadians from health risks associated with food, products, substances, and environments.

HDP activities are well-aligned with federal priorities to strengthen consumer safety. The federal government has devoted substantial resources over the past decade to broader initiatives intended to improve the safety of health products, including human drugs, through modernizing the regulatory framework for these products. Key principles of regulatory modernization include adopting a product lifecycle approach, whereby the risks and benefits of therapeutic products are assessed over their entire lifecycle; adopting regulatory interventions proportional to risk; and enhancing the transparency and openness of the regulatory system. The federal government recently signalled its commitment to the long-term sustainability of the HDP by including human drugs in recent updates to its cost recovery framework and user fees.

Performance - program implementation

Over the period of the evaluation,Health Canada has made progress in implementing its planned activities and, in the process, has responded to several emergent issues and challenges. However, a number of unresolved issues and challenges remain.

Clinical trials

Health Canada has strengthened the regulatory framework for clinical trials by implementing risk-based approaches to monitoring clinical trial adverse reaction reports, introducing an inspection program for clinical trial sites, and commissioning the development of new voluntary standards for Research Ethics Boards. Most recently, in May 2013, Health Canada launched a new public database of drug clinical trials it had authorized. Though mandatory for sponsors, the database contains more limited information than the registries that have been mandatory in the United States (US) since 1997 and the European Union (EU) since 2004. Furthermore, unlike the US (and soon the EU), Health Canada does not require sponsors to disclose the results of clinical trials. Further enhancing the amount of clinical trial information it makes publicly available, including the results of clinical trials, would be consistent with Health Canada's commitment to enhancing transparency and openness as part of regulatory modernization, and would further align its approach with its main international counterparts.

Recommendation 1:

Consistent with international trends and its commitment to enhancing transparency and openness under regulatory modernization, Health Canada should further enhance the amount of clinical trial information it makes publicly available, including the results of clinical trials.

Submission review and market authorization

Health Canada achieved a major milestone with the coming into force of the Fees in Respect of Drugs and Medical Devices Regulations in April 2011. The Regulations updated user fees for various regulatory services, including submission review and drug establishment licensing, with the goal of restoring the cost-sharing ratio of 50% that existed when user fees were first introduced in the mid-1990s. The increase in revenues stemming from updated user fees is expected to contribute to a more stable funding platform that will improve Health Canada's ability to provide regulatory services.

In addition to the new cost recovery framework, Health Canada also implemented a number of initiatives to improve the quality and efficiency of the submission review and market authorization process. These include implementing a project management approach to submission review; developing Good Guidance Practices and Good Review Practices for greater consistency in these processes; collaborating with the US Food and Drug Administration (FDA) under the Regulatory Cooperation Council (RCC) to develop the Common Electronic Submission Gateway, which will allow sponsors to make submissions simultaneously to Health Canada and the FDA; enhancing scientific review capacity; and taking steps to increase foreign review when reviewing Canadian applications.

The impact of these initiatives on the efficiency and quality of the review process is an area for continuous and future analysis. The timeliness of submission review has fluctuated over the period of the evaluation and Health Canada's performance target had usually not been met. However, since the implementation of the new cost recovery framework in April 2011, TPD has met most of its cost recovery performance targets, with the exception of generics, and submission review performance has been relatively consistent since that time.

Post-market surveillance

Adverse reaction reporting has historically been the pillar of Health Canada's approach to post-market surveillance. Generally speaking, mandatory reporting of adverse reactions by industry is unproblematic. To address the long-standing problem of under-reporting by health professionals, Health Canada commissioned the development of national adverse reaction reporting standards for accredited hospitals and health care institutions, which were released in January 2013. At that time, Health Canada indicated that it planned to monitor the impact of the new standards before deciding to proceed with regulatory amendments. Mandatory adverse reaction reporting for health care institutions had previously been included in Bill C-51 in 2008, a wide-ranging piece of legislation intended to address a variety of gaps in the existing regulatory framework for health products. Bill C-51 died on the Order Paper and did not become law.

Health Canada is currently in the process of developing information systems to receive adverse reaction reports electronically and to monitor and analyse them for safety signals. In particular, Health Canada is implementing electronic reporting of adverse reactions for industry, as well as strategies to systematically monitor adverse reaction reports through targeted surveillance and data-mining. While delays have been encountered due to technical difficulties, these initiatives are expected to enhance Health Canada's ability to analyse domestic adverse reaction reports for safety signals. Electronic submission of adverse reaction reports and data-mining are also expected to enable Health Canada to analyse foreign adverse reaction reports, which represent more than 90% of all adverse reaction reports received by the Branch. Until Health Canada has these information systems and functionalities in place, initiatives (such as the national adverse reaction reporting standards) to increase the number of adverse reaction reports received may be premature.

In recent years, like other regulators around the world, Health Canada has recognized the limitations of relying on spontaneous adverse reaction reporting for safety signals, and has sought to strengthen post-market surveillance in a variety of ways. Health Canada has expanded the sources of information it monitors for safety signals beyond adverse reaction reports, and has standardized the process for signal detection and assessment across its product lines. However, Health Canada lacks a standardized approach and centralized mechanism for systematically tracking its signal activities and its response to the recommended actions arising from completed signal assessments. At present, this information is inconsistently maintained and highly dispersed across HPFB. Given the potential implications for the health and safety of Canadians, a more comprehensive, consistent, and centralized approach to information management for Health Canada's post-market surveillance activities seems warranted.

Recommendation 2:

Health Canada should improve its information systems for post-market surveillance and monitoring. This should include:

fully implementing information systems to support electronic submission of domestic and foreign adverse reaction reports, as well as systematic monitoring and analysis of these reports through targeted surveillance and data-mining.
developing and implementing a comprehensive and centralized approach to information management for post-market surveillance activities. In particular, this should include a centralized mechanism for tracking signal activities, including Health Canada's response to the recommended actions arising from completed signal assessments.

Despite announcing plans to do so under the FCSAP, Health Canada has not implemented some elements of a strengthened approach to post-market surveillance that are in place elsewhere, namely the authority to compel manufacturers to submit risk management plans and periodic safety update reports. However, these can be submitted by industry on a voluntary basis in response to requests from Health Canada. While Health Canada has acknowledged that the absence of these authorities is a limitation of its approach, it is unclear if it intends to pursue these authorities in future. Another area of uncertainty is the extent to which Health Canada monitors manufacturers' compliance with the conditions imposed as part of Notices of Compliance with Conditions.

Recommendation 3:

Health Canada should examine whether there is an ongoing rationale for pursuing post-market regulatory authorities to which it committed under the FCSAP, namely, the authority to compel manufacturers to submit risk management plans and periodic safety update reports.

Compliance and enforcement

In the area of compliance and enforcement, Health Canada has strengthened clinical trial inspections by developing a risk-based process for site selection; strengthened oversight over imported products through the National Border Integrity Program; renamed the former Post Marketing Compliance Reporting Program (PMRC) as the Good Pharmacovigilance Practices (GVP) inspection program to be in alignment with international requirements; extended Good Manufacturing Practices (GMP) and drug establishment licensing (DEL) requirements to active pharmaceutical ingredients (APIs); and adopted a risk-based approach to GMP inspections of domestic drug establishments.

Recent events, such as the slowdown in production of a generic manufacturer after FDA inspections revealed several non-compliances, have focused public attention on, and raised questions about, Health Canada's domestic GMP inspection program. While some external key informants raised concerns about inconsistent interpretation of standards and quality of inspections, Health Canada indicated that the different inspection outcomes are likely due to differences between the two regulatory agencies in their approach to inspections and the scope of the inspections.

Another issue may be Health Canada's oversight of foreign facilities that produce drugs for import into Canada. While Canada has Mutual Recognition Agreements (MRA) with several countries recognizing the equivalence of their respective GMP compliance programs, a majority of the drug products and APIs imported into Canada come from countries with which Health Canada does not have an MRA. In the case of APIs, importation accounts for 85% of the Canadian market, with China and India being the major suppliers. Furthermore, although domestic establishments are inspected regularly according to risk-based criteria, Health Canada does not often conduct inspections of foreign facilities. The recent extension of GMP and DEL requirements to APIs is expected to reduce the possibility that low-quality or counterfeit health products will appear in the Canadian market.

To address the challenges associated with GMP inspections, Health Canada and the US FDA have launched an initiative under the RCC that aims to increase each country's reliance on GMP inspection reports prepared by the other country. Currently, the initiative applies to sites in Canada and the US, although it may be expanded to other jurisdictions in future. In addition, Health Canada is a member of the Pharmaceutical inspection Cooperation Scheme (PIC/S), a body of 43 international participating authorities that have come together to harmonize GMP inspection requirements and approaches across the world. Health Canada reviews PIC/S inspection reports to approve foreign sites for Canada importers.

Recommendation 4:

Health Canada should explore options for improving its oversight of foreign manufacturing sites.

Furthermore, although a key focus of the RCC initiative is the standardizing and sharing of GMP inspection reports, Health Canada and the FDA differ in their reporting approaches. While the FDA reports by product category, Health Canada aggregates all GMP reporting, regardless of the type of product involved. Achieving the objectives of the initiative may require a common approach to compliance reporting.

Finally, Health Canada included new administration and enforcement measures as part of Bill C-51. These measures included the power to issue mandatory recalls of therapeutic products and increased fines for non-compliances. As already described, Bill C-51 did not become law. In interviews, some external key informants expressed concerns about what they perceive as the relatively few enforcement options available to Health Canada.

Recommendation 5:

Health Canada should examine whether there is an ongoing rationale for pursuing compliance and enforcement measures to which it committed under the FCSAP, namely the authority to issue mandatory recalls of therapeutic products and the authority to levy increased fines for noncompliance.

Communications and stakeholder engagement

Health Canada has undertaken a number of initiatives to improve communications and stakeholder engagement. For example, since 2005, Health Canada has provided the public with information about review decisions through Summary Basis of Decision (SBD) documents. In 2012, in part to address concerns expressed by the Office of the Auditor General (OAG) that it was not disclosing information related to Notices of Compliance with Conditions (NOC/cs), rejections, and withdrawals of new drugs, Health Canada introduced the post-authorization activity table (PAAT). PAATs provide ongoing information about approved products. They include a brief summary of activities that affect the safe and effective use of the product, such as information related to submissions for a new use of the product (whether Health Canada's decision was positive or negative), submissions filed in order to meet conditions (for products approved under the NOC/c Guidance), and regulatory decisions such as the cancellation of the Drug Identification Number (DIN). Health Canada does not publish SBDs for negative decisions or otherwise provide information to the public about the reasons for negative decisions, unlike the FDA and the European Medicines Agency (EMA).

To improve the quality and availability of easy-to-understand drug product labelling, Health Canada has enhanced the product monograph, developed a common monograph with the FDA for over-the-counter (OTC) drugs under the RCC, and introduced regulatory amendments under the Plain Language Labelling Initiative. At present, Health Canada has limited authority to require manufacturers to modify product labelling once a product has received a Notice of Compliance. In contrast, the FDA has the authority to require product labelling to be updated as new safety information becomes available, including the authority to require a boxed warning containing new safety information. The EU's new pharmacovigilance legislation requires medicines subject to additional post-market monitoring to include an inverted black triangle on a drug product's leaflet.

Over the period of the evaluation, Health Canada has disseminated risk and safety information through the "advisories, warnings and recalls" page on the MedEffect website and a variety of other dissemination mechanisms. In early 2013, Health Canada launched the Recalls and Safety Alerts Database, which includes an advanced search feature and a new format for risk communications. Health Canada indicated that it is currently in the process of reviewing its existing performance targets for the content development of risk communications and the dissemination of risk communications, and is looking at areas where more focused improvements in risk communications could be made. Health Canada also recently initiated an evaluation of its risk communications for health products, including human drugs, following through on long-standing plans to assess the effectiveness of its risk communications products.

Health Canada provides a variety of opportunities for stakeholder engagement, such as holding public consultations on proposed guidance, policies, and regulatory amendments and establishing advisory committees to guide regulatory and policy development. In addition, Health Canada consults with industry through pre-submission meetings and the Bilateral Meeting Program (BMP). While these specific opportunities are not available to health care practitioners and consumers/patients, Health Canada indicated that it does meet with pharmaceutical associations, hospital associations, and medical associations representing health care practitioners. Some external key informants expressed concern that the engagement and consultation process may favour industry over other stakeholders.

Performance - outcomes achieved

Over the period under evaluation, Health Canada has engaged in many activities that are expected to contribute to the outcomes of the HDP. However, for various reasons, data to support definitive conclusions on outcomes achieved are relatively limited.

Immediate outcomes

In the immediate term, Health Canada's activities are expected to produceincreased awareness and understanding by non-industry stakeholders of risks and benefits related to human drugs. Surveys conducted between 2003 and 2007 identified opportunities to improve awareness among both consumers and health professionals of drug safety information available from Health Canada, although information specific to human drugs was not available. Health Canada is currently in the process of evaluating the effectiveness of its risk communications for therapeutic products.

In the immediate term, Health Canada's activities are expected to produce increased awareness and understanding among industry of Health Canada's regulatory activities for human drugs. The available evidence, though limited, points to some potential areas for improvement. For example, while industry survey respondents believe that their firm has a strong understanding of Health Canada's submission requirements for market approval, industry key informants suggested that greater clarity may be required in relation to the classification of emerging health products and Health Canada's use of foreign reviews and guidance in the review process.

In the short term, Health Canada's activities are also intended to produce increased safety and effectiveness of human drugs. There are pre-market and post-market processes in place that are designed to ensure that human drugs are safe and effective, but no concrete evidence of improvements in these areas. It could be argued that Health Canada's authority extends only to helping to ensure that products available on the Canadian market are safe and effective.

Finally, in the short term, Health Canada's activities are expected to lead to increased industry compliance with regulatory requirements. The available data suggest that serious non-compliance is relatively uncommon. Over the period of the evaluation, however, Health Canada has not reported regularly or consistently on the nature, seriousness, frequency, or prevalence of non-compliances related to human drugs, and has focused its reporting, instead, on quantifying activities and outputs. Furthermore, much of Health Canada's compliance data, including GMP, Good Clinical Practice (GCP), and GVP compliance data, is aggregated across multiple product categories, encompassing human drugs and biologic (and veterinary drugs, in the case of GMP).

Recommendation 6:

Health Canada should continue to build on its current approach to compliance reporting by increasing its emphasis on compliance and enforcement outcomes, and enhancing its ability to report on compliance by product category.

Intermediate outcomes

In the intermediate term, Health Canada activities are expected to lead stakeholders to adopt safe behaviours with respect to the use of human drugs. While there is evidence from the literature that unsafe practices such as abuse of prescription pharmaceuticals, are occurring in Canada, the extent to which Health Canada's activities may influence these practices is unknown. Health Canada's ongoing evaluation of the effectiveness of its risk communications may provide insights into the degree to which its activities have influenced stakeholder behaviour. In this context, it is important to note that the Practice of Medicine, which is regulated by the provinces and territories, also influences stakeholder behaviour.

Health Canada activities are also expected to result in increased use of scientific evidence and risk-benefit analysis to inform decision making related to human drugs. The use of scientific evidence and risk-benefit analysis is formally integrated into Health Canada's Decision-Making Framework for Identifying, Assessing and Managing Health Risks, and also into various pre-market and post-market processes. Health Canada has established a number of Expert and Scientific Advisory Panels to provide guidance on regulatory and policy development, and has implemented some, though not all, of their recommendations.

In the intermediate term, Health Canada hopes to achieve a timely response to identified risks related to human drugs. Recognizing that policy and regulatory development is often a lengthy process, the evaluation found some instances in which Health Canada's response has taken longer than expected to implement. For example, more than a decade elapsed between Health Canada's initial announcement of its intent to introduce GMP and DEL requirements for APIs and the regulatory amendment. As another example, Health Canada committed to increasing the transparency of clinical trial information at least as early as 2007, and implemented a public database of clinical trial information in May 2013.

Analysis of Health Canada's signal data revealed a number of areas in which delays may occur. The signal data show that although Health Canada uses many sources to detect safety signals relating to human drugs, including sponsor data, the two most common sources of safety signals are the scientific literature and other regulatory agencies. Since it can take a considerable amount of time for the results of scientific studies to be published and for other regulators to complete their safety assessment, this may have negative implications for the timeliness of Health Canada's response. Health Canada's current efforts to enhance adverse reaction reporting and analysis may go some way toward addressing this problem, since this may allow it to be less reliant on external studies.

The timeliness of the signal assessment process itself has improved since the introduction of service standards for signal assessment. In 2011-2012, 93% of post-market signal assessments for pharmaceuticals, medical devices, biologics, and natural health products were completed within the service standard of 130 days. However, signal assessment is only one aspect of Health Canada's overall response to safety signals. The evaluation found that delays can occur at other points during the process. In particular, a considerable period of time can elapse between when signals are first detected and when they are assigned for assessment, and between approval of the signal assessment and the posting of a risk communication.
This finding is consistent with the 2011 report of the OAG, which noted that the Department's assessment of, and response to, potential safety signals is not timely. While most safety assessments examined by the OAG as part of its audit were completed within established timelines, the Department's approach to measuring its performance did not consider the amount of time a potential safety issue may wait before an assessment begins; the amount of time an assessment may be placed on hold; the amount of time needed to obtain additional information from external parties (such as the manufacturer); and the total number of calendar days, instead of working days, taken to complete the assessment. The OAG noted that when these factors were considered, Health Canada took at least one year to complete 34 of its 54 assessments and in some cases took significantly longer - two or three years.

Program representatives reported that developing the risk communication typically requires a considerable amount of negotiation with the market authorization holder, and posting of the communication may be delayed until appropriate changes have been made to the product labelling. As a result of these factors, the time elapsed from signal detection to posting of a risk communication can be quite lengthy. In 14 cases for which complete data were available (representing signal assessments completed between 2010 and 2012), the total time elapsed between signal detection and posting of the risk communication ranged from 232 days to 1,481 days (4 years), with a median of 1.4 years. As noted, Health Canada is currently reviewing existing performance standards for the content development of risk communications and the dissemination of risk communications.

Recommendation 7:

Health Canada should take steps to improve the timeliness of its response to safety signals (from signal detection to posting of a risk communication).

In the intermediate term, Health Canada expects to achieve increased international harmonization of regulatory requirements for human drugs. Among many other international engagements, Health Canada is an official observer to and participant in the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH); has developed MRAs on GMP compliance with a number of other jurisdictions; has formal information-sharing agreements with the FDA, Australia's Therapeutic Goods Administration (TGA), and the European Community; and participates with the FDA in the RCC, the overall goal of which is to better align the two countries' regulatory approaches. As already noted, Health Canada is also a member of PIC/S. International key informants view Health Canada as a constructive participant in bilateral and multilateral engagement.

Long-term outcomes

In the long term, Health Canada's activities will likely contribute to reduced health risks and adverse events associated with the use of human drugs. The available survey data show a high level of confidence among consumers in the safety of drugs and the regulatory system, though information specific to human drugs is not available.
Ultimately, Health Canada hopes to achieve a sustainable, cost-efficient, responsive, and science-based regulatory system for human drugs in Canada. The evaluation found that Health Canada uses scientific evidence and consults with stakeholders in policy and regulatory development, and recent updates to user fees for human drugs will support the sustainability of the regulatory system for these products.

Performance - efficiency and economy

Changes in HPFB's approach to financial reporting over the period of the evaluation made it challenging to consistently match HDP expenditures and budgets, so as to compare and analyze this information over time. HPFB has recently restructured its financial reporting to comply with Treasury Board requirements, which should improve the accuracy of this information and facilitate future analysis.

While activity-based financial reporting has not taken place since 2008-2009, within TPD coding has been updated as of 2012-2013 so that functional activity information will be available in the future. Activity-based reporting is important to support activity-based costing, which in turn is important to analysing the efficiency with which program activities are carried out. HPFB undertook an activity-based costing exercise in 2007 to support the proposal for updated user fees, and used the results of this analysis to calculate unit costs for a variety of regulatory activities, including submission review. The available data indicate that the timeliness of submission review for most classes of human drugs has been improving under the new cost recovery framework. However, without analysing the unit costs of submission review and other cost-recovered activities, it is unclear if improvements in timeliness also represent improved efficiencies. In addition to enabling an assessment of the extent to which efficiencies may have been realized under the new cost recovery framework, such analysis would also assist the HDP in identifying where future adjustments to the framework may be necessary. To this end, HPFB is reviewing its costs, fees, and performance associated with the HDP, as per its commitment in the User Fee Proposal and the Regulatory Impact Analysis Statementassociated with the Fees in Respect of Drugs and Medical Devices Regulations.

Management Response and Action Plan
Human Drugs Program Evaluation ^{Footnote 1}
Recommendation	Response	Key Activities	Deliverables	Responsible Directorate	Timeframe
1. Consistent with international trends and its commitment to enhancing transparency and openness under regulatory modernization, Health Canada should further enhance the amount of clinical trial information it makes publicly available, including the results of clinical trials.	Agree	The department has advanced numerous initiatives to deliver on its commitment to make administrative information about clinical trials more accessible such that patients have adequate information to support their health decisions. In 2007, Health Canada (HC) published a Notice to sponsors encouraging voluntary registration of clinical trials in public registries recognized by the World Health Organization (WHO). The 2007 Notice was updated in 2012 to communicate that HC continues to support clinical trial transparency through registration of study protocols. HC also revised its No Objection Letter (authorization issued to clinical trial sponsors) in November 2011 to include a reminder about registration of all trial phases in public registries. In May 2013, Health Canada published the revised version of the Guidance Document Considerations for Inclusion of Women in Clinical Trials and Analysis of Sex Differences and made it available to stakeholders involved in drug development and research. Since the closing of this evaluation, in May 2013, the Department launched a database of authorized drug clinical trials. The database is a central source of high level information (protocol number, protocol title, drug name, medical condition, study population, authorization date, sponsor name, HC control number, and the start and end date of the clinical trial, if known) about Phase I, II and III clinical trials in patients. Between April 1, 2013 and October 22, 2013, 328 trials were posted by the Department. The database helps fill an existing information gap revealed by a 2011 survey of international registries that only 24% of clinical trials authorized by HC are publically registered; and of clinical trials in patients, only about 50% were registered. In addition, HC published Clinical Trials Application Guidance for Sponsors. HC also worked with the Canadian General Standards Board (CGSB) to create a new voluntary standard for Research Ethics Boards (REBs), encouraging REBs to make their Standard Operating Procedures publicly available. This standard is available through the Public Works and Government Services Canada website, at http://www.tpsgc-pwgsc.gc.ca/ongc-cgsb/publications/nouvelles-news/nncvcb-ncsreo-eng.html.	TPD will be developing a plan which will outline potential future enhancements to the database.	TPD	June 2014
		Regarding clinical trial results, currently, the Food and Drug Regulations do not oblige sponsors to provide the results of all clinical studies conducted throughout the drug's development unless there are issues that warrant reporting. In addition, sponsors who apply for market authorization may be required to submit additional clinical trial results if HC deems that the information provided in the drug submission is unsatisfactory or if reviewers are aware of other studies that could inform their decision-making. In certain circumstances, additional data will be requested in support of the drugs' safety, efficacy and quality. To date there is no international standard for the publication of results (i.e. content and format). As discussions on posting of study results evolve internationally, these developments will inform our future policy and regulatory work.
2. Health Canada should improve its information systems for post-market surveillance and monitoring. This should include: fully implementing information systems to support electronic submission of domestic and foreign adverse reaction reports, as well as systematic monitoring and analysis of these reports through targeted surveillance and data-mining, developing and implementing a comprehensive and centralized approach to information management for post-market surveillance activities. In particular, this should include a centralized mechanism for tracking signal activities, including Health Canada's response to the recommended actions arising from completed signal assessments.	Agree	With the assistance of the Canada Vigilance Expert Working Group, Health Canada is implementing electronic reporting of adverse reactions for industry and clinical trial sponsors. The reporting will include post-market and clinical trial adverse reaction information in accordance with international and Health Canada standards and will improve the department's ability to actively assess trends in pharmaceutical and human drugs adverse reactions both domestically and internationally. This will constitute one of the tools that will contribute to more active surveillance of post-market safety issues. The department continues to improve its strategies to monitor adverse reaction reports through targeted surveillance and improved data-mining (statistical analysis) techniques. Targeted surveillance in this context refers to focusing surveillance efforts on areas that have been identified as being high-risk, based on the identification of a potential safety issue through signal assessments and adverse reaction reports received. The Health Canada Vigilance Framework (available on the department website) identifies a number of targeted monitoring strategies for: Health product and targeted medical events (HP-TME): monitoring of specific health product and adverse reaction where a potential safety issue has been noted but more information is needed for confirmation;	In order to ensure robust post-market surveillance and monitoring information systems, the Health Products and Food Branch will develop a proposal on requirements relating to: improvement of existing tools; creation of new tools; creation of partnerships to leverage data from external stakeholders including international regulators, industry and provincial partners such as health care institutions.	MHPD
		Designated Medical Events (DME): monitoring for reports of specific adverse reactions that are often health-product related and potentially disabling or life-threatening such as serious skin reactions or liver failure; New Active Substances (NAS): monitoring of health products with an active ingredient that has not been previously marketed in Canada;		MHPD	February 2014
		The targeted surveillance and data-mining initiatives are relatively recent and still in progress. The Department is developing a plan for ongoing improvements to these strategies; this is a significant element in the transition from passive to a more active surveillance approach. In the October 2013 Speech from the Throne, the government committed to introducing new patient safety legislation that would include new post-market powers. On December 6, 2013, the Minister of Health tabled legislation in Parliament which proposed amendments to the Food and Drugs Act including a requirement for healthcare institutions to report serious adverse drug reactions. Health Canada will work with provinces and territories to create an efficient new reporting system by building on existing systems and best practices.		MHPD	December 2014
3. Health Canada should examine whether there is an ongoing rationale for pursuing post-market regulatory authorities to which it committed under the Food and Consumer Safety Action Plan (FCSAP), namely, the authority to compel manufacturers to submit risk management plans and periodic safety update reports.	Agree	In 2008 the government introduced Bill C-51 which included the authority to establish terms and conditions on a licence to sell a drug. The compulsory provision of Risk Management Plans (RMPs) and Periodic Safety Update Reports (PSURs) would have been supported by these provisions however the bill died on the order paper with the election call in 2008. In 2009 Health Canada adopted ICH E2E: Pharmacovigilance Planning and instituted processes to facilitate the voluntary submission of risk management plans. At that time, the Department also committed to developing an enabling pharmacovigilance framework through regulatory modernization which would support full implementation of risk management planning. In 2011, Health Canada amended the Food and Drug Regulations(FDR) to require manufacturers to prepare an annual summary report of all information relating to adverse drug reactions and serious adverse drugs reaction, and to notify the Minister of any significant safety changes identified in that report. The Minister can request a copy of the annual summary report and asks that it be submitted in accordance with ICH standards for PSURs. Outside of these requirements Health Canada only receives PSURs on a voluntary basis. New authorities are needed to further compel manufacturers to submit PSURs more routinely as a condition of market authorization. In the October 2013 Speech from the Throne, the Government committed to introducing new patient safety legislation to help identify potentially dangerous drugs. On December 6, 2013, the Minister of Health tabled legislation in Parliament which provides new regulatory authorities to enable terms and conditions on authorized products for post-market requirements, such as Risk Management Plans. These terms and conditions, if not complied with, will be subject to the new fines and penalties for therapeutic products.	Tabling in Parliament patient safety legislation which includes enabling regulatory authorities to compel post market safety data, such as risk management plans and periodic safety update reports.	PPIAD	December 2013 (Completed)
4. Health Canada should explore options for improving its oversight of foreign manufacturing sites.	Agree	Given the global nature of the drug manufacturing business and because no country can inspect all foreign manufacturing sites, countries have to work together to protect their citizens and help ensure medicines sold in their countries are safe no matter where they are produced and used. Health Canada leverages inspections by trusted regulatory partners of facilities abroad that produce medicines sold in Canada; they in turn rely on our inspections of Canadian facilities that do business overseas. However, when information or inspection results from formal Mutual Recognition Agreement partnerships, Pharmaceutical Inspection Convention and Pharmaceutical Inspection Co-operation Scheme member states or other sources of information cannot be leveraged, Health Canada uses a facility/product risk-based model when selecting sites for conducting foreign inspections.	1 Health Canada will complete at least three API inspections of foreign sites, which will be chosen based on a combination of timing, lack of alternative evidence and risk associated with the activities at or products from the facility.	Inspectorate	1 December 2015
		Health Canada also conducts on-site GMP inspections of foreign facilities if there are serious questions about manufacturing practices, even if the site has been inspected by a trusted partner. Drug products cannot be imported into Canada from a foreign site unless there is satisfactory evidence available to Health Canada to demonstrate that Canadian GMP standards are being met. Approved foreign sites from which medicines can be imported are identified on the Establishment Licence of a Canadian importer.	2 As part of RCC commitment, complete an analysis of the phase I observational inspections and prepare a report on the outcome of the FDA/HC observational inspections.	Inspectorate	2 March 2014
		Health Canada has introduced an amendment to the Food and Drug Regulations to increase oversight of Active Pharmaceutical Ingredient (API) manufacturers and importers in the drug supply chain. These regulations came into force on November 08, 2013. This will enhance Health Canada's ability to trace back to the foreign sites if problems related to Good Manufacturing Practices (GMPs) arise. An API Inspection programme was launched in November 2013. With respect to foreign manufacturers located in MRA countries, Health Canada has launched discussions to expand the scope to include APIs which would allow Health Canada to exchange Certificates of Compliance for GMP inspections of API sites carried out by health regulators from MRA countries. Health Canada is working with the US FDA under the Regulatory Cooperation Council (RCC) initiative to avoid duplicative inspections and to enhance collaboration on compliance and enforcement. This initiative has resulted in a stronger relationship with the US FDA which allows for the exchange of compliance and enforcement information pertaining to sites of common interest (nationally and internationally). Health Canada is working with the Australian TGA as part of the Regulatory Cooperative Initiative (RCI), to benefit from sharing information on foreign sites of common interest. Phase I of this project has recently been completed and a Phase II has been discussed and is being finalised.	3 As part of the phase II RCI commitment, complete an analysis of the feasibility of TGA's foreign site centric model by which Health Canada could approve foreign manufacturing sites for importation of drugs into Canada.	Inspectorate	3.December 2014
5. Health Canada should examine whether there is an ongoing rationale for pursuing compliance and enforcement measures to which it committed under the FCSAP, namely the authority to issue mandatory recalls of therapeutic products and the authority to levy increased fines for noncompliance.	Agree	Since the commitment under the FCSAP, Health Canada has undertaken a series of activities that support the ongoing rationale for pursuing compliance and enforcement measures, namely the authority to issue mandatory recalls. In spring 2008, the Government introduced Bill C-51 which included a new authority for mandatory recall. Stakeholder engagement related to the bill supported the need for the new authority. The bill died on the order paper with the 2008 election. In January 2011, Health Canada hosted a series of technical discussions with stakeholders on regulatory modernization for therapeutic products (drugs and medical devices) to test and validate proposed activities related to the regulation of a product throughout its lifecycle. The third session in a series of three, examined post authorization activities, including mandatory recall. In April 2013, an independent review was requested by the Government following the Alysena 28 recall. The report called for strengthening Health Canada's authority in responding to the hazards of drug products. The Government accepted the report's recommendations and committed to exploring expanded recall powers.	Tabling to Parliament patient safety legislation which proposes authorities to issue mandatory recalls of therapeutic products and the authority to levy increased fines for non-compliance.	PPIAD	December 2013 (Completed)
		In the October 2013 Speech from the Throne, the Government committed to introducing new patient legislation to help identify potentially dangerous drugs and ensure the quick recall of unsafe drugs. On December 6, 2013, the Minister of Health tabled legislation in Parliament which includes an authority that permits the Minister to order a person selling a therapeutic product to recall it or have it sent to a place that is specified by the Minister, when he/she is of the opinion that it presents serious or imminent risk to health. The proposed legislation also includes enhanced fines and penalties to better reflect the serious nature of the violation. The maximum penalty increases from $5,000 to $5,000,000 or 2 years in prison.
6. Health Canada should continue to build on its current approach to compliance reporting by increasing its emphasis on compliance and enforcement outcomes, and enhancing its ability to report on compliance by product category.	Agree	Health Canada (HC) currently uses the overall compliance rating (i.e. % compliance) of industry as an outcome based performance indicator to measure and assess the results of the Inspectorate's outputs (i.e. inspection targets, # of incidents opened/closed etc.) in achieving program objectives. This performance measure is reported on monthly and included in the HC Dashboard, as well as used in the Departmental Performance Report (DPR), Report on Plans and Priorities (RPP) and Performance Measurement Framework (PMF). Since the period reviewed in this evaluation, HC has also developed an Annual Inspection summary report for external publication. The report describes in detail the overall compliance rate of industry per program and lists the common observations cited in non-compliant establishments. It also provides specific examples of observations cited against the Food and Drugs Act and Regulations and provides links between inspections conducted by activity (i.e. manufacturer, importer, distributor, etc.), observations noted and their associated risk category. The aim of publishing this report is to address many of the performance outcomes in the PMF, including "Increased awareness and understanding among industry of Health Canada's regulatory framework", and "Increased industry compliance with Health Canada's regulatory requirements.	Annual inspection summary reports prepared for publication.	Inspectorate	March 2014
7. Health Canada should take steps to improve the timeliness of its response to safety signals, (from signal detection to posting of a risk communication).	Agree	The service standard for the completion of signal assessments is 130 days and the target is 90%. Health Canada consistently meets this target. In fact the average completion time for a signal assessment is 90 days. Health Canada makes ongoing improvements to the timeliness of its response to safety signals. The department is committed to timely completion of high quality safety assessments and risk communications. With the implementation of performance standards and targets for post-market surveillance review work in 2011, the department is able to monitor and implement strategies to improve the timeliness of safety assessments and related risk communications. These standards and targets are subject to periodic review for relevancy and accuracy. The department is also reviewing the feasibility of increasing the use of foreign review work which should decrease the overall time taken to conduct safety assessments and issue risk communications. This is another element in the transition from a passive to a more active surveillance model. The department developed a process for more proactive identification of major labelling updates requiring the issuance of a risk communication. The Department is also conducting an evaluation of the effectiveness of health product risk communications.	The Department is developing a strategy to move from a passive surveillance model to an active surveillance model for the management of health product risks. This strategy should enable more timely dynamic safety reviews and responses as well as publication of safety reviews.	MHPD	June 2014

1.0 Introduction

The Human Drugs Program (HDP) is managed through the Health Products and Food Branch (HPFB) of Health Canada. The main program partners involved in delivering the HDP are the Therapeutic Products Directorate (TPD), the Marketed Health Products Directorate (MHPD), the Health Products and Food Branch Inspectorate (the Inspectorate), the Regions and Programs Bureau (RAPB), the Policy, Planning, and International Affairs Directorate (PPIAD), and the Resource Management and Operations Directorate (RMOD). Program activities include creating and maintaining the regulatory framework, interacting and communicating with partners and stakeholders, performing risk-benefit assessments, conducting post-market monitoring and surveillance, and compliance and enforcement of the regulatory framework.

1.1 Organization of the report

The report is divided into several sections. Section 2 provides a profile of the HDP, and Section 3 describes the evaluation methodology. Sections 4 through 7 provide the evaluation findings pertaining to relevance, governance and implementation, achievement of outcomes, and efficiency and economy. Section 8 concludes and presents recommendations. Three appendices accompany the main report. Appendix A contains the evaluation matrix; Appendix B contains the list of references; and Appendix C contains supplementary data tables.

2.0 Profile of the Human Drugs Program

The HDP operates under the authority of theFood and Drugs Act. The Act defines a drug as "any substance or mixture of substances manufactured, sold, or represented for use in:

the diagnosis, treatment, mitigation or prevention of a disease, disorder or abnormal physical state, or its symptoms, in human beings or animals;
restoring, correcting, or modifying organic functions in human beings or animals; or
disinfection in premises in which food is manufactured, prepared, or kept." (GoC, 2008b, sec. 2).

The HDP encompasses human drugs (also known as pharmaceutical drugs), which includes prescription and non-prescription drugs intended for human use, excluding biologic-related drugs, vaccines, blood and blood products, cells, tissues, and organs, which fall within the scope of HPFB's Biologics Program. The ultimate objective of the HDP is to provide access to safe and effective human drugs and information for healthy choices to assist Canadians in maintaining and improving their health (Health Canada, 2012l).

2.1 Roles and responsibilities of program partners

The main partners in the HDP are the TPD, the MHPD, the Inspectorate, and the RAPB. Other partners involved in program delivery and/or management include the PPIAD and the RMOD. The roles and responsibilities of each of these partners are briefly described below.

2.1.1 Therapeutic Products Directorate

The TPD is responsible for regulating pharmaceuticals for human use under the authority of the Food and Drugs Act and Regulations. More specifically, TPD is responsible for the following:

developing new regulations, policies, guidance, and Standard Operating Procedures (SOPs), and maintaining/updating existing documents
conducting pre-market review of product submissions to determine safety, efficacy, and quality of these products, and the appropriateness of their labelling, as well as authorizing their sale in Canada
reviewing clinical trial applications, ensuring proper design and no undue risks to participants, and conducting surveillance of adverse reactions in clinical trials
evaluating post-market adverse events, complaints, and reports of problems (including performing Health Risk Assessments) to support the post-market activities of the MHPD and the Inspectorate
reviewing information that product manufacturers provide health care practitioners and consumers about products (e.g., product monographs and package inserts)
providing science-based information about these products to the Canadian public

The TPD is also responsible for allowing doctors access to both non-marketed human drugs and biologics through the Special Access Program when conventional therapies have failed, or are unsuitable or unavailable.

2.1.2 Marketed Health Products Directorate

The MHPD "works to assure that HPFB programs take a consistent approach to post-approval safety surveillance, assessment of signals and safety trends and risk communications concerning all regulated marketed health products" (Health Canada, 2012g). Responsibilities relevant to human drugs include the following:

managing the Canada Vigilance Program, Health Canada's post-market surveillance program that collects and assesses reports of suspected adverse reactions to health products marketed in Canada
monitoring adverse reaction and medication incident data collected through the Canada Vigilance Program
conducting comprehensive safety assessments, such as signal assessments, benefit-risk assessments and reviews for Risk Management Plans (RMPs) for marketed health products, as well as reviews for RMPs for certain health products submitted for authorization
monitoring regulatory advertising activities
communicating health product risks to the public and health care practitioners
developing policies and strategies for post-market surveillance

2.1.3 Inspectorate and RAPB

The main role of the Inspectorate is "to deliver a national compliance and enforcement program for all products under the mandate of the Branch, with the exception of food products that are the responsibility of the Canadian Food Inspection Agency" (Health Canada, 2011b).With respect to human drugs, the Inspectorate carries out the following activities:

developing policy related to compliance and enforcement of regulations
licensing drug establishments that are compliant with regulatory requirements and amending, suspending, or refusing to issue Drug Establishment Licences (DELs) when non-compliances are found
conducting monitoring inspections to ensure manufacturer compliance with regulatory requirements Good Clinical Practices (GCPs), Good Manufacturing Practices (GMPs), and Good Pharmacovigilance Practices (GVPs)^{Footnote 2}
conducting compliance verifications in response to specific complaints or identified risks relevant to human drugs and clinical trials
providing chemical, physical, and microbiological analysis services through the Laboratory Program to support inspection and investigation activities
conducting compliance promotion activities with industry stakeholders

With respect to controlled substances, the Inspectorate is also responsible for security inspections to ensure manufacturer compliance with regulatory requirements related to controlled drugs and substances, and for oversight of the destruction of substances subject to the Controlled Drugs and Substances Act (CDSA).

RAPB is the operational arm of Health Canada in the regions. Responsibilities include compliance promotion activities, as well as enforcement of laws and regulations through inspections, investigations, legal action, and evaluations of compliance with standards affecting manufacturing, packaging and labelling, analysis, importing, distributing, and wholesaling of consumer health products.

2.1.4 Other partners

Two other Health Canada partners also have responsibilities under the HDP.

PPIAD provides leadership and support on strategic planning, as well as policy development and planning on horizontal issues of strategic importance. It also provides leadership with respect to international affairs.
RMOD is responsible for resource allocation, including cost recovery policy and implementation, as well as rolling up performance measurement information to generate Branch and Departmental performance reports. These responsibilities were transferred to RMOD from PPIAD in July 2011.

2.2 Program activities and logic

The HDP consists of five main activities delivered by the program partners:

development and implementation of the regulatory framework
communication and stakeholder engagement
risk and benefit assessments related to clinical trial authorization and market authorization
post-market surveillance and monitoring
compliance and enforcement of the regulatory framework

These activities correspond with specific immediate, intermediate, and long-term outcomes. In the immediate term, program activities are expected to lead to increased external stakeholder awareness and understanding of risks and benefits related to human drugs. They are also expected to improve industry awareness and understanding of Health Canada's regulatory framework for these products, to enhance the safety and effectiveness of these products, and to increase industry compliance with the relevant regulatory framework.

The achievement of these immediate outcomes is expected to lead to intermediate outcomes of external stakeholder adoption of safe behaviours associated with human drugs; increased use of scientific evidence and risk-benefit analysis by Health Canada to inform decision making; timely regulatory response to identified risks; reduced exposure to health risks associated with the use of these products; and harmonization of Canada's regulatory framework with international approaches.

In the long term, Health Canada expects to reduce adverse events associated with the use of human drugs; increase public confidence in human drugs and the related regulatory system; and produce a sustainable, cost-efficient, responsive, and science-based regulatory system for these products in Canada.

A logic model that depicts the linkages between the activities, outputs, and expected outcomes of the HDP is describe below in (Section 2.3).

2.3 Description of the Logic Model^{Footnote 3}

The Human Drugs Program (HDP) uses the following resources (inputs) to deliver its activities, produce outputs and accomplish its outcomes: funding; human resources; facilities, infrastructure; Acts, regulations, policies, priorities; science and technology; and, research data.

The HDP consists of five main activities delivered by the Program partners, namely:

development, implementation and maintenance of the regulatory framework
communication with partners and stakeholders
risk and benefit assessments related to clinical trial authorization and market authorization
post-market surveillance and monitoring
compliance and enforcement of the regulatory framework

These activities are targeted at different groups, namely:

development, implementation and maintenance of the regulatory framework
- industry; media; consumers; international governments and organizations; health care practitioners; academia; Federal/Provincial/Territorial (F/P/T) governments; and, industry and professional associations
communication with partners and stakeholders
- industry; media; consumers; international governments and organizations; health care practitioners; academia; F/P/T governments; and, industry and professional associations
risk and benefit assessments related to clinical trial authorization and market authorization
- industry; international organizations; academia; health care professionals; and, F/P/T governments
post-market surveillance and monitoring
- international and Health Canada regulators; academia; and, science/research organizations
compliance and enforcement of the regulatory framework
- industry; Health Canada, F/P/T and international regulatory stakeholders and partners; and, consumers
industry; Health Canada, F/P/T and international regulatory stakeholders and partners; and, consumers

As a result of each activity, the Program generates a number of products and/or services, namely:

development, implementation and maintenance of the regulatory framework
- legislation and regulations; policies; guidance; Mutual Recognition Agreements (MRAs); and Memoranda of Understanding (MoUs)
communication with partners and stakeholders
- data sources/systems, websites; research findings; public and stakeholder educational material and events; publications, notifications, advisories (including Dear Health Care Practitioner Letters); and, correspondence/public enquiries
risk and benefit assessments related to clinical trial authorization and market authorization
- risk-benefit assessments (initial and on-going, including Health Risk Assessments); market authorizations; establishment licences; Clinical Trial Authorizations (CTAs); and, Special Access Program decisions
post-market surveillance and monitoring
- Canada Vigilance Program; data, analysis, evidence; reports, regulatory recommendations; signal identification and characterization; epidemiology studies; and, Risk Management and Mitigation Plans
compliance and enforcement of the regulatory framework
- compliance and enforcement reports; and, compliance and enforcement actions, including prosecution, licence suspension, seizures, stop sale and search, etc.

The ultimate outcome of the Program is to help ensure access to safe and effective human drugs and provide information for healthy choices.

2.4 Resources

The HDP is funded by Health Canada, and by industry through cost recovery. In 2011-2012, total HDP expenditures were $99.8 million. Revenues from cost recovery were $38.7 million.

2.5 Program context

Management and delivery of the HDP, like all federal regulatory programs, is influenced by a number of overarching policies, directives, and pieces of legislation, which provide context for such programs. Regulations are one of a number of government instruments to achieve policy outcomes. The 2012 Cabinet Directive on Regulatory Management requires regulatory programs to demonstrate efficiency and effectiveness by ensuring that the benefits of regulation justify the costs and by demonstrating tangible results for Canadians (GoC, 2012a). Inherent in the Directive is the need to consult with affected bodies at all stages of the regulatory process; to select the appropriate mix of government instruments, including regulatory and non-regulatory responses; to impose the least possible cost on business and Canadians that is necessary to achieve the policy objectives; to take advantage of opportunities for coordination and collaboration with provincial/territorial governments and internationally, including limiting the number of specific Canadian regulatory requirements and minimizing differences with key trading partners (i.e., the US); and to measure and report to Canadians in a timely manner on the performance of regulatory programs.

In May 2012, HPFB released the Regulatory Roadmap for Health Products and Food (Health Canada, 2012j). The Roadmap envisions an efficient and transparent regulatory system that protects and improves consumer safety, reduces regulatory burden for small business, supports scientific innovation, and increases the variety of health options and benefits available to Canadians. These goals are consistent with the Cabinet Directive on Regulatory Management, which requires federal departments to consider the impact on small businesses of new or amended regulations, while protecting and advancing the public interest in health and safety. These goals are also guided by the Government of Canada's Red Tape Reduction Action Plan, which aims to reduce administrative burden on business.

Regulatory programs that impose a user fee, such as the HDP, are subject to the User Fees Act (GoC, 2004).The Actrequires regulating authorities to consult with clients and service users and establish service standards comparable to those in other countries prior to amending fees or creating new ones.

The Communications Policy of the Government of Canada requires federal institutions to, among other things, provide the public with "timely, accurate, clear, objective and complete" information about policies, programs, services and initiatives in all regions of Canada and in a range of formats, and to consult the public when establishing priorities and planning programs and services (GoC, 2012b).

3.0 Evaluation methodology

This section of the report describes the evaluation methodology.

3.1 Evaluation design and methods

The evaluation design was developed based on the findings of an evaluability assessment completed as a first step in the evaluations. The evaluability assessment consisted of a preliminary review and assessment of available documents and administrative data.

The evaluation design and the evaluation matrix (Appendix A) were developed based on the evaluability assessment. The evaluation matrix addresses 10 key questions and a number of sub-questions relating to program relevance and performance (effectiveness, efficiency, and economy), in accordance with the Treasury Board's Policy on Evaluation.

The evaluation consisted of several data collection methods. Some of the data collection methods, in particular the survey of stakeholders, the key informant interviews, the case studies, and the administrative data review, were designed to capture information to support one or both of two concurrent evaluations of other HPFB regulatory programs, namely, the Biologics Program and the Medical Devices Program.

Literature review. The literature review addressed evaluation questions related to program relevance, long-term outcomes, and alternate approaches. Peer-reviewed (i.e., scientific and academic) and grey literature was considered in the review. Relevant literature was located through online searches.

Document review.The document review addressed all of the evaluation questions to the extent that supporting documents were available. The reviews encompassed government documents, primarily produced by Health Canada, related to HDP planning, management, and ongoing operations. Several thousand documents were reviewed as part of the evaluations.

Administrative data review. The administrative data review addressed evaluation questions related to program outcomes. The review considered data produced by the TPD, MHPD, and the Inspectorate. Although theoretically distinct, the document review and the administrative data review were, in practice, two aspects of the same task, as the majority of administrative data was included in program documents.

To support the literature, document, and administrative data reviews, a library of approximately 3,500 individual documents, sources of administrative data, and pieces of literature was developed, including approximately 1,650 documents; 980 web pages; 570 journal articles; and various other types of sources, which were reviewed over the course of the evaluation.

Case studies.Two case studies were conducted to support the HDP evaluation. One case study focused on Health Canada's approach to clinical trials. Although completed in the context of the HDP evaluation, because the same regulatory framework applies to biologics as well as human drugs, the clinical trials case study was also used to inform the evaluation of the Biologics Program.

The other case study examined Health Canada's signal detection, signal assessment, and risk management activities for both human drugs and biologics, and like the clinical trials case study, was used to inform both evaluations.

Survey of industry. The bilingual survey of industry used a web-based approach and focused on evaluation questions related to outcomes. Guidance and direction from Public Works and Government Services Canada on public opinion research and surveys limited the evaluation to surveying individuals who were known to have had contact with Health Canada for reasons related to the HDP. Initially, the scope was to include all manufacturers currently licenced to manufacture human drugs in Canada (which was the approach taken in the evaluation of the Medical Devices Program). Instead, the sample was drawn by Health Canada from the Department's Stakeholder Information Management System (SIMS) database, and likely does not represent a random or representative draw from the population of manufacturers operating in Canada.

After cleaning to remove duplicates and entries for which no email address was available, the final sample for the human drugs industry survey consisted of 380 manufacturers. These 380 manufacturers included only those that manufactured exclusively human drugs and did not include those that manufacture biologics.

The human drugs industry survey achieved 24 completions, representing a completion rate of 6.6%.^{Footnote 4} Although a 10% response rate is not uncommon for web-based surveys of industry, this response rate is lower than anticipated and may be due to the fact that Health Canada did not send an initial communication to potential respondents. Instead, the survey invitation was disseminated by the evaluator.

Survey of other stakeholders. A bilingual web-based survey was also used to reach users of human drugs, including health care professionals and patients/consumers. This survey was also intended to capture outcome information from users of biologics and medical devices to support concurrent evaluations of Health Canada's regulatory activities in those areas. In particular, the survey was intended to capture information on the impact of Health Canada's communication and consultation activities on stakeholder awareness and understanding of risks related to therapeutic products, as well as information on the impact of these activities on stakeholder use of these products.

As was also the case with the industry survey, the evaluation was limited to surveying individuals who were known to have had contact with Health Canada. Accordingly, the evaluation requested to use Health Canada's MedEffect listserv, which consists of over 20,000 subscribers to MedEffect's e-notice, as the survey sample. In light of concerns about privacy, the possibility that use of the listserv for survey purposes might cause subscribers to unsubscribe to MedEffect, and the fact that the listserv includes individuals who are not within the survey's target group (such as Health Canada employees and health care professionals and consumers residing outside of Canada), however, the listserv was not made available.

To support the survey, Health Canada provided a list of stakeholders identified by Health Canada through its SIMS database. After cleaning the sample, the final sample consisted of 651 potential respondents.

The questionnaire included separate modules for medical devices, human drugs, and biologics. A total of 16 stakeholders responded to the survey, representing a completion rate of 2.6% overall.^{Footnote 5} Of these, 10 (1.6% of potential respondents) responded to the module relating to human drugs. While the reasons for this low response rate are not clear, they may include the lack of an initial communication from Health Canada.

External key informant interviews. The key informant interviews focused on evaluation questions relating to program implementation and effectiveness. The interviews addressed questions relating to medical devices, human drugs, and biologics in order to support all three evaluations without overburdening key informants with multiple requests for interviews. Interviewees include industry representatives, researchers and academics, patient and consumer organizations, health care providers, professional associations, international key informants, and others.

Overall, a total of 93 potential key informants were identified, and 53 individuals participated in an interview or provided a written response to the interview questions. Of these, 42 key informants made comments relevant to human drugs. Interviews were conducted by telephone in each key informant's preferred official language. The interviews were recorded to ensure accuracy, and the notes were returned to key informants for review and approval. Key informants were assured of the confidentiality of their responses.

For final reporting, data from all lines of evidence was integrated or triangulated in order to arrive at the overall evaluation findings. Triangulation is a process through which answers to research questions generated by different data collection methods are compared. Where different methods produced similar findings, those findings were assumed to have greater validity, and, therefore, greater confidence in the results is warranted.

3.2 Limitations of the methodology and mitigation strategies

There are several important methodological limitations to note. First, given the vast scope and complexity of the subject matter, the document review and the literature review were both limited by the time and resources available to complete them. The document review was further limited to documents that were provided to the evaluation by Health Canada, or that were publicly-accessible. Although the evaluation made every attempt to request relevant documents from Health Canada as the research proceeded, it was not in a position to identify all documents that might be relevant. Some of the responsibility for identifying relevant documents therefore lay with the program partners themselves.

The stakeholder survey was limited as a direct result of the sample development process. In particular, the sample for the survey was small, especially in comparison to the actual health care provider and patient/consumer populations. Furthermore, for reasons that are not entirely clear, the survey achieved a very low overall response rate (2.6%), and only 10 stakeholders responded to the component pertaining to human drugs. While the use of the MedEffect list might not have produced a higher response rate, it would have, at minimum, increased the number of completions to a level worth reporting. Thus, it would have provided some data to support outcome questions relating to the impact of Health Canada's risk communications on stakeholder awareness, understanding, and behaviour. In the absence of survey data, there is very little information to support conclusions on these questions. That being said, even if the MedEffect list had been used, the opinions and information collected by the survey would not have represented the general population of stakeholders in Canada. Rather, the opinions and information would have reflected a group of stakeholders who receive frequent communications from Health Canada.

Like the stakeholder survey, the industry survey was also limited by a small and unrepresentative sample, combined with a low response rate. As a result, survey findings are used sparingly throughout this report. When survey results are included, they are treated as qualitative information.

With respect to the external key informant interviews, approximately half of those who were identified ultimately did not participate in an interview. It is unknown if the interview findings would have been substantially different had these individuals participated. However, it is important to note that many of those who chose not to participate indicated that they were not familiar enough with the program to give meaningful responses. Like all key informant data, the findings from the interviews should not be interpreted as representing the views of stakeholders in general.

Although interviews with Health Canada representatives were planned, they ultimately did not take place. The interviews were intended as the final data collection activity, and were meant to provide the program with an opportunity to respond to the preliminary evaluation findings and address any gaps in information. Due to time constraints, the evaluation proceeded to final reporting without completing the interviews. However, throughout the process, the evaluation consulted with program personnel by means of conference calls and email to request additional information and clarify issues.

4.0 Findings - relevance

This section of the report presents the evaluation findings on relevance.

4.1 Ongoing need

The evaluation confirmed an ongoing need for government oversight of human drugs in order to protect the health and safety of Canadians. Use of these products is growing due to demographic changes such as population growth and aging, as well as marketing by industry. As a result, more Canadians will be exposed to the risks, as well as the benefits, of these products, suggesting a need for continued oversight by Health Canada to manage these risks. Moreover, trends such as the emergence of combination products and globalization of the supply chain are creating uncertainties that further support the need for government intervention to protect the health and safety of Canadians.

Pharmaceutical drugs and biologics contribute significantly to the health of Canadians and represent an important component of the Canadian health care system.^{Footnote 6} Spending on drugs has increased consistently since the 1970s, reaching $32.0 billion (16% of total system spending) in 2010 (CIHI, 2011, p. 22; Health Canada, 2011a). Approximately half of Canadians take at least one prescription drug (Health Council of Canada, 2010, p. 17), as do 76% of seniors in private households and virtually all of seniors in health care institutions (Ramage-Morin, 2009). Furthermore, use of these products grew by 77.6% between 1999 and 2009, from 272 to 483 million prescriptions per year, respectively (Health Council of Canada, 2010, p. 18). Increased per capita use of drugs, attributable in part to marketing by the industry, and combined with population growth and aging, suggests that more Canadians are being and will continue to be exposed to both the risks and the benefits associated with Health Canada-approved medications (Lemmens & Bouchard, 2007, p. 312).

A variety of risks to human health and safety are associated with the use of human drugs, implying that some degree of regulatory oversight is necessary in order to manage these risks.

Risks arising from incomplete understanding of a product's effect on human health. Despite pre-market processes instituted in Canada and other jurisdictions since the 1960s, safety concerns periodically result in approved products being withdrawn from the market. Since 2004 in Canada, there have been 28 brand-name and generic drugs (involving 22 active ingredients) inactivated by the manufacturer for safety reasons (Health Canada, 2013b).^{Footnote 7} Although it is difficult to identify estimates in the literature of how many Canadians have been affected by exposure to unsafe drugs, the numbers are likely to be significant. For instance, Vioxx may have been associated with as many as 4,000 to 7,000 deaths in Canada before being withdrawn from the market in late 2004 (Abraham, 2005).
Risks arising from off-label use. The key issue associated with off-label drug use is the absence of clinical evidence demonstrating the efficacy and safety of a medication that is prescribed for uses other than those for which the drug was approved. Off-label prescribing appears common in Canadian health care, with one recent study estimating that 11% of drugs prescribed in primary practice were prescribed for an off-label indication; in 79% of these cases, there was no strong scientific evidence of the efficacy of the products concerned with respect to their intended uses (Eguale et al., 2012). Although Health Canada has minimal regulatory authority over the useof drugs, it can nonetheless influence use indirectly through other channels, such as the preparation and dissemination of knowledge products. Under the Food and Drugs Act, it can also prosecute companies that promote off-label use.
Risks arising from abuse/misuse of prescription drugs. The abuse or misuse of prescription drugs is a form of off-label use that presents unique challenges, since these drugs are often important for chronic pain treatment. The types of prescription drugs with the most potential for misuse are opioid pain relievers (e.g., fentanyl, Percodan, Demerol and OxyContin), stimulants (e.g., Ritalin, Concerta, Adderall, and Dexedrine), and tranquilizers and sedatives (e.g., benzodiazepine, Valium, Ativan, and Xanax) (CCSA, 2012). Recently, Canada became the second largest per capita consumer of prescription opioids worldwide, an increase that is correlated with increases in opioid-related overdoses, deaths, and addictions (CCSA, 2012). One study found that opioid-related deaths in Ontario doubled from 13.7 per million in 1991 to 27.2 per million in 2004, and that a significant portion of this increase was associated with the addition of long-acting oxycodone (i.e., OxyContin) to the Ontario drug formulary (Dhalla et al., 2009). OxyContin was the eighth top-selling drug in Canada in 2011 (see Table 1).
Risks arising during manufacturing and distribution. Concerns about the trend towards globalization of the supply chain for active ingredients and finished products relate to the perceived potential for counterfeiting, tampering, adulteration, contamination, and other risks that could harm the products' end users (Woo, Wolfgang, & Batista, 2008, p. 494). Currently, a large proportion of finished health products sold in Canada are now imported; and in the case of active pharmaceutical ingredients (APIs), Health Canada indicates that 85% of APIs are imported, with China and India being the major suppliers (Health Canada, 2012c). However, it would be misleading to suggest that quality is an issue solely in foreign manufacturing facilities. In addition to direct harm from contaminated or substandard active ingredients or even toxic chemicals, counterfeit or poor-quality drugs may fail to have the necessary therapeutic effect, resulting in patient harm or discomfort; additionally, low-quality ingredients or manufactured health products could promote drug resistance or create shortages if subject to recall (Health Canada, 2012c). In one high-profile example, hundreds of adverse events and potential deaths resulting from allergic-type reactions and hypotension in US patients undergoing dialysis were ultimately traced to adulteration in China of the blood thinner heparin with oversulfated chondroitin sulfate, a synthetic material with similar chemical properties to heparin but costing one hundred times less than the latter (Pew Health Group, 2011, pp. 16-20).^{Footnote 8}
Risks arising from drug compounding. Health Canada defines compounding as "the combining or mixing together of two or more ingredients (of which at least one is a drug or pharmacologically active component) to create a final product in an appropriate form for dosing" (Inspectorate, 2009b, p. 7).^{Footnote 9} Compounded products can pose risks to patient safety. The issue of compounding has recently drawn significant attention as a result of an outbreak of fungal meningitis in the US among patients receiving contaminated steroid injections; as of October 25, 2012, 328 patients had been affected across 18 states, resulting in 24 deaths (CDC, 2012).
Risks arising from drug shortages. Drug shortages in Canada have been an emerging issue for the last several years (Duffin, 2012; HESA, 2012; Labrie, 2012). For example, drug shortages were caused by recent slowdowns in production by a major generic drug manufacturer and, in particular, events at a Québec-based manufacturing facility that produces approximately half of Canada's generic injectable drugs (HESA, 2012, p. 3). Drug shortages can lead to increased reliance on inappropriate alternatives, delayed medical procedures, and higher likelihood of medication errors.

Based on this brief overview of health and safety risks associated with human drugs, an ongoing role for government seems warranted in order to protect human health and safety. In this context, it is important to make note of recent industry trends that are having implications for regulatory agencies. For example, for some time, the pharmaceutical industry been experiencing the "patent cliff," which refers to the loss of patent protection for many of its top-selling brand-name products, and has responded to this development in two ways. First, it has focused on extending patent protection for existing products, as well as developing new pharmaceutical drugs similar to products already in the marketplace (Ferguson & Lybecker, 2012). Second, it has shifted towards the development of high-cost products for less common medical conditions, most notably biologics. Although some biologics have been used in Canada for many years, biologics produced using biotechnology have rapidly achieved acceptance in the Canadian marketplace.

Table 1: Top-selling drugs in Canada, 2011
Rank	Product name	Therapeutic subclass	Total sales	2010 growth (%)	Company
1	Crestor	Cholesterol reducer	742.20	14.2	AstraZeneca
2	Remicade	Anti-arthritic	504.80	16.3	Schering
3	Plavix	Blood circulation	306.70	4.4	BMS
4	Humira	Anti-arthritic	301.90	22.4	Nycomed
5	Enbrel	Anti-arthritic	292.70	7.1	Amgen
6	Nexium	Stomach acid control	306.70	4.4	AstraZeneca
7	Lucentis	Ophthalmic preps	238.00	33.8	Novartis
8	Oxycontin	Pain killer	234.00	2.1	Purdue
9	Lyrica	Seizure disorders	206.00	14.0	Pfizer
10	Advair	Asthma therapy	202.90	2.9	Abbott

Source: (Industry Canada, 2012)

Furthermore, new classes of biologic drugs are beginning to emerge. These include second-generation biologics, also known as biobetters or biosuperiors, which are products that have been "structurally and/or functionally altered to achieve an improved or different clinical performance"; "me-too" or "non-innovator" biologics, which have been developed independently and may not have been compared directly against licenced products; and subsequent entry biologics (SEBs), also known as biosimilars, which differ from non-innovator biologics in that they are formally compared with a currently licenced product (Weise et al., 2011, p. 691). With patent protection for many biologic drugs nearing its end, a key emerging issue for regulators involves managing the approval and reimbursement of these classes of biologic drugs.

Similarly, the emergence of combination products is creating challenges for regulators. Examples of combination products include drug-eluting stents and "smart pills" incorporating a medical device which controls release of the active ingredient. From a regulatory standpoint, this creates challenges in determining which piece or pieces of legislation govern the product in question and/or which regulatory body or bodies have authority over its testing, approval, and use.

Finally, globalization of the supply chain for therapeutic products is creating challenges for national regulatory agencies in managing risk at various points in the manufacturing process and the product lifecycle, and creating pressure for increased harmonization of regulatory approaches. In short, the number of Canadians potentially affected, the risks associated with the use of human drugs, and current industry trends and developments all support the conclusion that continued government oversight of these products is necessary in order to protect the health and safety of Canadians.

4.2 Alignment with federal priorities

The evaluation found the HDP to be aligned with the priorities of the Government of Canada. The federal government has devoted substantial resources over the past decade to broader initiatives intended to improve the safety of health products in general through modernizing the regulatory framework for these products. The goal of renewal is to ensure continued, timely access by Canadians to safe and effective health products in the face of several major challenges to the existing regulatory system. According to Health Canada, these challenges include an outdated regulatory toolkit; the system's current incapacity to consider products through their entire lifecycle, from discovery through to examining "real-world" benefits and risks; the impact of scientific and technological changes, globalization of the marketplace, and a more informed and engaged public; and insufficient resources for long-term efficiency and sustainability (HPFB, 2007a).

The Branch's plans for regulatory renewal were first articulated in the 2007 Blueprint for Renewal (HPFB, 2007a). The broad goals of regulatory renewal, as expressed in that document, include the following:

taking a "product lifecycle" or "progressive licensing" approach to regulation, i.e., an approach to regulating health products that encompasses all stages of product development, and according to which products are assessed for quality, benefits, harms and uncertainties at pre- and post-market stages
implementing regulatory interventions proportional to risk
strengthening post-market surveillance, and compliance and enforcement
learning from and collaborating with international counterparts
enhancing transparency and openness
basing regulation on scientific evidence
ensuring the sustainability of the regulatory framework

The federal government has undertaken several major initiatives that are clearly aligned with the objectives of regulatory renewal. Beginning in 2003-2004, the federal government committed $190 million over five years under the Therapeutic Access Strategy (TAS) to "improve the timeliness of Health Canada's regulatory processes with respect to human drugs" (Department of Finance, 2003). A major focus of the TAS was making pre-market regulatory decision making more efficient, timely, and transparent.

Beginning in 2006, the federal government allocated $172.5 million over a five-year period to the Therapeutic Product Safety Initiative (TPSI). The main focus of the TPSI was on post-market processes for the safety of therapeutic products.

More recently, Budget 2008 devoted $489.4 million under the Food and Consumer Safety Action Plan (FCSAP) to ensure that consumer products and food are safe and beneficial to Canadians' health (Health Canada, 2008a). The FCSAP is a horizontal initiative - involving Health Canada, the Public Health Agency of Canada (PHAC), the Canadian Food Inspection Agency (CFIA), and the Canadian Institutes of Health Research (CIHR) - with an overall goal to modernize food, health, and consumer product safety regulations and practices in Canada. FCSAP activities are expected to produce the long-term outcome of improved safety of health products on the market.

In 2008, the Government of Canada introduced comprehensive legislation (Bill C-51) embodying several elements of regulatory modernization, including the following:

incorporating a progressive licensing framework where "the assessment of benefits and risks is to be based on scientific and objective evidence" and there is "ongoing assessment of information about a therapeutic product over its lifecycle"
requiring prescribed classes of health care institutions to report adverse reactions
introducing new administration and enforcement measures, including mandatory recalls of therapeutic products and federal power to recall the remaining stocks of a drug once it has been taken off the market due to safety concerns
introducing a modernized framework for monetary fines and penalties by substantially increasing the fines for non-compliances
expanding the powers of the Minister to compel holders of clinical trial authorizations, market authorizations, and establishment licenses to compile information, conduct tests or studies, or monitor experience about the effects of therapeutic products on health or safety and report the information or results to the Minister (GoC, 2008a).

With the dissolution of Parliament on September 7, 2008, Bill C-51 did not become law. Since then, the federal government has taken a more incremental approach to regulatory modernization. Its most recent vision for modernization is expressed in the Regulatory Roadmap for Health Products and Food, released in May 2012 (Health Canada, 2012j). The Roadmap envisions a phased approach to modernization, with full implementation of all phases expected to be completed within a minimum of five years. Phase I includes developing regulatory amendments to repeal Schedule F (prescription drugs) of the Food and Drug Regulations and replacing it with a Prescription Drug List; creating a regulatory framework for orphan drugs (i.e., drugs developed specifically to treat rare diseases); and implementing a Plain Language Labelling Initiative (HPFB, 2012d).

The federal government has signalled its commitment to regulatory modernization for health products by undertaking revisions to its existing cost recovery framework. The 2007 Cost Recovery Initiative (CRI) was in part a response to concerns raised by the Office of the Auditor General of Canada (OAG) in 2004 and 2006 about Health Canada's ability to fulfill its regulatory requirements given its resource levels.^{Footnote 10} The CRI was intended to contribute to a stable funding platform for regulatory services by increasing annual revenues for Health Canada from $47 million to $112.4 million, restoring the original cost-sharing ratio of 50%^{Footnote 11} that occurred when cost recovery fees were first introduced (Health Canada, 2010c). A major milestone of the 2007 CRI was achieved with the coming into force of Fees in Respect of Drugs and Medical Devices Regulations on April 1, 2011 (GoC, 2011).

The major federal initiatives described above clearly demonstrate alignment of the HDP with HPFB and federal priorities, particularly with respect to regulatory modernization for health products. Furthermore, HDP activities are consistent with Health Canada's current Program Activity Architecture (PAA) and, in particular, with the strategic outcome focused on helping to ensure that Canadians are informed of and protected from health risks associated with food, products, substances and environments and helping to ensure thatproducts Canadians use are as safe as possible, and that threats to health are addressed effectively (Health Canada, 2012b).

4.3 Consistency with federal roles and responsibilities

The HDP is consistent with federal roles and responsibilities, some of which are defined in legislation. Health Canada's mandate is set out in the Department of Health Act, which defines the Minister's duties to include, among other things, the preservation of Canadians' health and well-being, the protection of Canadians against risks to health and the spreading of diseases, investigation and research into public health, the establishment and control of safety standards and safety information on requirements for consumer products, and the collection, analysis, interpretation, publication and distribution of information relating to public health (GoC, 2006, sec. 4(2)). More broadly, the Minister's jurisdiction covers all matters related to the health of Canadians that have not otherwise been assigned by the government to another body.

The Food and Drugs Act and the Food and Drug Regulations define the parameters of a drug and give legislative support to Health Canada's role in regulating the use of human drugs and taking actions to enforce compliance with those regulations. The Act limits the purposes for which any food or health product in general may be advertised or sold (GoC, 2008b, sec. 3). Sections 8-11 of the Act relate directly to drugs, prohibiting the sale of unsanitary or adulterated drugs, the use of inaccurate labelling, the misrepresentation of substances for other drugs, and the unsanitary manufacturing of drugs. The Regulations elaborate on the requirements for the safety and effectiveness of drugs designed, manufactured, or distributed in Canada, the standards for labelling and advertising, establishment license requirements, GMPs, clinical trials, and adverse reaction/event reporting. The Regulations also specify the corrective actions that may be taken for drugs that are believed to not meet these requirements (GoC, 2012c).

Health Canada is also involved in the regulation of controlled drugs under the Controlled Drugs and Substances Act (CDSA) and Narcotic Control Regulations. The Inspectorate's responsibilities include conducting security inspections to assess proper storage of controlled drugs prior to issuing a license to a Licensed Dealer and facilitating destruction of controlled substances (Inspectorate, 2012d). Other responsibilities are carried out by various offices under the Healthy Environments and Consumer Safety Branch (HECSB), including the Office of Controlled Substances (OCS) and Office of Research and Surveillance (ORS). The OCS is responsible for issuing licenses for distributors of controlled substances, as well as developing legislation, regulations, policies, and operations to support the control of illicit drugs and other substances (HECSB, 2009), while the ORS is responsible for monitoring trends in illicit drug use in Canada and examining issues and emerging trends regarding drug abuse, both within and outside Canada (HECSB, 2009).

5.0 Findings: governance and implementation

This section of the report presents the evaluation findings on program governance and implementation.

5.1 Program governance

The HDP includes all activities undertaken by the TPD in relation to human drugs, as well as relevant activities conducted by the MHPD, the Inspectorate/RAPB, PPIAD, and RMOD. Health Canada representatives reported that the Director Generals' Coordinating Committee was established in 2010 to manage the HDP.^{Footnote 12}

In 2012-2013, HPFB implemented a new governance structure. Under the new structure, Pharmaceutical Drugs, Biologics, Medical Devices, Natural Health Products, Food Safety and Nutrition, and Nutrition Policy and Promotion are considered separate programs, each governed by a Program Executive Committee Sub-Committee. The Pharmaceutical Drugs Program (PDP) encompasses veterinary drugs, as well as human drugs. Within HPFB, a variety of committees and working groups also exist to manage various aspects of HDP activity.

According to information provided by the Branch, in the previous governance structure, planning was done at the Directorate level and decisions were reported along program lines to senior officials. In the new model, it is intended that Programs will drive planning and reporting and Directorates will provide functional activities to deliver Program outcomes. The new approach is expected to more directly align the work done by all Directorates with the strategies, outcomes, and priorities of the Programs.

5.2 Collaboration with external partners

HDP partners collaborate with a variety of other federal government departments and agencies, as well as other external partners, to deliver regulatory activities related to human drugs. As part of the HDP, the Inspectorate collaborates with the Canada Border Services Agency (CBSA) to ensure consistent administration of the Food and Drugs Act and Regulations through the Border Integrity Program, the objective of which is to strengthen Health Canada's ability to make and support admissibility decisions at the border as they relate to health products.

As part of the HDP, the Inspectorate also collaborates with the OCS of Health Canada's HECSB. The OCS is responsible for licensing manufacturers and distributors of controlled substances, authorizing the disposal of illegal drugs, managing exemption processes allowing individuals with scientific or health reasons to access controlled substances, and working with other groups to enforce compliance (HECSB, 2009). The Inspectorate's responsibilities include conducting security inspections to assess proper storage of controlled drugs prior to issuing a license to a Licensed Dealer and facilitating destruction of controlled substances.

TPD and the Inspectorate also collaborate with industry and other stakeholders through the Bilateral Meeting Program (BMP). Through this program, the directorates hold bilateral meetings with various national stakeholder organizations between one and four times per year. These meetings are held to discuss regulatory issues, exchange information, and share expertise and responsibilities. Finally, a variety of scientific and expert advisory committees and panels have played a role in guiding human drugs activities over the evaluation period.

5.3 Performance measurement

There are a number of performance measurement strategies (PMS) that are directly relevant to the HDP. These include frameworks for the TPSI, the FCSAP, and the CRI, as well as the HPFB logic model and accompanying PMS. Data on submission review performance, which include metrics on the volume of applications received, review times, workload, and backlog statistics have historically been the main focus of HDP performance reporting. Detailed annual reports on submission review performance are used to inform senior managers and other decision-makers, and are also shared with industry. In addition, monthly dashboard reports were implemented with the CRI and are used to inform decisions on priorities and resource allocation. Health Canada also reports annually against established performance standards for the HDP in the Departmental Performance Report, which is tabled in Parliament.

5.4 Program implementation

The HDP has made progress over the period of the evaluation in implementing planned activities and, in the process, has responded to several emergent issues and challenges. The discussion below provides an overview of the drug regulatory process by way of introduction, then goes on to describe pre-market activities, post-market surveillance and monitoring, compliance and enforcement, and communications and stakeholder engagement activities under the period of the evaluation.^{Footnote 13}

5.4.1 The drug regulatory process

The current system of drug regulation in Canada focuses mostly on pre-market activities (TPD, 2010b). It is a "point-in-time" system, which is characterized by discrete regulatory interventions at specific, defined points, and in which drugs are evaluated for their risks and benefits primarily at the pre-market stage. Figure 1 illustrates the process, showing the order of the components and the areas where Health Canada has regulatory authority. The regulatory process depicted below applies to all drugs, as defined in the Food and Drugs Act.

**Figure 1 - Overview of the drug regulatory process (TPD, 2007)**

Sections 5.4.2 through 5.4.5 describe the HDP's activities over the evaluation period in relation to the four main stages of the regulatory process - clinical trials, submission review and market authorization, post-market surveillance and monitoring, and compliance and enforcement. A description of the HDP's activities in the area of communication and stakeholder engagement follows.

5.4.2 Clinical trials

Clinical trial authorization is the first step in the licensing process and the first stage at which Health Canada has regulatory authority. To begin the process, sponsors submit a Clinical Trial Application (CTA). CTAs are required for authorization to conduct clinical trials of new drugs in Canada, or marketed drugs where the proposed use is different from the approved use. A CTA-A is an application that contains new information intended to update or modify a previous CTA approved by Health Canada.

After a sponsor submits a CTA or CTA-A, Health Canada begins the safety, efficacy, and quality review. Health Canada reviews the application for completeness and to determine if it is acceptable from a scientific and regulatory perspective, a process that is to be completed within 30 calendar days. If the Department identifies a deficiency, it will send, prior to the default date, a specific notice or letter to the sponsor, depending on the extent of the deficiency.

If Health Canada does not identify any deficiencies in the CTA or CTA-A, it will issue a No Objection Letter (NOL), and the sponsor will be free to conduct the clinical trial. Alternatively, if Health Canada fails to complete its review within the targeted 30-day period, the sponsor is allowed to proceed with the clinical trial. Health Canada has the authority to suspend and/or cancel the authorization to conduct a clinical trial, should it become aware of information that suggests the benefit/risk ratio of the clinical trial is not acceptable.

In 2010, Health Canada received 1,191 CTAs for human drugs, and approved 1,162. Since 2004, there has been a 30% decline in the number of CTAs for human drugs received and approved. By comparison, Health Canada received 272 CTAs for biologics in 2010, and approved 263, which represents an increase of 5% in biologics CTAs received, and an increase of 15% in approvals. These trends are consistent with the industry shift, described in Section 4.1, toward development of biologics. That said, Health Canada indicated that the downward trend for human drugs CTAs stabilized around 2010; recently (in June 2013), the number of CTAs increased.^{Footnote 14} See Appendix C for more detailed information.

Strengthening the regulatory framework for clinical trials

The current framework for clinical trials for drugs involving human subjects came into force on September 1, 2001. The regulations introduced a requirement that the clinical trial protocol be approved for each trial site by a research ethics board (REB) and that the clinical trial be conducted in accordance with GCPs, as well as requirements relating to record retention and adverse reaction reporting. An inspection program was introduced to verify that clinical trials comply with the new regulations.^{Footnote 15}

Since these regulations were introduced, Health Canada has further strengthened the regulatory framework for clinical trials by developing and updating guidance documents for sponsors, including, most recently, updated general guidance and guidance on the inclusion of women in clinical trials; implementing risk-based approaches to monitoring and assessing clinical trial adverse reaction reports and selecting inspections of clinical trial sites, in response to the 2011 recommendations of the OAG; and commissioning the Canadian General Standards Board to develop new voluntary standards for REBs, which were officially approved by the Standards Council of Canada in May 2013 and are now publically available for purchase through the Public Works and Government Services Canada website.

Most recently, on May 29, 2013, Health Canada launched a new public database of drug clinical trials it had authorized. The database is mandatory for industry and contains specific information relating to Phase I, II, and III clinical trials in patients, including protocol number and title; drug name; medical condition; study population; date of NOL; sponsor name; study start and end dates; and trial status. The database includes trials that were authorized by Health Canada starting April 1, 2013 (Health Canada, 2013f).

By implementing the drug clinical trials database, Health Canada addressed its long-standing commitment to increase the transparency of clinical trial information.^{Footnote 16} However, as Health Canada points out, the database is not a trial registry, and does not contain comprehensive information about each clinical trial (Health Canada, 2013f). In contrast, both the US and the EU have implemented mandatory registration of clinical trials, and have taken steps to require disclosure of clinical trial results.

In the US, registration of clinical trials has been a requirement since ClinicalTrials.gov was initiated as a result of the Food and Drug Administration Modernization Act of 1997, and a requirement for the submission of clinical trial results no later than 12 months after the completion of the clinical trial was introduced in 2007 with the Food and Drug Administration Amendments Act (NIH, 2012; US Government, 2007). Similarly, in the EU, since 2004, clinical trials must be registered with the EudraCT database. Since 2011, through the EU Clinical Trials Register, the public may access descriptive information on Phase II-IV adult clinical trials and paediatric clinical trials where the investigator sites are in the EU/European Economic Area (EMEA, 2012). Publication of results information is expected to be part of a revised Clinical Trial Directive, which is expected to be implemented in 2016.

Public disclosure of clinical trial results is important, given evidence from the literature that selective or biased reporting of clinical trial results - i.e., reporting only positive results and failing to report negative results - is quite common. Furthermore, as the case of Vioxx demonstrates, selective reporting can have serious consequences for patient safety. Further enhancing the amount of clinical trial information which is made publicly available, including information on the results of clinical trials, would be consistent with Health Canada's commitment to enhancing transparency and openness as part of regulatory modernization, and would further align its approach with its main international counterparts.

Another area of regulatory divergence concerns paediatric clinical trials. Unlike Health Canada, both the US and the EU have the authority to require manufacturers to conduct paediatric trials. Health Canada representatives indicated that the Department does require paediatric trials before market approval if the manufacturer intends to obtain a paediatric indication; otherwise, approval will not be granted, and the Department has taken various steps to encourage sponsors to conduct paediatric studies.^{Footnote 17} In the US, the Pediatric Research Equity Act of 2003 gave the FDA the authority to require manufacturers to conduct paediatric studies of new drugs that are likely to be used in a substantial number of children, or that are expected to provide meaningful benefits for children compared to existing treatments (IOM, 2008). Similar legislation was enacted in the EU in 2007, including a requirement that all applications for market authorization for new medicines include the results of all studies carried out as part of a mandatory Paediatric Investigation Plan, the aim of which is to ensure that the necessary data are generated "to determine the conditions in which a medicinal product may be authorized for the paediatric population" (MHRA, 2009).

The absence of similar regulations in Canada prompted the Senate Standing Committee to recommend in its 2012 report that Health Canada introduce regulations to require clinical trials to be designed to reflect the same population that can be "reasonably expected to consume the drug" once approved for sale in Canada and modify the drug approval process so that market approval is only granted if clinical trial evidence of the product's safety and efficacy includes data on all population groups that can be reasonably expected to consume the drug (SSCSAST, 2012, p. 35). In response to the Committee's recommendation, Health Canada indicated that while it does not currently mandate clinical trial design, it has adopted ICH guidelines aimed at encouraging and supporting the development of drugs for use in special populations such as geriatrics and paediatrics; has developed a Canadian addendum to the ICH guideline for drug development in the paediatric population; and has been working with the Canadian Institutes of Health Research and a number of stakeholder groups to develop Ethical Best Practices for Health Research Involving Children and Adolescents.

The policy statements contained in Health Canada's recently-published guidance on the inclusion of women in clinical trials seem consistent with the Committee's recommendations, including recognition that clinical trials should be composed of subjects representative of the populations that the sponsor expects will use the product (Health Canada, 2012e).

5.4.3 Submission review and market authorization

The second step in the licensing process is the regulatory submission. After the sponsor conducts the clinical trials, it may choose to seek market authorization for the drug by submitting an application to Health Canada. Health Canada's role at the pre-market review stage is to evaluate the safety, quality, and efficacy of human drugs, ensuring that these products demonstrate clear benefits relative to their potential risks. The pre-market approval process applies to all drugs, as defined in the Food and Drugs Act.

Main types of drug submission

DIN application (Division 1)
New Drug Submission (NDS) (Division 8)
Priority NDS request
Request for Notice of Compliance with conditions (NOC/c)
Abbreviated New Drug Submission (ANDS) (generic drugs)
Supplemental New Drug Submission (SNDS)
Supplemental Abbreviated New Drug Submission (SANDS)
Extraordinary Use New Drug (EUND) submission

Source: Submission types and process

There are several types of drug submission. The DIN application process applies to "Division 1" drugs. ^{Footnote 18} A Division 1 drug is usually one that has been on the market for many years and whose safety and efficacy are well-established. Often, these drugs are available over the counter (OTC) (i.e., no prescription is required). Successful applications are issued a Drug Identification Number (DIN).

The New Drug Submission (NDS) process applies to "Division 8" drugs. Division 8 drugs are new drugs, or those which have not been available for a long period of time. An NDS is similar to a DIN submission, with some major differences, including much more rigorous regulatory requirements. The NDS process results in a Notice of Compliance (NOC), which authorizes the drug to be sold on the market, as well as the issuance of a DIN.

To begin the NDS process, a sponsor may submit a regular NDS (as described above), make a request for priority NDS status, or make a request for a Notice of Compliance with Conditions (NOC/c). A request for priority status may be made when there is substantial evidence to suggest the proposed drug will be useful against a debilitating or life-threatening disease or condition or for which there is no other therapy marketed in Canada, or when the benefit/risk profile is an improvement over existing therapies marketed in Canada. The screening and review times are shorter for priority NDS than for regular NDS.

Just as for priority review status, an NOC/c may provide earlier access to "potentially life-saving drugs." However, an NOC/c is different from a priority review in that it places a condition on the manufacturer to conduct further studies to confirm the benefits of the drug. These post-market surveillance studies are intended to monitor the safety and effectiveness of the product.

Alternatively, the sponsor may submit an Abbreviated New Drug Submission (ANDS) when the proposed drug claims to be pharmaceutically-equivalent and bioequivalent to an existing drug on the market. In this case, the submission must provide information that compares the performance of the "generic" product to the brand-name product. A Supplemental New Drug Submission (SNDS) or Supplemental Abbreviated New Drug Submission (SANDS) is submitted when significant changes, such as changes to the dosage, manufacturing process, or labelling, are made to new drugs that have already been issued an NOC.

Finally, since 2011, sponsors may submit an Extraordinary Use New Drug (EUND) submission, which provides a pathway for the authorization of new drugs under extraordinary circumstances by allowing sponsors to use results of animal studies in conjunction with results from limited data from human safety and efficacy studies to support their drug submission (Health Canada, 2013a). EUNDs are intended for extraordinary use in response to exposure to a chemical, biological, radiological, or nuclear substance where action is required to treat, mitigate, or prevent a life-threatening or other serious disease or disorder resulting from that exposure, or for preventative use in persons who are at risk of such exposure.

Review and market authorization

Once submitted, the screening processes and the safety, efficacy, and quality review processes take place. In comparison to Division 1 drugs, whose safety and efficacy are well-known, the review process for NDS involves a more in-depth clinical assessment of animal and human studies. When deficiencies or non-compliance issues are found, a company may submit responses for review before a final decision can be reached. Thus, multiple review cycles may be required.

After successfully completing the screening process and the safety, efficacy, and quality review, the drug is issued a DIN, as well as an NOC. The NOC certifies that the drug is compliant with the Food and Drugs Act and Regulations, and authorizes it for sale on the market. Alternatively, the drug is issued an NOC/c, which grants market authorization of an acceptably-safe high-quality product, as long as manufacturers continue to comply with associated conditions (HPFB, 2011b). The goal of issuing an NOC/c is to provide access to a new or greatly improved treatment, prevention, or diagnosis of severe diseases or conditions for which there are no or inferior relevant drugs available, while continuing to monitor the risks presented by the drug and to evaluate its overall risk/benefit profile.

Post-market changes

After a product has received market authorization, sponsors are responsible for notifying Health Canada of any changes to the product. A different process is followed depending on the nature of the change and its likely or potential impact on safety, quality, and efficacy. Even if there are no changes to a drug, the sponsor must still submit annual notification of this to Health Canada (Health Canada, 2011c).

Special Access Program

Finally, under the Special Access Program (SAP), health care practitioners can gain access to drugs that have not yet received market authorization in Canada. The SAP is intended "for serious or life-threatening conditions where conventional therapies have failed, are unsuitable, or are unavailable either as marketed products or through enrollment in clinical trials" (HPFB, 2008). It may also apply in communicable disease outbreaks to clarify the circumstances in which authorization may be issued or denied; providing for authorizations to be further reviewed where circumvention of clinical trial or NDS requirements are suspected; allow for block releases of products in the event of public health emergencies; and establish an ethical framework for permitting compassionate access to new therapies outside of clinical research (HPFB, 2007a).

As part of regulatory renewal, Health Canada planned a review and modernization of the regulatory framework for the SAP, which was established in 1966. The status of this initiative is unknown.

Improving the submission review and market authorization process

Improving the quality and efficiency of the submission review and market authorization process for therapeutic products, including human drugs, has been a priority for HPFB over the evaluation period. Facing increasing costs and substantial backlogs in the review process, HPFB has undertaken a variety of initiatives in order to reduce costs and improve efficiency and quality in submission review. These initiatives are briefly described below.

Service standards for cost-recovered activities. To comply with the requirements of the User Fees Act, Health Canada implemented service standards for cost-recovered activities, including submission review (GoC, 2004, sec. 4.(1)). Service standards for submission review vary by submission type and class; for most NDS and ANDS, the standard is 300 days and 180 days, respectively (HPFB, 2011c).
New cost recovery framework. In response to OAG concerns about HPFB's ability to provide regulatory services given its existing resources, a new cost recovery framework for drugs and medical devices was implemented in April 2011 with the coming into force of the Fees in Respect of Drugs and Medical Devices Regulations (GoC, 2011; HPFB, 2011a). At the same time, HPFB adjusted its performance target for submission review. Prior to implementation of the new framework, HPFB's performance target was completing 90% of all first decisions for each submission class within the target time frame. While HPFB is still accountable for achieving this performance target, an additional metric was established with the implementation of the new cost recovery framework: requiring submissions, on average, to meet the target review time for first review for their submission class.^{Footnote 19} In theory, processing submissions within established performance targets is expected to facilitate timely access to effective therapeutic products, thus increasing available treatment options without putting end users at greater risk.
Project management approach to submission review. Health Canada implemented a project management approach to submission review, which involved the creation of new "regulatory project manager" positions within TPD to coordinate and guide each submission through the process, as well as the establishment of Workload Management Forums to provide a monthly review of workloads with directors and division managers, and the initiation of Workload Management Reports.
Good Guidance Practices (GGPs) and Good Review Practices (GRPs). TPD has developed GGPs and GRPs for greater consistency in the development of guidance documents and submission reviews, respectively.
Electronic submission. Health Canada has adopted the electronic Common Technical Document (eCTD) format established by the International Conference on Harmonisation (ICH) of Technical Requirements for Registration of Pharmaceuticals for Human Use for E-Review. Despite slower-than-anticipated progress in implementing electronic submissions, Health Canada expanded the types of submissions in eCTD electronic-only filing format in 2011, and the number of electronic submissions has been increasing. More recently, under the RCC, Health Canada and the US FDA have been working toward the implementation of a Common Electronic Submission Gateway, using the current US gateway, that will allow "industry clients the ability to submit large electronic documents seamlessly to Health Canada and the US FDA" (GoC, n.d.). Through the Electronic Common Submission Gateway, submissions will come to Health Canada through the FDA's information technology infrastructure. A three-month pilot of the gateway project is underway.^{Footnote 20}
Enhanced review capacity. TPD launched a Scientific Experts Database in 2007 to assist in accessing external expert advice to work on drug submission reviews and participate on Scientific Advisory Panels and Committees. Health Canada reported that a total of 564 scientific experts are included in the database, who may be contracted to conduct aspects of submission review. In the case of pharmaceuticals, these contracts are primarily for pre-clinical reviews, although other aspects of the review, such as label review, chemistry and manufacturing review, or bioequivelancy review, among others, may also be contracted out. According to Health Canada, external contracting costs have increased from approximately $350,000 in 2006-2007 to a forecasted $1.0 million in 2013-2014.
Pre-submission meetings. Although pre-submission meetings with sponsors were well-established by the early 1990s, both the Blueprint for Renewal and the FCSAP identified plans to improve this process. The main, though not the sole, purpose of the pre-submission meetings - which are optional at the discretion of sponsors - is to "provide scientific and regulatory advice to industry at early stages of product development" (HPFB, 2007a, pp. 22-23). In 2010-2011, guidance to industry on pre-submission meetings was made publically-available. Data provided by Health Canada show thatbetween 2002 and November 23, 2012, TPD held 1,279 pre-submission meetings, of which 13% were pre-CTA meetings and 59% were pre-NDS meetings (HPFB, 2012a). Industry key informants identified opportunities to improve the pre-submission meeting process by addressing long waits for meetings and ensuring that the principal reviewer is in attendance.
Use of foreign reviews. In October 2011, as part of broader plans to improve review efficiency through increased use of foreign reviews, Health Canada launched the Use of Foreign Reviews Pilot Project, which was to run until March 2013. Products covered by the pilot include NDS, ANDS, SNDS, SANDS, NOCs, and risk management plans (RMPs). It is unknown how many human drug submissions were made under the pilot. In a March 2013 update to the OAG, Health Canada reported that the "results of the pilot have been encouraging" (Health Canada, 2013l), while an internal document from mid-2012-2013 reports that the pilot project was "achieving limited success due to lack of participation and information from foreign sources" (HPFB, 2012c). External key informants encouraged Health Canada to accept and use more information and approvals from foreign sources as a means of improving review efficiency.^{Footnote 21}

These initiatives are expected to improve the efficiency and quality of the submission review process for human drugs. The available data on submission review performance, which covers the period from the beginning of calendar year 2004 to the end of calendar 2012 (see Table 2) show the following:

For each submission type, the timeliness of review has fluctuated over the years, and the 90% target has often not been met.^{Footnote 22} That being said, the most recent available data - for 2012 - show an improvement over the previous year in the proportion of review cycles meeting the service standard for all submission types. For NDS, the proportion of review cycles meeting the service standard has been improving since 2009, when 71% of review cycles met the standard. By 2012, this proportion had reached 97%.
Timely review of ANDS (generic submissions) and SANDS has been challenging in recent year. The ANDS review workload continued to increase from 81 in 2007 to 282 in 2011. In 2011, 11% of ANDS review cycles were completed within target, the median approval time had reached 645 calendar days, and 61% of ANDS submissions were in backlog, a higher proportion than in any other year since 2004. On a positive note, improvements were realized in 2012, with a 125% increase in the number of reviews completed (320 in 2012 compared to 140 in 2011), which reduced the year-end ANDS review workload to 222, with a backlog of 45% (down 16 percentage points from 2011). In 2012, 18% of ANDS review cycles were completed within target and median approval time had declined to 543 days.
Between 2004 and 2012, the median approval time declined by 40% for NDS and by 23% for SNDS. In the same time period, the median approval time increased by 48% for ANDS and by 23% for SANDS. In 2012, the median approval time for NDS was 349 calendar days, compared with 543 calendar days for ANDS.

Health Canada's recent performance in reviewing generic submissions seems to be due, at least in part, to a growing number of these submissions. Industry key informants attributed Health Canada's performance in generic reviews not only to growing volumes, but also to its decision to limit the resources allocated to these reviews. The evaluation did not have access to information on human resources devoted to review of generic drug and other submissions, and could not analyse the relationship between number of submissions, resource levels, and timeliness of review.

Overall, submission review performance for human drugs has improved under the new cost recovery framework for all submission classes. However, except for ANDS, improvements in review performance were already underway for all submission classes prior to the implementation of the new framework, and may be attributable to Health Canada's other initiatives to improve the efficiency and quality of the review process, described above.

A more detailed analysis of submission review data, taking into account factors such as changes in resource levels and number of applications over time, would provide further insight into Health Canada's review performance, including the extent to which efficiencies may have been realized under the new regime. For example, it would be informative to consider annual review performance in light of the ratio of the number of full-time equivalents (FTEs) to the number of applications each year for each submission class. Similarly, it would be informative to consider the unit costs of completing reviews for each submission class (see Section 7.0 for a more detailed discussion).

Finally, in the context of submission review it is important to note in passing that some external key informants expressed concern about the efficiency of Health Canada's approach to regulating combination products. As an example, external key informants noted that the current regulatory framework has hindered Health Canada's efforts to provide appropriate oversight of e-cigarettes, which can be viewed as both a drug and a medical device. As another example, the regulation of kits containing several classes of product is perceived to be very onerous, particularly when the contents are not all produced by the same manufacturer. Combination products were discussed in detail in the evaluation of the Medical Devices Program, and the discussion is not repeated here.

Table 2: Submission review performance - human drugs
Submissions	2004	2005	2006	2007	2008	2009	2010	2011	2012
New Drug Submissions (NDS)
Number of submissions received	47	51	53	51	53	64	54	53	69
Number in workload (at year end)	45	47	46	38	51	49	54	42	45
Percentage in backlog	18%	4%	0%	0%	12%	8%	15%	0%	0%
Number of review cycle completions	84	68	74	77	58	75	67	86	65
Percentage of review cycle completions within target	17%	62%	84%	92%	88%	71%	73%	80%	97%
Number of approvals	40	32	40	40	27	34	35	61	39
Median approval time (calendar days)	581	430	397	384	345	428	433	375	349
Sources: (TPD, 2008a, 2010a). Data for 2011 and 2012 provided by Health Canada and were calculated using the same methods employed prior to reporting changes introduced with the Cost Recovery Initiative in 2011-2012.
Supplemental New Drug Submissions (SNDS)
Number of submissions received	114	133	186	154	154	132	101	111	162
Number in workload (at year end)	95	91	122	116	104	92	76	88	95
Percentage in backlog	8%	1%	0%	1%	2%	7%	4%	2%	1%
Number of review cycle completions	163	146	160	195	175	158	138	110	160
Percentage of review cycle completions within target	38%	73%	92%	97%	92%	88%	79%	94%	95%
Number of approvals	103	93	123	158	139	121	103	76	131
Median approval time (calendar days)	369	344	337	323	322	325	338	336	285
Abbreviated New Drug Submissions (ANDS)
Number of submissions received	117	137	124	142	161	179	185	208	215
Number in workload (at year end)	71	77	92	81	97	153	212	282	222
Percentage in backlog	32%	0%	3%	6%	5%	35%	44%	61%	45%
Number of review cycle completions	111	165	195	222	204	168	164	140	320
Percentage of review cycle completions within target	19%	64%	90%	92%	90%	45%	24%	11%	18%
Number of approvals	82	77	81	121	122	96	107	86	225
Median approval time (calendar days)	368	378	525	532	453	467	536	645	543
Supplemental Abbreviated New Drug Submissions (SANDS)
Number of submissions received	8	7	26	12	19	32	28	28	60
Number in workload (at year end)	5	12	10	4	8	20	36	36	33
Percentage in backlog	60%	0%	0%	0%	0%	35%	64%	67%	30%
Number of review cycle completions	15	14	25	21	13	19	26	33	62
Percentage of review cycle completions within target	33%	57%	92%	95%	92%	68%	12%	27%	37%
Number of approvals	11	10	17	20	10	12	18	23	51
Median approval time (calendar days)	328	382	323	305	245	220	414	507	402
Definitions: Submissions received is the number of submissions received in a year using the filing date (the date the submission is considered administratively complete by Health Canada. Workload is the number of submissions in review status as measured at year end. These include both submissions undergoing first review as well as submissions where a subsequent review cycle was required to review company responses to issues raised by Health Canada. Backlog is workload that is over target. Approvals are Notices of Compliance (NOCs) issued or issuable. An NOC issuable is when a submission's NOC is placed "on hold" awaiting authorization to market, due to Patent regulations or due to de-scheduling (from prescription to OTC). Approval time is the number of calendar days between the submission's filing date and approval date, and includes any company time. Review cycle completion is counted upon the conclusion of an in-depth scientific review that results in a decision of approval or non-approval. It is considered to have occurred within target when the days taken to complete the review are within the performance standard.

5.4.4 Post-market surveillance and monitoring

Following market authorization, Health Canada is responsible for undertaking post-market surveillance - also known as pharmacovigilance - to ensure that a product's benefit/risk balance remains favourable. Health Canada defines pharmacovigilance as "the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other drug related problem" (MHPD, 2009).

Historically, adverse reaction reporting has been the pillar of Health Canada's (and other regulators') approach to pharmacovigilance. However, over the past decade, regulators have increasingly recognized the need to enhance post-market surveillance in order to better protect patient health and safety. Health Canada has acknowledged several limitations of its current approach to post-market surveillance, including the following:

relying on routine pharmacovigilance (i.e., spontaneous adverse reaction reports) for safety signals that are not easily identified by case report data^{Footnote 23}
lack of follow-up and risk evaluation in populations not studied in clinical trials
failure to address issues related to off-label use
failure to identify potential risks in the Canadian context and follow them up
lack of authority to compel post-market studies or other risk management strategies
lack of systematic evaluation of risk management practice to determine effectiveness (MHPD, 2009, 2012g)

As part of its broader regulatory modernization initiative and its intended shift toward a product lifecycle approach to regulation, Health Canada has taken steps to address these limitations and strengthen its approach to post-market surveillance and monitoring.

Enhancing adverse reaction reporting and analysis

Under the Food and Drug Regulations, "serious adverse drug reaction reporting" is mandatory for manufacturers. In addition, health professionals, institutions, and members of the public may submit adverse reaction reports on a voluntary basis. The list below provides more detailed information on adverse reaction reporting.

Adverse reaction reporting in Canada for human drugs

Manufacturers: Under the Food and Drug Regulations, "serious adverse drug reaction reporting" is mandatory for manufacturers. Manufacturers must submit all information relating to the incident within 15 days of receiving or otherwise becoming aware of a serious adverse reaction, whether it occurs inside or outside of Canada. Completed reports may be mailed or faxed to Health Canada. Since 2011, manufacturers are also required to prepare annual summary reports on all adverse reactions within the previous year and the manufacturer's analyses of those incidents, and submit these reports to the Minister of Health upon request.
Health professionals, institutions, and members of the public: Adverse reaction reporting is voluntary for these stakeholders. Standardized reports are available for consumers and health professionals, and may be mailed, faxed, or completed online. Reporting may also be completed by telephone.

All adverse reaction reports are submitted to the Canada Vigilance System, which was established in 2007 as the new name for the Canadian Adverse Drug Reaction Monitoring Program. Since then, adverse reaction reporting has increased dramatically.

Figure 2 shows the growth in the number of domestic adverse reaction reports relating to pharmaceutical drugs received by Health Canada between 1999 and 2011; biologics are included by way of comparison.^{Footnote 24} In total, 147,667 adverse reaction reports have been received for pharmaceuticals. The annual number of reports relating to pharmaceuticals grew 456% over this period, from 4,870 reports in 1999 to 27,077 reports in 2011. The bulk of this increase has occurred since 2007.

Domestic adverse reaction reports are only a fraction of the total number of adverse reaction reports received by Health Canada, since foreign reports, which represent more than 90% of all reports received, are not entered into the Canada Vigilance Database.

**Figure 2 - Domestic adverse reaction reports received, pharmaceuticals and biologics**

Health Canada is undertaking several initiatives to enhance adverse reaction reporting and analysis.

Addressing under-reporting by health care professionals

To address the long-standing problem of under-reporting of adverse drug reactions by health care professionals, Regional Canada Vigilance Program coordinators conduct outreach and education activities to provide information on how to report and why it is important to do so.^{Footnote 25} In addition, Health Canada intended to introduce mandatory serious adverse reaction reporting for health care institutions under the FCSAP, and a provision for mandatory reporting by health care institutions was included in Bill C-51 in 2008, which did not become law. Since then, Health Canada has produced a guidance document to assist health professionals in reporting adverse events (MHPD, 2011a); piloted an electronic reporting system in 2011, allowing health professionals to submit adverse reaction reports via the MedEffect website (MHPD, 2011a); and engaged Accreditation Canada, the body responsible for accrediting Canadian hospitals and other health care institutions, to establish national adverse reporting standards within the existing accreditation system.

In January 2013, Accreditation Canada released guidance on the reporting of adverse drug reactions in its latest Medication Management Standards for health care facilities; to support the new guidance, Health Canada plans to undertake marketing activities aimed at health professionals to provide information on how to report adverse reactions, why it is important to do so, and how to stay up-to-date on new safety information (Health Canada, 2013i). The guidance and marketing activities may help to address one of the main reasons identified by external key informants for under-reporting on the part of health care providers, namely a perceived lack of clarity regarding reporting requirements. Stakeholders identified numerous examples of the types of issues they encounter when deciding whether or not to report an observed event, including who is responsible for reporting, especially if a patient is being seen by multiple clinicians; whether the reaction should be reported if it was expected, or if it is attributable in whole or in part to inappropriate use of a product; how serious the event must be to merit reporting; and how or if to report adverse events associated with low-risk products. Other reasons for under-reporting, according to external key informants, include time constraints, a reporting process that is not perceived as user-friendly, concern about potential negative consequences, and lack of feedback from Health Canada regarding how and whether the information they provide will be used.

External key informants' recommendations for increasing voluntary reporting included expanding marketing efforts, engaging professional and industry associations to encourage their members to report; streamlining the reporting process, providing feedback to those who report, and examining approaches employed in other jurisdictions, as well as reimbursing practitioners for reporting and implementing mandatory reporting. With respect to the latter point, at the time the new standards were released, Health Canada indicated its intention to monitor the impact of the new standards before deciding whether to proceed with mandatory reporting (Toronto Star, 2013a). Notably, adverse drug reaction reporting is not, at present, mandatory for health care facilities or providers in the US or Australia (FDA, 2012a; TGA, 2012). Reporting requirements for health professionals vary across the EU, with some states (such as Denmark, France, Austria, and Italy) requiring health care professionals to report serious adverse drug reactions, and others (including the United Kingdom, Ireland, the Netherlands, and Romania) relying on voluntary reports from health professionals (Srba, Descikova, & Vlcek, 2012, p. 1061).

In comparison to voluntary reporting, mandatory reporting of adverse drug reactions by industry appears to be, for the most part, unproblematic, with external key informants and industry survey respondents generally agreeing that Health Canada has clearly defined the requirements for mandatory reporting and that industry understands and is compliant with these requirements.

Implementing electronic submission of adverse reaction reports

To enhance adverse reaction reporting and analysis, including analysis of foreign adverse reaction reports, Health Canada has begun to implement electronic reporting of adverse reactions for industry. Implementation has taken longer than anticipated due to technical requirements (Health Canada, 2013l). As of May 1, 2013, program representatives reported that one market authorization holder (MAH) had been registered to submit reports electronically, with additional MAHs to begin electronic submission over the next few years.Electronic submission has been mandatory for industry in the US since 2000 (FDA, 2012b). Electronic submission is also possible in the EU.

Enhancing analysis of adverse reaction reports

Health Canada conducts case reviews of every domestic adverse reaction report submitted to Canada Vigilance and aims to complete 95% of case reviews within its established service standard of 15 days for review of priority initial reports (i.e., those involving death or a life-threatening situation).^{Footnote 26}

In addition, Health Canada has begun developing and implementing strategies to systematically monitor adverse reaction reports, including targeted surveillance and data-mining, for potential safety signals. The implementation of the Canada Vigilance Database was intended to improve Health Canada's ability to monitor and analyze adverse drug reaction data, using a new software analytical tool called Safety Mart - later renamed agSignals - which was first implemented in March of 2008 as a component of the new database (MHPD, 2010). Due to a significant delay in implementing the agSignals software, implementation of data-mining is now expected to coincide with the implementation of electronic submission. Since 2010, targeted surveillance for health product-targeted medical events, designated medical events, and new active substances has been implemented, with surveillance for special populations expected to be implemented in 2013. Plans for targeted surveillance of reports with fatal outcomes have been replaced with a new risk-based Work Instruction on assessing adverse reaction reports submitted electronically to the Canada Vigilance Database.

Since the targeted surveillance and data-mining initiatives are relatively recent and still under development, it is unclear how systematically Health Canada has been approaching the analysis of adverse reaction reports over much of the evaluation period. In any case, until Health Canada has information systems in place to receive reports electronically and to monitor and analyze these reports systematically for potential safety signals, initiatives to increase the number of adverse reaction reports received may be premature.

A subset of the adverse reaction report data is publicly-accessible through the Canada Vigilance Online Database, which is updated quarterly. Health Canada has identified numerous caveats surrounding the use of these data (Health Canada, 2012f). In a 2008 study by the Standing Committee on Health, criticisms arose that access to the database was too limited, and there was a lack of transparency around the analysis conducted using database reports (HESA, 2008).^{Footnote 27}

Enhancing signal detection and assessment

In addition to enhancing adverse reaction reporting and analysis, Health Canada has also sought in recent years to enhance the detection and assessment of safety signals by broadening the sources of information it monitors for potential signals and developing SOPs to guide the process. Health Canada defines a safety signal as a "new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, that is judged to be of sufficient likelihood to justify verificatory [sic] action" (MHPD, 2012e).

Sources monitored for potential safety signals include the scientific literature, media communications, public communications from other regulatory agencies, safety information from MAHs (such as adverse reaction reports and Periodic Safety Update Reports), adverse reaction information voluntarily submitted to the Canada Vigilance Program or other adverse reaction databases, and information from Health Canada's pre-market bureaus (MHPD, 2013). Within MHPD, three bureaus are responsible for monitoring these sources for potential signals, including two which have responsibilities relating to human drugs, as summarized in Table 3. The Marketed Health Products Safety and Effectiveness Information Bureau monitors adverse reaction reports for potential safety signals, while the Marketed Pharmaceuticals and Medical Devices Bureau is responsible for monitoring all other information sources for potential safety signals relating to human drugs (and medical devices).

Table 3: MHPD bureaus responsible for signal detection
Bureau	Responsibility
Marketed Health Products Safety and Effectiveness Information Bureau	Monitors spontaneous adverse reaction reports submitted to the Canada Vigilance Program using case review and case series analysis, targeted surveillance, and data-mining.
Marketed Pharmaceuticals and Medical Devices Bureau	Four specialized working groups within this bureau are responsible for detecting pharmaceutical signals originating from foreign regulators, MAHs, the medical/scientific literature, and safety information from TPD, respectively.
Marketed Biologicals, Biotechnology and Natural Health Products Bureau	Within this bureau, the Signal Identification and Coordination Working Group is responsible for detecting potential signals relating to biologic products by monitoring all information sources.

After a potential signal is detected and given a preliminary assessment, it is presented to and discussed by the signal prioritization committee. The committee is responsible for classifying potential signals in one of three ways:

an identified signal, if information suggests a link exists between product and event, and risk management is possible
a potential product issue awaiting further information, if there is insufficient information to support the link now, but information could become available within 18 months and allow for risk management
a dismissed signal, if information to support the link between product and event is not currently available, is unlikely to become available soon, or if risk management is impossible regardless of the investigation findings (MHPD, 2012e)

Identified signals are assigned for signal assessment, during which evaluators conduct a comprehensive critical appraisal of the evidence available, analyse possible risk management options, and make recommendations for action (MHPD, 2012f). Potential recommendations may include standard or enhanced monitoring; requesting additional information from the MAH; requesting the MAH to undertake a benefit-risk assessment, develop a pharmacovigilance plan, or develop a risk management plan; recommending changes to the product labelling; recommending issuance of a risk communication; requesting consideration by the Drug Safety and Effectiveness Network (DSEN); and recommending a full issue analysis to determine if the product should be withdrawn from the Canadian market.

A large number of SOPs have been developed over the last few years to support the signal detection, prioritization, and assessment process, while others are still in development.

Based on analysis of administrative data provided by Health Canada, the following observations can be made about MHPD's signal detection and assessment activities:

Of 1,980 detected signals relating to human drugs between 2006 and 2012, approximately 81% were dismissed and 7% were prioritized for signal assessment. By comparison, approximately 97% of the 6,505 detected signals relating to biologics were dismissed, and fewer than 1% were prioritized for signal assessment.^{Footnote 28}Although MHPD has developed a standardized approach to signal detection and analysis over the past several years, the differences in the number of signals detected for pharmaceuticals and biologics, as well as the differences in the number and proportion dismissed, may suggest that MHPD has nonetheless used different processes or applied different criteria in the signal detection process for pharmaceuticals and biologics.
A total of 272 signal assessments for human drugs were completed between 2003 and 2012. The most commonly-recorded sources of safety signals were an international regulator or agency (26%), most frequently the FDA (14%); the scientific literature (26%); and the Canada Vigilance Database (8.5%).
The most common recommendations resulting from completed signal assessments were changes to the product labelling (54%); standard monitoring (36%); and requesting the issuance of a risk communication (24%).

Please see Appendix C for more detailed information.

The above analysis was complicated by a variety of limitations in the data. For example, signal detection and assessment activities are tracked through a series of Excel spreadsheets maintained by various groups within MHPD, rather than through an integrated or centralized database accessible to all parties involved. There is no way to easily link or match records contained within these spreadsheets; unique identifiers are assigned to signal assessments, but not to signal detection activities, and, as a result, matching signal detection with assessment records must be done manually. In addition, there are many inconsistencies in the ways in which data are recorded, as well as missing data, with implications for data reliability and validity. There is also widespread use of open-ended, text-based fields for data entry, necessitating time-consuming coding to enable quantification or analysis.

In addition, there has historically been no centralized mechanism or tool in place for systematically tracking Health Canada's response to the recommended actions arising from completed signal assessments. In 2011, the OAG recommended that such a system be implemented, in response to which Health Canada indicated that SOPs for new formalized tracking systems were under development and that such systems would be implemented by March 2013 (Health Canada, 2011d, p. 27). In March 2013, Health Canada reported that a tracking tool for safety recommendations for all pharmaceuticals had been finalized and that operating procedures had been implemented (Health Canada, 2013l). The tool is specific to TPD and is limited to actions that fall within its area of responsibility (i.e., changes to product labelling). It is unclear whether a similar system has been implemented to track Health Canada's response to recommended actions that fall within the purview of other directorates, or its response to recommendations for biologics signals.

Implementing risk management and pharmacovigilance planning

Risk management planning involves detailing the potential risks posed by a health product and steps to be taken to reduce the potential for harm (Health Canada, 2009b), while pharmacovigilance planning refers to the pre-authorization development of a monitoring process to collect safety information on an ongoing basis after a drug has reached the market. Under the FCSAP, Health Canada intended to implement a more structured, comprehensive and systematic approach to pharmacovigilance planning, and, as a complement to pharmacovigilance plans (PVPs), create a new program requesting industry to submit RMPs on a voluntary basis. Health Canada intended eventually to propose legislative and regulatory amendments to compel manufacturers to submit RMPs and PVPs routinely as part of the pre-market process.

Requirements for RMPs in Canada

Mandatory components:

the Safety Specification that summarizes all important safety information concerning the product, including gaps in knowledge
the Pharmacovigilance Plan that includes known and potential safety concerns and the methods by which the manufacturer will monitor new safety information
a Risk Minimization Plan that proposes methods for minimizing known and potential safety risks

RMP submission:

as a component of a product license submission
on request by Health Canada in any case where the department deems it relevant to risk/benefit decisions, such as a new active substance, a change in indication, or the introduction of a product to a class with a previously-existing safety risk

Since these plans were announced, pharmacovigilance planning has been subsumed within the broader concept of risk management planning. In 2009, Health Canada adopted ICH E2E on pharmacovigilance planning and established interim requirements for RMPs based on the guidelines of the European Medicines Agency (EMA).

Data provided by Health Canada show that the number of RMPs has been generally increasing since 2005 for both human drugs and biologics, from a total of 2 in 2005 to a total of 75 in 2012. In 2012, there were 49 RMPs associated with TPD. It is unclear if these data represent the number of RMPs requested, the number received, or the number reviewed by Health Canada. For more information, please see Appendix C.

Health Canada recently announced that it would require RMPs for generic oxycodone, which it approved in November 2012. It is unknown if RMPs are routinely requested for any other type of pharmaceutical drug.^{Footnote 29}Health Canada does not currently have the regulatory authority to compel submission of RMPs, and the status of its plans to propose amendments that would give it such authority is unknown.

Both the EMA and the FDA have legal authority to compel submission and levy fines for noncompliance (MHPD, 2009). The FDA has had the authority since 2007 to require sponsors to submit a Risk Mitigation and Evaluation Strategy (REMS) to ensure that the benefits of a drug outweigh the risks (US Government, 2007), and may do so for new drugs and drugs already on the market, depending on specific criteria. In the EU, since new pharmacovigilance legislation was implemented in July 2012, RMPs are mandatory for all applications for a new marketing authorization; they are also usually required when there is a significant change in authorization, such as a new dosage, route of administration, or indication. RMPs can also be requested by the regulator whenever there is a concern about a risk affecting the benefit-risk balance of the medicine, or be submitted on the initiative of the manufacturer for newly-identified safety issues. The EU legislation also requires a summary of the RMP to be made public (EMEA, 2013a; MHPD, 2009).

Enhancing post-market safety reporting by manufacturers

The Periodic Safety Update Report (PSUR) is an internationally-standardized report by a manufacturer to regulatory authorities on the worldwide safety of a marketed drug product (MHPD, 2012d). In Canada, PSURs are expected to cover information in six-month intervals. They may be submitted voluntarily by manufacturers, or Health Canada may request a PSUR from the manufacturer following the identification of a safety issue (MHPD, 2012d). The submission of PSURs may also be required as the result of an NOC/c or other commitment following market authorization, or be negotiated during the authorization process. Otherwise, PSURs are not mandatory under current regulations.

Unlike RMPs, which have been slowly increasing in number, there have been no clear trends in the number of PSURs over time. Data provided by Health Canada show that since 2010, the number of PSURs has declined, from a total of 72 to 28 in 2012. PSURs associated with TPD declined from 48 to 19 over this period. As with RMPs, it is unclear whether these data represent PSURs requested, received, or reviewed by Health Canada. See Appendix C for more information.

Under the FCSAP, Health Canada planned to enhance its existing program for collecting and reviewing PSURs and advance amendments to the Food and Drugs Act to give it the legislative authority to compel manufacturers to submit PSURs. Health Canada has introduced two levels of PSUR review and has set performance standards for the review of PSURs. Most recently, it announced that it was moving toward implementing ICH E2C(R2) guidance and Periodic Benefit Risk Evaluation Report (PBRER) reporting "in order to align with international best practices and reduce the burden on industry by allowing them to submit either [a PSUR or a PBRER] to satisfy the applicable regulatory requirements in Canada" (Health Canada, 2013j).

The "applicable regulatory requirement" in question is C.01.018 of the Food and Drug Regulations, which since 2011, has required MAHs to analyze adverse drug reaction data and prepare an annual summary report, which is to be submitted when requested by Health Canada. Thus, the PBRER may now be submitted to satisfy the requirement for an annual summary report.

Elsewhere, submission of PSURs has been mandatory in the EU since 2001 for all authorized products (Council Directive, 2001, p. 96), although the new pharmacovigilance legislation waives the obligation to submit PSURs routinely for generic medicinal products and well-established medicinal products (EMEA, 2013d, p. 7). Similarly, since at least 1996, the FDA has had the authority to require applicants to submit post-market periodic safety reports for each approved application, on a quarterly basis for the first three years following the US approval date and annually thereafter (US Government, 1996, sec. 314.80(2)).^{Footnote 30}

In addition to mandating submission of periodic safety reports, both the EMA and the FDA, unlike Health Canada, have the legislative authority to compel MAHs to prepare and submit post-authorization safety studies (EMEA, 2013b; FDA, 2011a).

Implementing service standards for post-market activities

As part of the effort to implement a standardized approach to post-market surveillance, MHPD has established performance targets for some of its post-market surveillance activities. For its targets related to adverse reaction reports, MHPD has set a goal to meet the standard in 95% of cases; for its targets related to other post-market activities, MHPD has set a goal to meet the standard in 90% of cases. Targets are currently under development for causality assessments, for the content development of risk communications, and for the dissemination of risk communications (MHPD, 2012c). See Appendix C for more information.

MHPD adopted unofficial performance standards for signal assessments, based on the number of workdays between the signal assessment being assigned and the assessment being approved, in April 2007. The service standard varied according to the signal's assigned priority level: 200 working days for low-priority signal assessments; 130 working days for medium-priority assessments; and 80 working days for high-priority assessments. These performance standards became official in November 2011, but were subsequently adjusted in April 2012 to a single 130-day target. Program representatives indicated that these changes were made because of MHPD's decision in January 2012 to stop assigning priority levels to signals; however, the rationale for this decision is unclear. A discussion of the timeliness of Health Canada's post-market activities is in Section 6.7.

Expanding partnerships and use of existing databases

Finally, Health Canada is emphasizing more active post-market surveillance by partnering with external organizations and using existing databases. For example, under the FCSAP, Health Canada initiated the development of the DSEN. The DSEN is being developed as a virtual network connecting centres of excellence across the country, led by a few primary centres with distinctive leadership in a thematic area of research methodologies and competencies (Risk Sciences International, 2012). The DSEN is expected to improve Canada's ability to engage urgent research issues immediately after they appear and to provide national coordination for post-market surveillance, ultimately informing regulatory decision making.

MHPD is also examining the potential for using existing databases and electronic health records (e.g., provincial databases, Canadian Institutes for Health Information, Statistics Canada) as potential sources for studies to support surveillance and signal detection (MHPD, 2011c). Since 1996, MHPD has partnered with the Canadian Paediatric Society in the Canadian Paediatric Surveillance Program (CPSP), which collects data from over 2500 paediatricians and paediatric sub-specialists monthly to monitor rare diseases in Canadian children (Canadian Paediatric Society, n.d.). The CPSP has grown from three conditions under study in the pilot year to 50 conditions currently under study (Canadian Paediatric Society, n.d.).

The US FDA's Sentinel System is one model of active post-market surveillance using existing databases and electronic health records. Launched in 2008, the Sentinel System was required by legislative changes in the Food and Drug Administration Amendments Act (Section 905), which called for active post-market safety surveillance and analysis (FDA, 2008). The system is currently operating as a "Mini-Sentinel" pilot project involving over 20 partners. Key elements include active surveillance of post-market activity through assessment of routinely collected electronic health care data in response to FDA concerns; assessment of changes in the use of medical products in response to FDA regulatory action; use of a distributed data network in which data partners retain control over data, but use standardized computer programs within their institutions and share aggregated results; and quality control (e.g., testing and improving on statistical methods) (Mini-Sentinel, n.d).

Several external key informants encouraged Health Canada to coordinate the collection of pharmacovigilance data from other federal departments, as well as from provincial governments and other bodies, such as poison control centres. They also encouraged sharing of post-market data with regulators in other jurisdictions, since safety issues will come to light more quickly when a larger population is monitored; in turn, this will allow regulators to respond more quickly to health and safety risks.

5.4.5 Compliance and enforcement

While compliance and enforcement is often thought of as the final step in the drug regulatory process, Health Canada is responsible for monitoring compliance and enforcing the regulatory framework for human drugs at both pre-market and post-market stages. To this end, the Inspectorate conducts monitoring inspections to ensure manufacturer compliance with regulatory requirements relating to GCPs, GMPs, and GVPs, as well as monitoring inspections to ensure compliance with regulatory requirements for controlled substances. Health Canada also conducts compliance verifications in response to specific complaints or identified risks. In the event of noncompliance, the Inspectorate clarifies what is necessary to achieve compliance, and may take enforcement action if the regulated party does not voluntarily undertake compliance measures (see the list below for a summary of available voluntary and enforcement measures).

Available compliance and enforcement measures

Voluntary measures

Consent to forfeiture
Product detention
Product disposal
Stop-sale
Recall

Regulatory/enforcement measures

Customs activities
Injunction
Prosecution
Forfeiture
Public warning or advisory
Letters to trade and regulated parties
Regulatory stop-sale
Search and seizure
Seizure and detention
Suspension or cancellation of marketing authorization or product license
Refusal, suspension, or amendment of establishment license
Warning letter

Strengthening compliance tools and approaches, with a view to achieving increased industry compliance with the regulatory framework for human drugs (pharmaceuticals and biologics), has been a key objective of Health Canada during the evaluation period. Activities in this area include implementing new authorities and tools for compliance and enforcement, strengthening inspections of clinical trials, and introducing a risk-based approach to GMP inspections, among others. Planned enhancements to Health Canada's authorities for compliance and enforcement were not implemented since proposed legislative changes did not proceed.

Implementing new authorities and tools for compliance and enforcement

Both the Blueprint for Renewal and the FCSAP describe Health Canada's intention to seek new legislative and regulatory authorities for compliance and enforcement, such as authorities to order corrective actions and modernization of the current fines and penalties framework (HPFB, 2007a). Bill C-51 included new administration and enforcement measures, including mandatory recalls of therapeutic products and federal power to recall the remaining stocks of a drug once it has been taken off the market due to safety concerns. It also included a modernized framework for monetary fines and penalties, substantially increasing the fines for non-compliances. As already noted, Bill C-51 did not become law.

To date, Health Canada has not obtained these authorities, and it is unclear whether it intends to pursue them in future. It is also unclear whether plans to develop non-legislative tools for noncompliance (Inspectorate documents mention a "ticketing scheme" for noncompliance, for example) have materialized. In interviews, some external key informants expressed concern about what they perceived as the relatively limited enforcement options available to Health Canada, and noted that the adoption of a risk-based approach to compliance and enforcement means that Health Canada may be less likely to respond to issues perceived as presenting a low risk to consumers. Furthermore, while risk-based approaches to enforcement and compliance are appropriate in a context of limited resources, they do not necessarily provide a strong understanding of compliance across the regulated industry as a whole, nor are they always well-equipped to identify new or emerging risks (Sparrow, 2000, p. 290).

Strengthening clinical trial inspections

Health Canada introduced a clinical trial inspection program in 2002 with the twofold objective of protecting the safety of trial participants and verifying the quality of clinical trial data. Under the inspection program, up to 2% of all Canadian clinical trials (including human drugs and biologics) are selected for inspection each year. Health Canada estimates that there are approximately 4,000 clinical trials underway in any given year in Canada; given a 2% target, this represents approximately 80 inspections each year. Site selection is based on risk criteria, including the number of clinical trials conducted at the site; the number of subjects enrolled; the number of serious unexpected adverse drug reactions at the site; and observations made during past inspections. Two ratings are possible: compliant or non-compliant.

In recent years, both the OAG and the Senate Standing Committee identified shortcomings in Health Canada's approach to clinical trial inspections, including failure to collect regularly all of the information necessary to assess risk criteria for site selection, to meet the 2% target for inspections, and to set timelines for notifying sponsors of non-compliances (OAG, 2011; SSCSAST, 2012). In response, Health Canada has committed to publishing annual reports on its clinical trial inspections, has revised its inspection procedures to include timelines for key steps in the process, including notification of non-compliant ratings and review of proposed corrective measures, and has developed a risk-based process for selecting clinical trial sites for inspection. The risk-based criteria used to select sites for inspection include, but are not limited to, the phase in the drug development process, the complexity of the trial design, including its subject population, its level of risk to Canadians, novel therapies/dosage forms, significant or frequent reports of adverse events, notices from sponsors of protocol deviations, and emerging themes or trends (Inspectorate, 2013). According to Health Canada, other considerations, such as international best practices, may also be used.

Health Canada indicated that training on the new process was provided to Inspectorate GCP staff in May 2013 and TPD reviewers in October 2013. The new process was implemented on June 1, 2013, with a phased-in approach using both the old and the new selection systems. Once fully implemented and documented, the risk-based process for site selection may help to address concerns expressed in interviews regarding a lack of transparency with respect to how clinical trials are selected for inspection.

A few external key informants expressed concern that the inspection program itself focuses on factors that in their view are irrelevant to the safety of trial participants or the quality of the research, or already subject to oversight from other sources.

Enhancing drug establishment licensing and GMP inspections

Establishments engaged in fabrication, packaging/labelling, testing, importing, distributing, and wholesaling of drugs are required to obtain a Drug Establishment License (DEL) in order to operate legally. To receive a DEL, an establishment must pass an assessment verifying compliance with Part C, Divisions 2 to 4 of the Food and Drug Regulations, which relate to Good Manufacturing Practices (GMPs). This includes requirements for the premises, equipment, personnel, sanitation, raw material testing, manufacturing, quality control, packaging material testing, finished product testing, record-keeping, sample retention, sterile product handling, and product stability (GoC, 2012c). GMP inspections of domestic establishments occur in cycles of 24 months for fabricators, packagers/labellers, and testing laboratories, and 36 months for importers, distributors, and wholesalers. New establishments that are applying for a DEL must be ready for an inspection when submitting the application, as the Inspectorate works to perform initial inspections within three months following the receipt of the request. A regular inspection follows the initial inspection within 12 months.

Domestic establishments are rated as either compliant (C) or non-compliant (NC) according to the risk given to observations noted during inspections (Inspectorate, 2012f). There are three levels of risk, with Risk 1 being the highest level of risk and Risk 3 being the lowest. If one or more observations are classified as Risk 1, the overall inspection rating may be NC.

Until recently, DELs were required with respect to drugs in dosage form and bulk intermediates of Schedule C (radiopharmaceutical) and Schedule D (biological) drugs (Inspectorate, 2012e). In May 2013, Health Canada followed through on a long-standing commitment to extend the requirements for GMP and DELs to APIs (Health Canada, 2013e). The new regulations, which will come into force in November 2013, harmonize Health Canada's requirements with those of its counterparts in the US, the EU, Australia, and Japan, where GMP requirements for APIs have been in place for approximately the past decade (ICH, 2013).^{Footnote 31}

GMP compliance of domestic sites

Over the period of the evaluation, Health Canada reviewed its domestic GMP inspection program, intending to improve effectiveness and create efficiencies that would allow it to attain stability in the program and avoid backlogs, which had been a significant problem spanning several years (Inspectorate, 2007a). Recommendations arising from the review addressed piloting risk-based inspections, inspections of establishments with many sites, and "targeted add-on to inspection" (Inspectorate, 2012b, p. 6). Since the review was completed, Health Canada has eliminated renewal inspections that are normally conducted when annual establishment licenses expire; this is expected to enable inspectors to focus on high-risk drug establishments (OAG, 2011, p. 29). It is unclear what other steps have been taken to implement the recommendations arising from the review. The move toward risk-based inspections is consistent with the approach taken by the FDA.^{Footnote 32}

Recent events have focused attention on Health Canada's GMP inspection program.

In 2011, the FDA issued a Warning Letter to Sandoz Canada's Quebec plant after three inspections conducted that year revealed that the company failed to follow proper procedures "to prevent microbiological contamination of drug products purporting to be sterile" and for incomplete or inaccurate documentation and investigation practices (FDA, 2011b; Globe and Mail, 2012a). The repercussions were extensive. Sandoz scaled back production of certain drugs (mostly painkillers, antibiotics, and anaesthetics) to upgrade operations, leading to drug shortages and a cancellation of some elective surgeries (CBC News, 2012). Ultimately, Health Canada announced plans to fast-track new sources of certain medications in response to the shortages (CBC News, 2012).
In February 2013, the FDA issued a Warning Letter to Apotex Inc. after its inspections revealed "significant violations" of current GMPs, and threatened to impose an import ban on Apotex products manufactured at two Toronto-area facilities (FDA, 2013f). Canadian inspectors had not inspected the facilities in question since 2011 (Globe and Mail, 2013b). The FDA had previously imposed a two-year import ban on Apotex products in 2009 after inspections revealed major deficiencies in current GMPs (Globe and Mail, 2013b).

In interviews, some external key informants expressed concern about the training and qualifications of inspectors, noting that they have observed inconsistent interpretation of standards and an inconsistent quality of inspections. From Health Canada's perspective, the reasons for the discrepancies between its findings and the FDA's are complex, and are more likely due to differences between the two regulatory agencies in their approach to inspections and the scope of the inspections.

GMP compliance of foreign sites

Foreign sites that manufacture drugs for import into Canada by Canadian manufacturers are, like domestic sites, subject to establishment licensing and GMP requirements, but are seldom inspected by Health Canada. Rather, the Inspectorate assesses the initial application for a foreign-site DEL, and the GMP compliance of foreign sites, based on evidence provided by the sponsor. The evidence that must be provided differs depending on whether or not the site is located in a country with which Health Canada has a Mutual Recognition Agreement (MRA), which is an agreement between Health Canada and another country on the equivalence of their respective GMP compliance programmes (Health Canada, 2009a). Canada currently has MRAs with the European Community, Switzerland, Iceland, Liechtenstein, Norway, and Australia (Inspectorate, 2012i). However, Health Canada estimates that 76% of the drug products imported into Canada come from countries with which it does not have an MRA; in the past five years, the Department has undertaken 35 inspections of foreign sites (Cassels, 2012). In the case of APIs, Health Canada indicates that importation accounts for 85% of the Canadian market, with China and India being the major suppliers (Health Canada, 2012c).

The lack of GMP inspections of foreign facilities in countries with which Canada has not signed an MRA was a concern for some external key informants, who worried about the potential for Canadians to be exposed to low-quality or counterfeit health products. As an example, in 2013, Health Canada warned Canadian women about recalls of two generic birth control pills after packages were found to contain a placebo pill where an active pill should have been; a third recall was recently announced since the manufacturer "was unable to rule out the possibility" that the pill was impacted by the same issue (Globe and Mail, 2013a). Two of the recalled pills were manufactured in India (Toronto Star, 2013b); the other in Spain by Apotex Inc. The extension of GMP and DEL requirements to APIs is expected to help to address the possibility that low-quality or counterfeit health products will appear in the Canadian market.

Industry key informants, for their part, asserted that Health Canada's approach to foreign-site inspections places an undue burden on stakeholders who import health products from abroad (e.g., the low frequency of inspections forces importers to assume responsibility for the compliance of firms in other countries). It was suggested by industry that the process of completing GMP inspections of foreign sites would be simplified if Health Canada were willing to accept inspections carried out by foreign regulators or foreign-approved third-party auditors.

In the US, the FDA has taken an increasingly aggressive approach to overseeing foreign facilities in the aftermath of the adulterated heparin incident (Grosh et al., 2013, pp. 30-36). Despite these efforts, the literature suggests that the FDA does not have the resources to conduct a sufficient number of foreign inspections (Liu, Zhang, & Linhardt, 2009).

An initiative recently launched between Canada and the US under the Regulatory Cooperation Council (RCC) aims to address the challenges associated with GMP inspections, by increasing each country's reliance on GMP inspection reports of drug manufacturing facilities prepared by the other country, rather than unnecessarily duplicating efforts (RCC Personal Care Products and Pharmaceuticals Working Group, 2012a). The main activities included in the work plan are routine exchange and assessment of inspection reports between the countries and regular engagement of stakeholders through web postings and quarterly meetings. The countries ultimately intend to develop a framework to assure reliance on each other's inspection reports, which could include: a joint GMP database to standardize the exchange of inspection reports; routine sharing of inspection reports and significant regulatory actions/changes; joint inspections of selected establishments; and a forum in which to have discussions and adjust to regulatory changes by either country.

Despite the focus in the GMP workplan on standardizing and sharing of inspection reports, Health Canada and the FDA currently differ in their reporting approaches. While the FDA reports based on the centre responsible for regulating various categories of product - reporting inspection data separately for the Center for Veterinary Medicine (veterinary drugs), the Center for Drug Evaluation and Research (human drugs), and the Center for Biologics Evaluation and Research (biologics) (FDA, 2013d) - Health Canada aggregates GMP reporting across product lines. Health Canada indicated that this approach is taken because its approach to GMP inspections is facility-based and that it is currently not possible to track inspections by product line. Nonetheless, achieving the objectives of the GMP work plan may require a common approach to compliance reporting.

Finally, Health Canada is a member of the Pharmaceutical Inspection Cooperation Scheme (PIC/S), a body of 43 international participating authorities that have come together to harmonize GMP inspection requirements and approaches across the world. Health Canada reviews PIC/S inspection reports to approve foreign sites for Canada importers.

Other initiatives to strengthen compliance and enforcement

Finally, Health Canada has undertaken several other initiatives to strengthen compliance and enforcement of the regulatory framework for human drugs.

Introducing post-marketing reporting inspections. In 2004, Health Canada introduced the Post Market Reporting Compliance Program (PMRC) with the objective of "verify[ing] manufacturers are in compliance with the regulatory requirements for the receipt, analysis, and submission of drug safety information to Health Canada" (Inspectorate, 2012a, p. 2). In 2013, the program was renamed as the Good Pharmacovigilance Practices (GVP) inspection program; Health Canada representatives reported that this change was made in order to more closely align with international regulators. In February 2013, a risk-based approach for inspection scheduling was implemented, along with new guidance for industry.
Strengthening oversight of imported products. Health Canada strengthened regulatory oversight of imported products by developing the National Border Integrity Program in collaboration with the CFIA, the CBSA, and the FDA. The objective of the Border Integrity Program is to strengthen Health Canada's ability to make and support admissibility decisions at the border relating to health products. Pillars of the program include the authority to inspect and take samples of health products intended for importation, request the CBSA to target and detain shipments for an admissibility determination, and engage in compliance and enforcement action; use of partnerships and best practices; awareness and education activities; and efficient use of information technology through participation in the Single Window Initiative led by the CBSA, which is intended to create an automated, risk-based approach to identifying high-risk goods and accelerating the flow of low-risk goods (Inspectorate, 2010a).
Developing a policy on counterfeit health products. In a related initiative, Health Canada issued a policy on counterfeit health products in 2010, the primary objective of which is "to manage the risk to Canadians and to have the counterfeit product removed from market using the most appropriate level of intervention and notifying parties at risk" (Inspectorate, 2010c). According to Inspectorate documentation, an Anti-Counterfeit Strategy and phased implementation plan have been developed; an Anti-counterfeit Taskforce has been created; and the Inspectorate and the Royal Canadian Mounted Police have collaboratively developed a Memorandum of Understanding (MOU) (Inspectorate, 2009a, p. 24). However, the evaluation did not find further information on the phased implementation plan, the Anti-Counterfeit Task Force, or the MOU.

5.4.6 Communications and stakeholder engagement

The preceding sections described Health Canada's activities in relation to the four main stages of the regulatory process: clinical trials, submission review and market authorization, post-market surveillance and monitoring, and compliance and enforcement. Throughout the process, Health Canada also communicates and engages with stakeholders such as consumers/patients, health care providers and industry. Over the evaluation period, Health Canada has pursued two main objectives in relation to stakeholder communications and engagement: providing Canadians with more information on health products, including more timely and accessible information, and making the regulatory system more open to consumer and other stakeholder input and involvement. Its activities in these areas are described below.

Enhancing the transparency of the drug review process

Consistent with its objective of providing Canadians with more information on health products, Health Canada launched the Summary Basis of Decision (SBD) initiative in 2005 to enhance the transparency of the drug review process. SBD documents provide health care professionals, consumers, and patients with the scientific and benefit/risk-based considerations involved in granting market authorization. Between 2005 and September 2012, a total of 131 SBDs were published for human drugs.

An evaluation of the first phase of the initiative, along with the OAG's 2011 report on regulating pharmaceuticals, resulted in several changes in Phase II, including reducing the target time for SBD publication, limiting the ability of MAHs to request changes and to appeal the SBD, introducing a question-and-answer (Q&A) format and more information on Health Canada's risk/benefit analysis, and publishing a post-authorization activity table (PAAT) for eligible products (TPD, 2012c). Phase II of the SBD project applies to new drugs involving new active substances, as well as SEBs, authorized after September 1, 2012.

The PAAT was introduced in part to address the OAG's concerns that Health Canada was failing to disclose information related to NOC/cs, rejections, and withdrawals of new drugs (OAG, 2011, p. 18) . PAATs provide ongoing information about approved products. They include a brief summary of activities that affect the safe and effective use of the product, such as information related to submissions for a new use of the product (whether Health Canada's decision was positive or negative), submissions filed in order to meet conditions (for products approved under the NOC/c Guidance), and regulatory decisions such as the cancellation of the DIN.

To date, Health Canada has not introduced SBDs for negative decisions - although it has indicated that it may consider this in future (TPD, 2012c, p. 1) - and does not provide the public with updates on progress made by manufacturers in fulfilling post-market conditions. By comparison, the FDA provides updates on progress made by manufacturers in fulfilling post-market conditions, such as progress on post-approval studies. In addition, the FDA's Drugs@FDA website provides a searchable database containing documentation relating to approved brand-name and generic prescription and OTC human drugs and biological therapeutic products, consisting of the entire drug approval package (including the FDA's complete review reports, as well as approval letters, though not the submissions themselves) and all versions of product labels for products approved since 1998. However, a summary document such as the SBD does not appear to be available (FDA, 2013a).

In the EU, the EMA publishes a European Public Assessment Report (EPAR) for every medicine granted a central marketing authorization by the European Commission. The information available includes a summary in Q&A format and the package leaflet. Information is also provided on medicines that have been refused a marketing authorization or that have been suspended or withdrawn after being approved (EMEA, 2013e).

Some external key informants stated that in their view, Health Canada releases relatively little information about the pre-market review process to the public. They noted that Health Canada does not provide information about what drugs are currently under review; it provides relatively little information about how the review was carried out and what the reviewers concluded about the products in question; and it provides no information on pipeline meetings with industry. Furthermore, in the case of NOC/c, Health Canada provides no information about what measures companies are taking to satisfy the conditions imposed when they were initially approved for sale in Canada. It was suggested that if issues associated with a drug are serious enough to merit attaching conditions to approval, Health Canada should be enforcing their fulfillment, as well as reporting to the public on manufacturers' compliance with conditions. The evaluation could not determine the extent to which Health Canada currently monitors sponsors' compliance with conditions imposed with NOC/cs.

Enhancing drug product labelling

Health Canada has undertaken several initiatives to improve the quality and availability of easy-to-understand drug product labelling. Drug product labelling refers to the product monograph, materials included in the product's packaging, and materials supplied to the consumer at the time of purchase. This includes package labelling, leaflets, fact sheets, and any other material containing drug product specific information (TPD, 2011). Having access to clearly-written and accurate drug information is important because it enables health professionals and consumers to make informed decisions about drug therapies (TPD, 2011). Initiatives in this area include the following:

Introducing enhancements to the product monograph. A product monograph is a factual, scientific document that includes information on the drug's properties, conditions of use, and any other information required for optimal, safe, and effective use. A review of the product monograph is part of the drug submission review process, and an NOC is not given if a product monograph is not available. The product monograph consists of three sections, each aimed at different audiences: health professionals, scientific professionals, and (since 2004) consumers (TPD, 2008b). In 2008, as part of the FCSAP, Health Canada began publishing product monographs, including the consumer portion, on its website through the Drug Product Database (DPD).^{Footnote 33} Since then, Health Canada has also revised the product monograph template to include a new "boxed message insert" for overdose management information (TPD, 2010c).
Undertaking to develop common product monographs with the FDA for OTC drugs. Under the RCC, Health Canada and the FDA are undertaking a project to develop and adopt aligned monograph elements (including indications, warnings, conditions of use, and other information) for low-risk products, although it is important to note that the main objectives of this initiative are not necessarily to enhance product labelling, but rather "to streamline costs for manufacturers and distributors and enhance consumer access to these types of therapeutic products on either side of the border" (RCC Personal Care Products and Pharmaceuticals Working Group, 2012b, p. 2).
Amending regulations to require manufacturers to list non-medicinal ingredients (NMIs) on non-prescription drug labels. Health Canada amended regulations to require manufacturers to list NMIs on non-prescription drug labels, which is expected to help inform consumers who have a history of adverse reactions to certain NMIs, reducing adverse reactions and eventually reducing costs to the health care system. The proposed changes became official regulation in May of 2010 and came into force in 2012 (GoC, 2010).
Introducing labelling requirements for some specific drugs. Health Canada has introduced labelling requirements for certain orally administered paediatric non-prescription cough and cold products (TPD, 2008c); produced a guidance document on acetaminophen labelling (HPFB, 2009); and held consultations on a draft guidance document for acetylsalicylic acid labelling (TPD, 2012a, 2012b).
Introducing regulatory amendments concerning labelling, packaging, and brand names of drugs for human use. Most recently, in June 2013, under the Plain Language Labelling Initiative - a component of Phase I of Health Canada's plans for regulatory modernization - proposed amendments to the labelling requirements under the Food and Drug Regulations were published in Canada Gazette. The proposed regulations would introduce a general requirement for drug labels to use plain language and a format or presentation that does not impede comprehension. They would also introduce a number of new pre-market requirements, including listing of additional company contact information for consumers who experience a problem with the product; a standard table format for non-prescription drug labels; submission of mock-up labels and packaging as part of the pre-market submission process; and codification of current policy on look-alike/sound alike (LA/SA)^{Footnote 34} products (GoC, 2013).

Similar requirements for plain language labelling have been implemented elsewhere. The EU implemented requirements for standardized, easy-to-read labelling over a decade ago. Directive 2001/83/EC indicates that labelling for all medicinal products (including non-prescription products) must be "easily legible, clearly comprehensible, and indelible," follow a mandatory EMEA format to ensure consistency, and provide full colour mock-ups for review (EMEA, 2009). In the US, the 1999 OTC Drug Facts tableregulations required manufacturers to implement easy-to-read, standardized labels for most OTC drug products by 2002 (FDA, 2009). In 2006, the agency introduced the Physician Label Rule (PLR) Requirements for prescribing information (i.e., package inserts), which sought to enhance the safe and effective use of prescription drug products by giving health care professionals clear and concise information that was easier to access, read, and use (FDA, 2013e). The PLR Requirements regulation requires the prescribing information to have specific pieces of information in summary "highlight" format. ^{Footnote 35}

The proposed regulations would align Health Canada with its international counterparts in requiring easy-to-understand product labels. Nevertheless, Health Canada currently has limited ability to require a manufacturer to modify a product monograph or label once a product has received an NOC (TPD, 2011).^{Footnote 36} By comparison, the FDA has the authority to require product labelling to be updated as new safety information becomes available about the drug (US Government, 2012b). For example, the FDA has the authority to require a boxed warning to be placed in both the summary of prescribing information (highlights) and in the full prescribing information pamphlet; the boxed warning contains a bold text warning that the drug may cause serious injury or death and provides a brief explanation of the risk before referring readers to the appropriate section of the prescribing information pamphlet for more information (US Government, 2012a). Thus, the FDA uses the label's boxed warning as a tool to communicate safety and risk information to physicians (FDA, 2010).

Similarly, the EU's new pharmacovigilance legislation requires medicines that are subject to additional post-market monitoring to include an inverted black triangle on the drug product's leaflet, along with statements indicating that the product is undergoing additional monitoring and encouraging patients to report adverse reactions (EMEA, 2013c). The symbol and statements are included on biologics, biosimilars, and pharmaceuticals containing new active substances authorized as of January 1, 2011, as well as on products undergoing post-authorization safety studies, or products that are subject to conditions or restrictions on safety and efficacy, as per a risk management plan (European Parliament, 2010a, 2010b).

Enhancing post-market safety and risk communications

Since 2005, the MedEffect website has been Health Canada's main mechanism for communicating post-market safety and risk information to health professionals and the public. The MedEffect Canada Initiative was created to improve access to safety information and adverse reaction information for marketed health products, while also providing a single window approach to post-market surveillance activities. The single window approach is intended to provide the public and health professionals with centralized, easy-to-find information online, with the goal of increasing awareness of the importance of reporting adverse events related to health products, while also making it easier for health professionals and members of the public to report adverse events.

Until recently, Health Canada published safety and risk information regarding health products on the "advisories, warnings and recalls" link on the MedEffect website. Other distribution methods were and are still used, including mail-outs and hard copy publications, dissemination to professional associations to encourage posting on their website and publication in their journals and newsletters, and distribution to licensing bodies, provincial ministries, and foreign regulatory agencies. In addition, the free MedEffect e-notice delivers advisories, warnings, and recalls; the quarterly Canadian Adverse Reaction Newsletter; and MedEffect content updates directly to approximately 20,000 subscribers. To communicate product safety risks, Health Canada uses a variety of risk communications documents, targeting the public and health professionals and intended for high-, medium- and low-risk situations.

Analysis of the risk communications relating to human drugs posted on MedEffect shows that between 2005 and August 23, 2012 (MHPD, 2012b), there were 419 risk communications related to human drugs posted on MedEffect, of which over half (51%) targeted the public and one third (34%) targeted health professionals. The analysis also revealed some limitations associated with the risk communications. For example, communications concerning product recalls did not always clearly identify whether a product recall was being communicated. Furthermore, until recently, information on product recalls was available in two locations on Health Canada's website: on MedEffect's "advisories, warnings and recalls" site and on the Drug Recall Listing, leading to the potential for confusion and inconsistency in the information posted on the two sites.^{Footnote 37}

In February 2013, Health Canada launched the Recalls and Safety Alerts Database, a revised system for disseminating risk communications to health professionals and the public.^{Footnote 38} The new database seems to be an improvement over the "advisories, warnings and recalls" listing on MedEffect in several ways, including the availability of an advanced search feature and a new format for risk communications. All risk communications are now classified into one of two main categories - advisories and recalls - which should eliminate any uncertainty as to whether or not a product recall is being communicated.^{Footnote 39} Even more recently, in mid-March 2013, Health Canada eliminated the online Drug Recall Listing and consolidated product recall information on the Recalls and Safety Alerts Database. This change has eliminated the potential for inconsistency, though there is no longer a complete listing of recalls of all health hazard types available on Health Canada's website.

To further improve the risk communications process, MHPD indicated that it is currently in the process of reviewing its existing performance targets for the content development of risk communications and for the dissemination of risk communications, which are contained within its guidance document for industry on the issuance of risk communications. The guidance indicates that health professional communications "should be developed and disseminated within 12 working days after the date on which the Request Letter is received" and that the accompanying public communication "should be issued 3 working days after the issuance" of the health professional communication (Health Canada, 2010b, p. 7). The Request Letter refers to a letter sent by Health Canada to the MAH, outlining the nature of the safety issue and requesting the MAH to draft risk communications for health professionals and the public. MHPD indicated that there is currently no specific timeline for the issuance of the Request Letter; this depends on the nature of the risk.

Additionally, MHPD indicated that Health Canada is looking at areas where more focused improvements in risk communications could be made. Finally, HPFB is drafting internal guidance on the risk communication process for human health products, detailing the process(es) to be followed and the responsibilities of the parties involved, including the science directorates (i.e., TPD in the case of pharmaceutical drugs), the Inspectorate, MHPD, the Communications and Public Affairs Branch (CPAB), the MAH, and the Assistant Deputy Minister (ADM) (Health Canada, 2013g).

Opening the regulatory system to stakeholder input and involvement

In accordance with the established process for regulatory development at the federal level, Health Canada conducts public consultations on proposed amendments to the Food and Drug Regulations and other related regulations. Health Canada has also undertaken several initiatives to make the regulatory system more open to stakeholder input and involvement. Some examples are listed below.

Developing a Public Involvement Framework in 2005, following internal and external consultations with industry, patient and consumer groups, as well as academia and health professionals. The framework is intended to guide public involvement activities across all of the Branch's responsibilities (HPFB, 2005b, p. 1).
Developing, in 2007, a policy on public input into the review process for regulated products that is intended to resolve shortcomings in the old regulatory framework, which put the onus on manufacturers to provide all information on a product submitted for approval and did not explicitly allow information from consumers, patient groups, physicians and others to be considered in the decision-making process. The policy sets out the processes to be used when the Branch "identifies a situation where decision making regarding a regulated product would benefit from public input "(HPFB, 2007b, p. 1). The policy also states the Branch would make public (via the Health Canada website) reports on the input received from the public. Such reports would state the objectives and methods of the public input and provide an overview of the input received. Finally, the policy provides a definition of confidential business information and outlines HPFB's treatment of such information.
Establishing various scientific and expert advisory panels and committees to guide regulatory and policy development and advise on specific scientific issues. Some examples include the Expert Advisory Committee on the Vigilance of Health Products, which provides HPFB with external expertise on post-market surveillance, risk communications, and regulatory advertising oversight (MHPD, 2008); the Paediatric Expert Advisory Committee which provides expert advice and public involvement in the development, licensing, and continued vigilance for health products on the market destined for children, pregnant and nursing women" (Health Canada, 2012h).
Conducting public, stakeholder, and industry consultations on draft guidances and proposed policies (in addition to consultations on proposed regulatory amendments).
Meeting regularly with industry and other stakeholders through the Bilateral Meeting Program (BMP). Through these meetings, TPD meets regularly with industry and other stakeholder associations; the meetings also often involve participation from other HPFB directorates and are an opportunity for stakeholders to discuss and consult on regulatory issues of mutual interest, exchange information, and share expertise. Industry associations involved in TPD's BMP include the Canadian Association of Chain Drug Stores, the Canadian Consumer Specialty Products Association, the Canadian Cosmetic, Toiletry and Fragrance Association, Consumer Health Products Canada, Canadian Generic Pharmaceutical Association, Canada's Research-based Pharmaceutical Companies, the Canadian Pharmacists Association, the Direct Sellers Association, the Groupement provincial de l'industrie du médicament, and the National Association of Pharmacy Regulatory Authorities. Topics include submission review performance, compliance activities, cost recovery, proposed regulatory changes, and development of new guidance documents.
Posting information for industry on the application and review process for human drugs, along with associated policies and guidance documents.
Holding pre-submission meetings with sponsors prior to their filing a NDS or CTA.
Conducting compliance education and promotion activities, such as road shows and presentations, to inform industry of its obligations.

An Office of Public Ombudsman, created under the TPSI to provide responses to public and stakeholder complaints on regulatory issues (Health Canada, 2006, p. 9), does not appear to be any longer in existence.

Some external key informants believe that the stakeholder consultation and engagement process may favour industry over consumers, patients, and health care practitioners. It was noted that industry may be most likely to be aware of consultation opportunities because of its strong interest in how its products are affected by policy or regulatory changes, and may have greater resources to participate than other stakeholders, particularly in cases where travel is required.^{Footnote 40} The evaluation could not determine whether industry participates more often than other stakeholder groups in public consultations. Industry does have opportunities to consult with Health Canada through the BMP and through pipeline and pre-submission meetings, which are not available to consumers, patients, or health care practitioners.

6.0 Findings - outcomes achieved

This section presents the findings with respect to the evaluation questions on outcomes. While Health Canada has engaged in many activities that should produce the expected outcomes, data to support a definitive conclusion regarding the extent to which expected outcomes have been achieved are relatively limited. For this reason, the evaluation findings pertaining to program outcomes should be considered as a baseline.

6.1 Stakeholder awareness and understanding

In the short term, Health Canada's activities, and in particular its communication and stakeholder engagement activities, are expected to produce increased awareness and understanding by non-industry stakeholders (i.e., consumers and health professionals) of risks and benefits related to human drugs. Some relevant information is available from a series of studies sponsored by Health Canada since 2003.^{Footnote 41} Overall, these studies suggested opportunities to improve awareness among both consumers and health professionals of drug safety information available from Health Canada, although results are not disaggregated by product type (human drugs or biologics). For example, the studies found the following:

In both 2003 and 2006, about one third of consumers were aware that new drug safety information was available through Health Canada's website. However, only about 10% had accessed this information within the last six months. Conversely, about two thirds were aware of advisories and warnings issued through the media (Decima Research, 2003, 2006). Very few respondents (1%) had subscribed to the MedEffect e-Notice (Decima Research, 2006).
In 2003, just over half of health professionals were familiar with manufacturer-issued Dear Health Professional Letters (DHPLs) and the Canadian Adverse Reaction Newsletter, but fewer were familiar with Health Canada-issued DHPLs, Health Canada's online drug safety advisories, and Health Canada's electronic mailing list (Decima Research, 2003). In 2007, 12% of health professionals identified Health Canada/MedEffect as a source for new drug safety information (Environics Research Group, 2007). A qualitative study completed the same year found that very few health professionals were aware of the MedEffect website, and only one had subscribed to the e-Notice (The Antima Group & TNS Canada, 2007).
In both 2003 and 2007, about half of health professionals reported being familiar with how to report an adverse reaction. Pharmacists, and to a lesser degree, physicians, tended to be familiar with this process, while nurses, naturopaths, and dentists tended to be unfamiliar (Decima Research, 2003; Environics Research Group, 2007). Similarly, the 2007 qualitative study found a lack of awareness among health professionals on how and why adverse reaction reports should be made, and who should be making these to Health Canada (The Antima Group & TNS Canada, 2007).

See Appendix C for a more detailed summary of the findings from these surveys.

Since these studies were completed, Health Canada has not conducted any further research into the effectiveness of its risk communications or the information it provides to stakeholders. The survey of stakeholders was intended to fill this gap, but due to methodological limitations and a poor response rate, the results are unreliable and are not included in this report. As a result, information is limited to that obtained from external key informants, who generally did not express strong views of Health Canada's risk communications or the other information Health Canada provides to stakeholders. They did, however, identify a number of issues and/or suggestions for improvement, as summarized below.

External key informant perceptions of Health Canada's communications and information

Risk communications may be too technical for some members of the public to understand.
Risk communications could provide more detailed information for both health care practitioners and patients.
Risk and safety information should be released to the public in a more timely fashion.
Health Canada's website is difficult to navigate and search; the design could be improved by creating separate pages for consumers and health care practitioners or improving links within the website.
Web postings, while necessary, are not in themselves sufficient for communicating with stakeholders; Health Canada should expand its information dissemination activities through increased use of social and conventional media.
Health Canada should provide more information to the public regarding pre-market reviews and analysis of adverse reaction reports.

Health Canada has recently undertaken to follow through on its long-standing commitments (dating to at least 2007) to assess the effectiveness of its health product risk communications. The Evaluation of the Effectiveness of Risk Communications project will evaluate the effectiveness of Health Canada issued/endorsed Dear Healthcare Professional Letters, Notices to Hospitals, and Public Communications over a one- year period between September 1, 2011 and September 1, 2012 (Health Canada, 2013d). The evaluation is expected to be completed in December 2013.^{Footnote 42}

6.2 Industry awareness and understanding

In the short term, Health Canada's activities are expected to produce increased awareness and understanding by industry stakeholders of the regulatory frameworks for human drugs. As described earlier in this report, industry has opportunities to provide input into proposed regulations and policies through Health Canada's online consultation process and through the Government of Canada's gazetting process for proposed new regulations, as well as opportunities to consult with Health Canada through the BMP and through pipeline and pre-submission meetings.

In the absence of baseline data, the available evidence does not support conclusions on the degree to which industry awareness and understanding may have increased, although it does point to some potential gaps in understanding and/or areas for improvement.

Submission requirements. Most respondents to the human drugs industry survey indicated that their firm has a strong understanding of Health Canada's submission requirements for market approval. Qualitative information from the interviews with industry suggests that in some areas, greater clarity may be required. In particular, industry key informants identified a need for clearer guidance or information with respect to combination products, classification of emerging health products, and Health Canada's use of foreign reviews and foreign guidance in the drug review process.
Adverse reaction reporting requirements. A majority of respondents to the human drugs industry survey believe that Health Canada has clearly outlined the instructions and requirements for adverse reaction reporting and defined what reactions must be reported, and agreed that the mandatory reporting forms are clear and understandable. Most also agreed that their firm could provide the required information and complete the report in the required time frame.
Other requirements. Most respondents to the human drugs industry survey reported that their firm has a strong understanding of Health Canada's requirements relating to GMPs, establishment licensing, and regulatory compliance activities and related enforcement actions. A minority of industry survey respondents rated their understanding of GCP requirements as strong.

Industry key informants and survey respondents were generally satisfied with the information provided by Health Canada. A few external key informants recommended that Health Canada reduce its reliance on industry associations as a means of conveying policy and regulatory information to manufacturers, since firms that are not members of industry associations will tend to be excluded.

6.3 Safety and effectiveness

In the short term, Health Canada's activities are expected to result in increased safety and effectiveness (i.e., efficacy) of human drugs on the Canadian market. Overall, while Health Canada has in place processes and authorities that are intended to contribute to the safety and efficacy of human drugs on the market, there are also a number of gaps that, if addressed, would further contribute to this objective.

Pre-market stage

At the pre-market stage, the information requirements for pre-market reviews are complex and detailed. External key informants generally agreed that Health Canada considers the appropriate type and quality of information in pre-market review, although some would like to see Health Canada incorporate more foreign data, including foreign approvals, into the review process. Respondents to the human drugs industry survey agreed that Health Canada's pre-market reviews are rigorous, and that Health Canada has adequate pre-market processes in place to ensure the quality and the efficacy of these products, as well as processes to ensure an adequate level of product safety.

On the other hand, since many products are not tested on the population for whom they are intended, the data included in a drug submission do not always reflect the full safety or efficacy profile of the product (paediatric populations are a case in point). Furthermore, some US and Canadian studies have found a link, though not a causal relationship, between the speed of the approval process and subsequent post-market safety issues.^{Footnote 43} An important empirical question for regulatory agencies, therefore, is whether faster approval times result in more unsafe products reaching the market.

More generally, the existence of a rigorous review process does not necessarily support the conclusion that, over time, safer and more effective drugs are being made available on the Canadian market, or that the overall safety and effectiveness of drugs on the market has improved. In fact, generic human drug submissions represented, on average, 43% of all human drugs submissions between 2004 and 2012 and, in recent years, have been growing as a proportion of all submissions, reaching 52% in 2011 before declining to 42% in 2012. By definition, generic drugs are no safer and no more efficacious than what is already on the market.

Post-market stage

At the post-market stage, Health Canada, along with other regulators, has recognized the need to enhance post-market surveillance in order to better protect patient health and safety. As described in Section 5.4.4, progress has been made in some areas, and the available data suggest that Type I recalls - issued when there is reasonable probability that use of or exposure to the product will cause serious adverse health consequences or death - are relatively rare. Between 2005 and August 2012, 10% of the recall notices relating to human drugs were Type I; see Appendix C for more detailed information.

That said, Health Canada has not implemented some elements of a strengthened approach to post-market surveillance that are in place elsewhere, namely the authority to compel RMPs and post-market studies, and has recently acknowledged that the absence of these authorities is a limitation of its current approach. Other limitations acknowledged by Health Canada include not addressing issues related to off-label use and lack of follow-up or risk evaluation in populations not studied in clinical trials.

In the area of compliance and enforcement, Health Canada does not currently have the authority to compel industry to initiate recalls, although Bill C-51 included such a provision. Similarly, Health Canada has not obtained legislative authorities to increase penalties for noncompliance, and some external key informants expressed concern about what they perceive as the relatively few enforcement options available to Health Canada to address noncompliance issues. Other concerns raised by external key informants relate to the extent to which Health Canada monitors compliance with NOC/cs and oversees foreign manufacturing facilities.

A majority of respondents to the human drugs industry survey agreed that Health Canada's post-market surveillance and monitoring activities, as well as its compliance and enforcement activities, effectively support an adequate level of human drug safety and quality in Canada.
Finally, it is important to note that some evolution may be taking place with respect to the way in which regulatory agencies understand product safety. The FDA's recent decision not to approve generic forms of OxyContin on the grounds that it poses "an increased potential for certain types of abuse" (FDA, 2013b) suggests a view of product safety that incorporates consideration of how a drug is used once it is on the market. Such a concept of product safety may be more consistent with the product lifecycle approach to regulation - which emphasizes real-world drug use and safety information - than the traditional conception of safety, based on an assumption of "use as indicated".^{Footnote 44}

6.4 Industry compliance

In the short term, Health Canada's activities are expected to lead to increased industry compliance with regulatory requirements relating to human drugs. Overall, while the available information suggests a high level of industry compliance, a variety of data limitations, including inconsistencies in reporting over time, inability to extrapolate compliance rates to the regulated industry as a whole, reporting that is aggregated across several product lines, and minimal information on the nature, seriousness, frequency, or prevalence of non-compliances, hamper a more detailed understanding.

The following information is available on industry compliance:

Clinical trial compliance. Compliance among inspected clinical trial sites is high. According to a summary report of inspections conducted between April 2004 and March 2011, 92% of inspected sites received a compliant rating (Inspectorate, 2012g). The majority of the observations noted during these inspections pertained to deficiencies in quality systems and procedures (primarily GCP and records). Of the observations, 54% were considered minor, 39% were considered major, and 1% were considered critical. Given that only 2% of sites are targeted for inspection, this compliance rate cannot be generalized to the entire clinical trial industry. Furthermore, since GCP inspection and compliance data is not tracked by product category (human drug or biologic), it is not possible to separately analyse GCP compliance within these two industries.
GMP compliance. There is a high level of compliance with GMP among domestic drug establishments. Between 2005-2006 and 2010-2011, GMP compliance among inspected establishments ranged between 94% and 98% (Inspectorate, 2007b, 2010b, 2012h).^{Footnote 45} Like clinical trial inspection data, GMP inspection data is not disaggregated by product category, but includes domestic GMP inspections for human drugs and biologics, as well as veterinary drugs. Therefore, it is not possible to separately analyse the GMP compliance of the human pharmaceutical drugs industry.
Compliance with post-market reporting requirements. Between 2004-2005 and 2009-2010, the PMRC inspection program found 100% compliance with post-market reporting requirements among inspected human drugs and biologics establishments. As with GMP and GCP compliance data, these data are not disaggregated by product type. It is unclear if this compliancy rate can be extrapolated to establishments in general.
Compliance of imported products. National Border Integrity Program data show the following:
- A total of 41,669 shipments of Schedule F prescription drugs were inspected in 2010-2011 and 2011-2012. Overall, 94% of Schedule F shipments were refused entry in these two years, including 24% of all shipments that were refused on suspicion of containing counterfeit product. While this may seem like a large proportion, shipments were presumably targeted for inspection based on risk criteria. Thus, this noncompliance rate is not necessarily reflective of imported Schedule F drugs in general.
- A total of 3,812 shipments of unscheduled/OTC drugs were inspected in 2010-2011 and 2011-2012, of which 13% were refused entry due to noncompliance; none were suspected counterfeit.

See Appendix C for more detailed information on industry compliance.

The available data seem to suggest that serious non-compliance is relatively uncommon. Over the period of the evaluation, however, Health Canada has not reported regularly or consistently on the nature, seriousness, frequency, or prevalence of non-compliances related to human drugs, and has focused its reporting, instead, on quantifying activities and outputs. Furthermore, much of Health Canada's reporting on compliance, including GMP, GCP, and GVP compliance data, is aggregated across multiple product categories, both pharmaceutical and biologic drugs.

That said, the Inspectorate has recently developed an annual inspection summary report that will be published on the Health Canada website. The 2012-2013 report includes a description of Inspectorate activities and outputs, describes the overall compliance rate of industry, describes the risk ratings of observations noted during inspections, and gives examples of the common observations cited in non-compliant establishments (Health Canada, 2013h). According to the report, Health Canada conducted 1,387 inspections in 2012-2013 across all of its inspection programs (covering human drugs, medical devices, natural health products, blood, donor semen, and cells, tissues and organs), made hundreds of observations requiring corrective actions, and issued 33 non-compliant ratings. The report does not contain any information on actions taken by the Inspectorate in response to non-compliance, nor does it report GMP, GCP and GVP compliance information separately for pharmaceuticals and biologic drugs.

6.5 Adoption of safe behaviours

In the intermediate term, Health Canada's activities are expected to lead external stakeholders to adopt safe behaviours associated with human drugs. The extent to which this has occurred could not be determined by the evaluation.

To date, Health Canada has not evaluated the impact of its activities on stakeholder behaviour, although it has plans to evaluate the effectiveness of its risk communications during 2013-2014. The survey of stakeholders was intended to provide evidence to support conclusions on this outcome, but achieved a low response rate and could not fulfill this role in practice.

The literature review found some evidence of unsafe behaviours and/or practices associated with the use of pharmaceutical drugs. One recent study estimated that 11% of drugs prescribed in primary practice were prescribed for an off-label indication; in 79% of these cases, there was no strong scientific evidence of the efficacy of the products concerned with respect to their intended uses (Eguale et al., 2012). The abuse of prescription pharmaceuticals is a form of off-label use that can result in addiction, overdose, and death.

While Health Canada currently does not have jurisdiction to regulate use, it can influence use indirectly through other channels, such as product labelling and the preparation and dissemination of knowledge products and risk communications. The Branch is also responsible for product reviews, and has the ability to refuse market authorization to products that are unsafe.

6.6 Use of scientific evidence and risk-benefit analysis

In the intermediate term, the HDP envisions an increase in the use of scientific evidence and risk-benefit analysis to inform Health Canada decision making. Health Canada has integrated the use of scientific evidence and risk-benefit analysis into its decision-making processes and activities, although the evaluation could not determine whether their use has increased over the period of the evaluation.

The use of scientific evidence and risk-benefit analysis is formally integrated into Health Canada's decision-making process. The Health Canada Decision-Making Framework for Identifying, Assessing and Managing Health Risks sets out an approach to decision making in which risk analysis and management activities are central (Health Canada, 2000).The framework is used to guide the HDP decision-making process, although the evaluation could not assess the consistency with which it is applied in practice.

The use of scientific evidence and risk-benefit analysis^{Footnote 46} (and/or risk-based analysis) is also formally incorporated into Health Canada's pre-market and post-market processes. Some examples are given below (note that this is not intended to be an exhaustive list).

The information requirements for human drugs submissions are complex and detailed.
MHPD has recently developed and implemented a standardized approach to post-market activities for all of the product lines regulated by HPFB. Standardized processes are in place for signal detection, assessment, and risk management, which incorporate consideration of scientific evidence and risk-benefit analysis.
The Inspectorate's recently-developed Compliance and Enforcement Risk Evaluation Guide sets out a structured approach to evaluating and managing health risks through its compliance and enforcement activities (Inspectorate, 2012c).
The Inspectorate's inspection programs, including its GMP and GCP programs, are risk-based, and are intended to allow the Inspectorate to focus its resources on areas of high risk.

In addition to incorporating scientific evidence and risk-benefit/risk-based analysis into its processes, Health Canada has established a number of expert committees and panels to provide guidance on regulatory and policy development in various areas. Although it has not implemented all of the recommendations of any advisory group, it has used some of their recommendations to guide policy and regulatory development. A few examples are briefly described below; for a more complete list, please see Appendix C.

The Expert Advisory Committee on the Vigilance of Health Products was established in 2006 to provide HPFB with ongoing expert strategic policy advice on the safety and effectiveness of marketed health products for human use. Since its inception in 2006, the Committee has advised HPFB on risk communications, consumer reporting to Canada Vigilance, signal detection and assessment, benefit-risk assessment, tracking of adverse reactions related to off-label use, and enhancing post-market surveillance through collaboration with provincial systems and researchers through the DSEN. MHPD used the Committee's advice in developing the Recommendations for the Appropriate Use of Cough and Cold Products in Children document.
The Paediatric Expert Advisory Committee (PEAC) was established in 2009 to provide HPFB with expert advice and public involvement in the development, licensing, and post-market surveillance of health products destined for children and pregnant and nursing mothers. The PEAC's examination of off-label drug use in the paediatric population led to a "Mind the Gap" study initiated by the Office of Paediatric Initiatives to address issues related to off-label drug use in children.
The Scientific Advisory Committee (SAC) on Respiratory and Allergy Therapies developed two draft documents relating to submission requirements for subsequent market entry corticosteroid products and nasal products for use in the treatment of asthma and allergic rhinitis.
Several committees have advised Health Canada regarding specific products, and their advice has informed Health Canada's decision making regarding these products. For example, based on recommendations from the Adderall XR New Drug Committee, Health Canada suspended the NOC for Adderall XR.^{Footnote 47}

A variety of other committees have been established - some fairly recently - to address metabolic and endocrine therapies, oncology therapies, pharmaceutical sciences and clinical pharmacology, bioequivalence of gender-specific drug products, and non-prescription drugs, and have been active to varying degrees. Several committees, addressing human reproductive technologies, modified-release dosage forms, opioid analgesic abuse, and the SAP, have completed their mandates, and two (advising on musculoskeletal therapies and neurological therapies) have been cancelled.

From an evaluation perspective, it is unclear how an increase in the use of scientific evidence and risk-benefit/risk-based analysis could be meaningfully achieved or measured. Instead, it may be more reasonable to expect Health Canada to use scientific evidence and risk-benefit/risk-based analysis consistently to inform decision making.

6.7 Timely regulatory response to risks

A timely regulatory response to identified risks is expected to result from HDP activities in the intermediate term. Arriving at a broad conclusion on this outcome is complicated by the variety of ways in which it may be measured and by challenges in determining when to consider risks to have first been identified by Health Canada.

Timeliness with which Health Canada develops regulatory/policy responses

One way of measuring this outcome is to examine the timeliness with which Health Canada develops regulatory and policy responses to identified risks. Often it is difficult to know when Health Canada first identified a specific risk. Moreover, while it is well-known that regulatory (if not policy) development is typically a lengthy process, there are no objective standards against which the timeliness of Health Canada's response can be assessed.^{Footnote 48}

With those caveats in mind, the evaluation found several instances in which Health Canada's response to acknowledged risks has taken longer than expected to implement. One example is the development of a regulatory framework for APIs. As described in Section 4.1, approximately 85% of APIs sold in Canada are imported, primarily from China and India, raising concerns that contaminated or substandard ingredients may result in patient harm. Health Canada announced its intent to adopt ICH Q7 and proceed with developing a regulatory framework for APIs in 2002. In May 2013, regulations requiring DELs and GMPs for APIs were pre-published in Canada Gazette I (Health Canada, 2013e). Health Canada representatives explained that the 10-year delay between the initial announcement of intent and the regulatory amendment was due to the complexity of the regulation and changes in HPFB priorities with respect to regulatory development over time. Elsewhere, including in the US, the EU, and Japan, similar regulations have been in place for the past decade.

Another example is Health Canada's response to recommendations that it enhance disclosure of clinical trial information. Health Canada promised to enhance public access to clinical trial information in the 2007 Blueprint for Renewal. A public database of clinical trial information was announced in 2013. Health Canada acknowledges that the database contains more limited information than the registries that have been mandatory in the US since 1997 and the EU since 2004. In addition to mandatory registration, disclosure of trial results has been mandatory in the US since 2007, and the EU is currently implementing similar plans.

Delays in Health Canada's response may be due to the complexity of the regulations and changing priorities over time, although other factors may also have contributed. With respect to the latter point, it is important to note that in some cases, despite having identified risks and developed a regulatory response to those risks, Health Canada has been unable to proceed with fully implementing some of its planned initiatives due to the failure of enabling legislation to be passed into law. Several planned initiatives were affected by the fact that Bill C-51 did not become law, including the implementation of a progressive licensing framework, mandatory adverse reaction reporting for health care institutions, and new administration and enforcement measures, including the power to order mandatory recalls, among others. In other cases, the evaluation could not determine why planned regulatory amendments have not proceeded. Examples include introducing regulatory authorities to compel manufacturers to produce RMPs and PSURs, plans for which were announced as part of the FCSAP.

Timeliness of response to post-market safety issues

Another way of assessing the timeliness of Health Canada's response to identified risks is to examine the timeliness of its response to post-market safety issues. Health Canada's DPR for 2011-2012 reports that 93% of post-market signal assessments for pharmaceuticals, medical devices, biologics, and natural health products were completed within service standards in that fiscal year (Health Canada, 2012a). A detailed breakdown by product category was not provided.

The OAG examined the timeliness of Health Canada's response to safety signals in its 2011 report. The OAG noted that while 34 of the 54 safety assessments it examined as part of the audit were completed within the established timelines, the Department's approach to measuring its performance did not consider the amount of time a potential safety issue may wait before an assessment begins; the amount of time an assessment may be placed on hold; the amount of time needed to obtain additional information from external parties (such as the manufacturer); and the total number of calendar days, instead of working days, taken to complete the assessment (OAG, 2011, p. 24). The OAG noted that when these factors were considered, Health Canada took at least one year to complete 34 of its 54 assessments and in some cases took significantly longer - two or three years (OAG, 2011, p. 24). The OAG concluded that the Department's "assessment of, and response to, potential safety issues is not timely" (OAG, 2011, p. 22).

To shed further light on the timeliness of Health Canada's response to post-market safety issues for human drugs (and biologics), as part of the case study on post-market surveillance, the evaluation undertook a detailed analysis of the Department's response to safety signals using administrative data provided by MHPD. Detailed data are provided in Appendix C. The analysis showed the following:

At the signal detection phase, as already reported, Health Canada relies significantly on the scientific literature and other regulatory agencies to detect safety signals for human drugs. Since it can take a substantial amount of time for the results of scientific studies to be published and for other regulatory agencies to complete their safety assessments and issue risk communications, this may have implications for the timeliness of Health Canada's signal detection activities.
In the case of human drugs, a considerable period of time can elapse between when a signal is received and when it is assigned for signal assessment. Human drugs signals waited, on average, 65 days to be assigned for assessment, and 16% waited more than one year to be assigned. By comparison, biologics signals waited, on average, 14 days to be assigned.
For human drugs, the timeliness of the signal assessment process has improved since the introduction of (unofficial) performance standards for high-, medium-, and low-priority signals in April 2007. Just over half (51%) of human drugs signal assessments assigned after April 2007 were completed within 130 working days. Signal assessments completed in this period had a median processing time of 128 working days.
In the case of biologics, 92% of signal assessments assigned between November 2008 and October 2012 were completed within 130 working days. ^{Footnote 49} Signal assessments completed in this period had a median processing time of 46 days.

The explanation for these differences in the timeliness with which human drugs and biologics signal assessments were completed is unknown, but may relate to the number of signal assessments in relation to resource levels and/or the relative complexity of the files. The evaluation could not explore the relationship between resource levels and timeliness of signal assessments, since it did not have access to information on the number of FTEs devoted to this activity.

While it is important for Health Canada to complete signal assessments in a timely fashion, it is also important, from a safety perspective, that it implement the recommended actions stemming from completed assessments in a timely fashion. The evaluation therefore examined a subset of human drugs signal assessments completed in 2010, 2011 and 2012 that produced a recommendation for a risk communication.^{Footnote 50} This analysis showed that the median time elapsed between approval of the signal assessment and posting of the risk communication was 200 days for low-priority signals, and 176 days for medium-priority signals. For the one high-priority signal assessment, 203 days elapsed between approval of the signal assessment and posting of the risk communication.

Due to small sample size, it is difficult to draw conclusions on the basis of these data. Furthermore, as already described, Health Canada's existing performance standards for the content development and dissemination of industry-issued risk communications do not incorporate the period between Health Canada's approval of the signal assessment and its request to industry to develop a risk communication.

Health Canada indicated that it is currently in the process of reviewing these performance standards. Program representatives explained that developing a risk communication involves a considerable amount of negotiation between Health Canada and the MAH, and that drafting and posting of a risk communication may be delayed until appropriate changes have been made to the product's labelling. Indeed, HPFB's draft guidance document on the risk communications process builds this negotiation into the process (Health Canada, 2013g). It is unclear what occurs if negotiations fail or are especially protracted. It is also unclear if Health Canada may, in some circumstances, issue risk communications without the MAH first making changes to the product labelling or without reaching agreement with the MAH regarding the content of the risk communication.

Finally, the evaluation attempted to analyse the total time elapsed between Health Canada's initial detection of a safety signal and the eventual outcome or action taken by the Department. Complete data were available for only a small number of completed human drugs signal assessments (n=14).^{Footnote 51} In these 14 cases, the initial detection of the safety signal occurred between 2007 and 2011, and the signal assessments were completed between 2010 and 2012. Analysis of these records shows that the total calendar time elapsed between signal detection and posting of a risk communication ranged from 232 days in the case of Pradax to 1,481 days (4 years) in the case of Avandia, with a median of 515 days (i.e., 1.4 years). Both of these signals were classified as low priority. In the absence of a larger sample for analysis, it is not possible to draw any conclusions on the basis of these data.

While it is important for Health Canada to establish and strive to meet performance standards for specific post-market activities, it is also important to understand the timeliness of its overall response to safety risks, from initial detection to action taken. Further objectivity of the analysis could be obtained by comparing Health Canada's response against that of its international counterparts.^{Footnote 52}

At present, the data necessary to complete any detailed analysis of Health Canada's post-market activities is highly dispersed and inconsistently maintained. Given Health Canada's stated intention to enhance post-market surveillance, a more comprehensive, consistent, and centralized approach to data collection and information management for its post-market surveillance activities seems warranted. While a database solution would be ideal, improvements could also be made through relatively straightforward and low-cost adjustments to the current Excel-based system. Examples include standardizing data entry through the use of predefined codes, assigning unique identifiers to files at the signal detection stage, and using these unique identifiers through all stages of the signal process, in order to enable linking of records across spreadsheets.

6.8 International harmonization

In the intermediate term, HDP activities are expected to produce increased international harmonization of regulatory requirements for human drugs. TPD defines harmonization as "the development, adoption, and implementation of international technical standards for the development, registration, and control of pharmaceuticals and medical devices," as well as "the convergence of regulatory practices and processes" (TPD, 2004, p. 9).

In this area, Health Canada has focused on developing and strengthening regulatory cooperation and work-sharing activities with key international counterparts; being actively involved in harmonization of international standards/technical requirements and in regulatory convergence initiatives; and strategically engaging with countries whose regulatory systems are in development, in order to build capacity. The evaluation found evidence that the HDP has been working toward greater international harmonization and has made progress in some areas.

Health Canada is an official observer to and active participant in the ICH, which was established by the regulatory authorities and pharmaceutical industries of Europe, Japan, and the US to harmonize technical requirements and ensure the safety, quality, and efficacy of human pharmaceuticals (HPFB, 2012b, p. 9). In this capacity, Health Canada participates in the development and revision of ICH guidance and standards, as well as a variety of committees and working groups, and chairs a special task force to examine the development of a strategic plan for ICH training. Health Canada also participates in the development of an electronic submission standard (eCTD) (HPFB, 2012b, p. 9). Health Canada has adopted some ICH standards, including the Common Technical Document (CTD) (Health Canada, 2003) and other standards described earlier in this report, although it is unclear exactly how many ICH standards and guidelines Health Canada has adopted to date.
HPFB has MRAs on GMP compliance with the European Community, Switzerland, Iceland, Liechtenstein, Norway, and Australia.
Health Canada is a member of the Pharmaceutical Inspection Cooperation Scheme (PIC/S) and shares inspection reports and participates in various inspection-related activities.
HPFB has formal information-sharing agreements with the FDA, Australia's Therapeutic Goods Administration (TGA), and the European Community.
Health Canada and the FDA participate in the RCC, the overall objective of which is to better align the two countries' regulatory approaches. Initiatives are underway in relation to developing a common gateway for electronic submissions, common monographs for OTC products, and GMPs.
HPFB and the TGA have created an Enhanced Work-Sharing Initiative, which has conducted work in the areas of generic drugs, desk-based GMP site evaluations, and OTC monographs (HPFB, 2012b, p. 36).
HPFB is involved in an Embedded Experts Initiative with the EMA, whereby an HPFB staff person is embedded within the EMA to work on joint projects. Proposed activities include observing monthly Pharmacovigilance Risk Assessment Committee meetings, implementing strategies for evolving ICH pharmacovigilance guidelines, harmonizing approaches to benefit-risk evaluation assessment standards and methodologies, and developing effective risk minimization and communication strategies. HPFB also participates in meetings of various other EMA working parties and groups related to pharmacovigilance, quality, and GCP inspections (HPFB, 2012b, pp. 33-35).
Health Canada participates in a number of initiatives of the World Health Organization (WHO). It contributes to the WHO's Prequalification of Medicines Program and International Drug Monitoring Program, and participates in the Council for International Organizations of Medical Sciences.
Other international fora in which Health Canada participates include the Asia-Pacific Economic Cooperation, the Heads of Agencies Consortium, the International Laboratory Forum on Counterfeit Medicines, the International Generic Drug Regulators' Group, the Official Medicines Control Laboratories Network, the Pan American Health Organization. the Pan American Network for Drug Regulatory Harmonization Cooperation, and many others.

All international key informants regard Health Canada as a constructive participant in bilateral and multilateral engagement, describing Health Canada as consistent, reliable, strategic, creative, well-informed, like-minded, cooperative, and constructive. Several pointed out that Health Canada is perceived as a more neutral and thoughtful participant in international fora than some of the larger players. Nonetheless, international key informants noted that Health Canada, like other regulators, may face a variety of obstacles to increased harmonization and further international collaboration, including limited resources to contribute to international activities; political or economic considerations taking priority; legislative constraints to harmonization (i.e., lack of a legislative basis for harmonized processes); increasingly prescriptive and detailed national legislation; and philosophical differences (perceived as existing primarily between the US approach and the rest of the world).

Finally, it is important to note that Health Canada's main forum for supporting access to medicines in developing countries is the Canada Access to Medicines Regime, which came into law in 2004. Under the regime, eligible countries may import generic, less expensive versions of patented drugs and medical devices from Canada (Health Canada, 2008c). In eight years, the regime has been used to send two batches of one generic drug to one country (Globe and Mail, 2012b, p. 1). In November 2012, a private member's bill to eliminate some of the obstacles that prevent Canadian companies from becoming involved in the regime was defeated in the House of Commons.

6.9 Long-term outcomes

In the long term, HDP activities are expected to contribute to reduced health risks and adverse events associated with the use of human drugs, as well as increased public confidence in these products and the related regulatory systems. HDP activities likely have an impact in these areas, although many other factors may also influence these outcomes.

Program activities such as approval of safe human drugs whose benefits outweigh their risks, post-market surveillance and monitoring, and compliance and enforcement of the regulatory frameworks should contribute to reduced health risks and adverse events associated with the use of these products. For example, removing unsafe products from the market through recalls and preventing non-compliant products from entering Canada likely avert adverse health effects in humans that would have occurred had these actions not been taken.

While these and other program activities have certainly contributed to reducing health risks and adverse events associated with the use of human drugs, finding concrete evidence of these outcomes is challenging. The available indicators lend themselves to a variety of interpretations. For example, the number of adverse reaction reports submitted to the Canada Vigilance System has been increasing steadily over time. However, this is likely due to the greater number and variety of products on the market along with increased recognition of the need or obligation to report adverse reactions, and not necessarily to an increase in unsafe products per se. Indeed, the proportion of adverse reaction reports that are classified as "serious" - i.e., that involve congenital anomaly, death, disability, hospitalization, another medically important condition, or that are life-threatening (Health Canada, 2012d) - has remained stable since 2001, accounting for just over two thirds of all reports submitted each year (MHPD, 2012a).

With respect to public confidence, survey data show a high level of confidence among consumers in the safety of drugs and the regulatory system. In 2003 and 2006, 84% and 86%, respectively, of consumers were confident in the safety of prescription drugs, while in both years, 75% were confident in the safety of non-prescription drugs (Decima Research, 2003, 2006). However, the proportion expressing confidence declined slightly in 2010, when 65% of Canadians reported feeling that the medications approved by Health Canada are safe (Nanos Research, 2010).

Most consumers also expressed confidence in the regulatory system. In 2003, 85% of consumers expressed confidence in the systems and safeguards in place to ensure the safety of prescription drugs for sale in Canada (Decima Research, 2003). In 2006, about 78% of respondents expressed confidence in how the federal government monitors and regulates drug safety and effectiveness (Decima Research, 2006).

Ultimately, Health Canada hopes to achieve a sustainable, cost-efficient, responsive, and science-based regulatory system for human drugs in Canada. The evaluation found that Health Canada uses scientific evidence and consults with stakeholders in policy and regulatory development, and recent updates to user fees for human drugs will support the sustainability of the regulatory system for these products.

6.10 Unintended consequences

External key informants identified a number of unintended consequences stemming from Health Canada's approach to regulating human drugs. For the most part, these related to the increased burden and cost associated with complying with regulatory requirements. It was also suggested that in some cases, this may ultimately have a negative effect on research, product development and/or some areas of medicine, and access to health products. The following examples were given:

Current regulations governing clinical trials may result in declining research activity in academic settings relative to industry, since only larger pharmaceutical and biotechnology companies are likely to have the resources necessary to comply with the regulations.^{Footnote 53}
Although OTC drugs compete in the marketplace with natural health products, regulations for the former are more stringent; this confers a competitive advantage on the latter.
As existing drugs developed for a new indication are sometimes not eligible for data protection in Canada, some of these products are not being marketed in the country.

The evaluation could not assess the extent to which these unintended consequences have, in fact, occurred.

7.0 Findings - efficiency and economy

Overall, the available financial and human resource information is insufficient to support an analysis of efficiency and economy. As a result, the evaluation could not assess the extent to which program resources were used as planned, whether program outputs were produced efficiently, or whether expected outcomes were produced economically.

For fiscal year 2007-2008 and earlier, financial reporting was based on Health Canada's former PAA. Under this PAA, human drugs activities fell under the Health Products and Food Program Activity. Program sub-Activities were as follows:

pre-market regulatory evaluation and process improvement
information, education, and outreach on health products, food, and nutrition
monitoring safety and therapeutic effectiveness and risk management
transparency, public accountability, and stakeholder relationships

Financial reporting was linked to these four Program sub-Activities and to sub-sub-Activities representing, at a more detailed level, the functional activities carried out by Health Canada personnel as part of the sub-Activities. However, it is not possible to link this information to what, since 2008-2009, has been considered the HDP.

In 2007-2008, Health Canada's PAA was restructured. Under this version of the PAA, the relevant Program Activity is Health Products, and the Program sub-Activities are as follows:

Pharmaceutical Human Drugs
Biologics and Radiopharmaceuticals
Medical Devices
Veterinary Drugs

Further restructuring of the PAA in 2012 resulted in the creation of a larger PDP by combining the Pharmaceutical Human Drugs Program and the Veterinary Drugs Program.

From 2008-2009 through 2011-2012, financial reporting was linked to the four Program sub-Activities listed above. Table 4 presents the available data on HDP expenditures for 2009-2010 to 2011-2012, with and without revenues. It is important to note that the available expenditure data do not include overhead, which is not coded at the sub-Program level but only at the Program level (i.e., Health Products). As a result, the available financial information does not provide an accurate depiction of total Program expenditures.

Table 4: Total expenditures ($), HDP, 2009-2010 to 2011-2012
Directorate	2009-2010			2010-2011			2011-2012
Directorate	w/o revenue	w/ revenue	Revenue	w/o revenue	w/ revenue	Revenue	w/o revenue	w/ revenue	Revenue
Therapeutic Products	44,791,057	22,990,326	-21,800,731	42,944,109	21,697,816	-21,247,248	49,412,876	29,126,690	-20,225,641
Inspectorate	3,609,648	-336,913	-4,027,954	4,225,428	1,099,902	-3,125,526	7,861,380	3,096,316	-4,765,064
Marketed Health Products	13,730,333	13,730,333	0	13,623,268	13,617,856	0	15,634,943	13,288,412	-2,351,741
Veterinary Drugs	0	0	0	0	0	0	1,979,897	1,341,185	-638,687
Other HPFB Costs	5,990,811	5,990,811	0	4,584,730	4,584,730	0	7,825,883	4,786,885	-3,038,998
HPFB - Regions	2,146,947	2,146,947	0	815,712	815,712	0	0	0	0
Total HDP - HPFB	70,268,796	44,521,504	-25,828,685	66,193,247	41,816,016	-24,372,774	82,714,979	51,639,488	-31,020,131
RAPB	12,135,809			12,285,455			17,098,529	9,412,834	-7,685,695
Total HDP - HPFB and RAPB	82,404,605			78,478,702			99,813,508	61,052,322	-38,705,826
Notes: Figures include Employee Benefit Plans (EPB). Within HPFB, overhead was not coded to sub-program levels prior to 2011.2012. Therefore, sub-program expenditures do not include overhead in 2009.2010 and 2010.2012. In 2011.2012, corporate overhead costs (such as Legal, lease and fit up, etc.) were not yet coded to sub-program levels. HPFB Regions includes British Columbia, Alberta, Nunavut, Northwest Territories, Ontario, Quebec, Atlantic and Manitoba. Other HPFB Costs consists of ADM, Litigation, Office of Consumer and Public Involvement, and PPIAD. As of 2011.2012, Veterinary Drugs expenditures are shared between two programs (Pharmaceutical Drugs Program and Food Safety Program). RAPB figures do not include Non-controllable or Capital. For RAPB, there is no revenue for 2009.2010 and 2010.2011. Cost recovery revenue was allocated to RAPB starting in 2011-2012. Sources: HPFB and RAPB.

Similarly, limited information is available on HDP budgets. For example, in 2010-2011 and 2011-2012, financial reporting at the Program sub-Activity level did not include budgeted amounts. Instead, budgeted amounts were only reported at the Program level (i.e., Health Products). This, combined with the difficulties in understanding total HDP expenditures, makes it impossible to compare budgeted amounts against actual expenditures over time.

Health Canada representatives indicated that HPFB has recently implemented a number of improvements to its approach to financial reporting.
Although expenditures incurred by overhead organizations such as ADM and PPIAD were not coded consistently to the HDP in the past, since 2011-2012, procedures have been put in place to ensure that all expenditures are coded to the appropriate Sub-activity (Pharmaceutical Drugs, Medical Devices, etc.) and not at the Program (i.e., Health Products) level.
Beginning in 2013-2014, notional budgets were revised to ensure proper allocation to the Sub-activity level.
Beginning in 2014-2015, TBS will require all government expenditure management reporting to be done at the Sub-activity level, or lower, depending on the approved PAA. Health Canada representatives indicated that HPFB is well along the path of being able to plan, budget, and report at this new required level.

In addition to the limitations, described above, of HPFB's historic approach to reporting on the budgets and expenditures of the HDP, a number of other shortcomings in the data also limit their usefulness for analyzing efficiency and economy. For example, while there is information on the number of FTEs allocated to various HPFB directorates and other entities involved in the HDP, the number of FTEs allocated to various program activities is unknown.^{Footnote 54}

A related challenge, from the point of view of assessing efficiency and economy, is that reporting by functional activities, which took place under the previous PAA, has not taken place since 2008-2009. Some examples of functional activities (also referred to as Functional Areas) include screening, product assessment, new submissions, monitoring and surveillance, and education and outreach. Such task- or activity-based reporting is essential to analyzing efficiency and economy because it reflects the time spent by program staff performing various tasks or activities and producing various outputs. As such, this information is important in assessing:

allocative efficiency, which focuses on the relationship between resources and outcomes;
operational efficiency, which focuses on the relationship between resources and outputs; and
economy, which focuses on the optimization (including the minimization) of the use of resources.

Within TPD, coding has been updated as of 2012-2013 so that functional activity information will be available in the future. In a recent status report to Treasury Board on the financial management and performance of the human drugs (including biologics) and medical devices programs under the new cost recovery regime, Health Canada reported that it has introduced financial management controls, including an improved time tracking system, to help validate costing data (Health Canada, 2013k). It is therefore possible that reporting by functional activities has been resumed. In addition, Health Canada reported that it has introduced a variety of other measures to support the efficient delivery of regulatory services, including the following:

streamlining review processes
putting systems in place - such as SOPs, guidelines, and training - to support more efficient reviews and applying them consistently to enable improved submission planning
using foreign review reports and working with other regulatory agencies
strengthening scientific capacity through hiring and retaining qualified staff
improving information technology systems to support the submission review process

In the same report, Health Canada noted that the increased revenue resulting from updated user fees has "allowed activities to be better resourced" and that "cost recovery targets for first decisions met 100% service standards in all program areas"; the same information was also reported in the 2011-2012 DPR.

As reported in Section 5.4.3, overall, submission review performance for human drugs has improved under the new cost recovery framework for all submission classes. However, except for ANDS, improvements in review performance were already underway for all submission classes prior to the implementation of the new framework, and may be attributable to the variety of initiatives that Health Canada has undertaken in recent years to improve the review process.

It is also important to note that improvements in timeliness do not necessarily mean that submission review is being carried out more efficiently, i.e., it is not necessarily the case that program outputs - in this case, completed reviews - are being produced at lower cost. A more detailed costing analysis, such as the analysis completed in 2007 to support the proposal for updated user fees, which calculated a unit cost for a variety of program activities subject to cost recovery (Health Canada & Spearhead Management Canada Limited, 2007), would shed light on the extent to which efficiencies may have been realized under the new cost recovery regime.

Health Canada is reviewing its costs, fees, and performance associated with the HDP as per its commitment in the User Fee Proposal and the Regulatory Impact Analysis Statement associated with the Fees in Respect of Drugs and Medical Devices Regulations. Preliminary results are expected in 2014.

8.0 Conclusions and recommendations

Relevance

Performance - program implementation

Clinical trials

Health Canada has strengthened the regulatory framework for clinical trials by implementing risk-based approaches to monitoring clinical trial adverse reaction reports, introducing an inspection program for clinical trial sites, and commissioning the development of new voluntary standards for Research Ethics Boards. Most recently, in May 2013, Health Canada launched a new public database of drug clinical trials it had authorized. Though mandatory for sponsors, the database contains more limited information than the registries that have been mandatory in the US since 1997 and the EU since 2004. Furthermore, unlike the US (and soon the EU), Health Canada does not require sponsors to disclose the results of clinical trials. Further enhancing the amount of clinical trial information it makes publicly available, including the results of clinical trials, would be consistent with Health Canada's commitment to enhancing transparency and openness as part of regulatory modernization, and would further align its approach with its main international counterparts.

Recommendation 1:

Submission review and market authorization

In addition to the new cost recovery framework, Health Canada also implemented a number of initiatives to improve the quality and efficiency of the submission review and market authorization process. These include implementing a project management approach to submission review; developing Good Guidance Practices and Good Review Practices for greater consistency in these processes; collaborating with the FDA under the RCC to develop the Common Electronic Submission Gateway, which will allow sponsors to make submissions simultaneously to Health Canada and the FDA; enhancing scientific review capacity; and taking steps to increase foreign review when reviewing Canadian applications.

Post-market surveillance

Recommendation 2:

Health Canada should improve its information systems for post-market surveillance and monitoring. This should include:

fully implementing information systems to support electronic submission of domestic and foreign adverse reaction reports, as well as systematic monitoring and analysis of these reports through targeted surveillance and data-mining.
developing and implementing a comprehensive and centralized approach to information management for post-market surveillance activities. In particular, this should include a centralized mechanism for tracking signal activities, including Health Canada's response to the recommended actions arising from completed signal assessments.

Recommendation 3:

Compliance and enforcement

In the area of compliance and enforcement, Health Canada has strengthened clinical trial inspections by developing a risk-based process for site selection; strengthened oversight over imported products through the National Border Integrity Program; renamed the former Post Marketing Compliance Reporting Program (PMRC) as the Good Pharmacovigilance Practices (GVP) inspection program to be in alignment with international requirements; extended GMP and DEL requirements to APIs; and adopted a risk-based approach to GMP inspections of domestic drug establishments.

Another issue may be Health Canada's oversight of foreign facilities that produce drugs for import into Canada. While Canada has MRAs with several countries recognizing the equivalence of their respective GMP compliance programs, a majority of the drug products and APIs imported into Canada come from countries with which Health Canada does not have an MRA. In the case of APIs, importation accounts for 85% of the Canadian market, with China and India being the major suppliers. Furthermore, although domestic establishments are inspected regularly according to risk-based criteria, Health Canada does not often conduct inspections of foreign facilities. The recent extension of GMP and DEL requirements to APIs is expected to reduce the possibility that low-quality or counterfeit health products will appear in the Canadian market.

To address the challenges associated with GMP inspections, Health Canada and the US FDA have launched an initiative under the RCC that aims to increase each country's reliance on GMP inspection reports prepared by the other country. Currently, the initiative applies to sites in Canada and the US, although it may be expanded to other jurisdictions in future. In addition, Health Canada is a member of the PIC/S, a body of 43 international participating authorities that have come together to harmonize GMP inspection requirements and approaches across the world. Health Canada reviews PIC/S inspection reports to approve foreign sites for Canada importers.

Recommendation 4:

Health Canada should explore options for improving its oversight of foreign manufacturing sites.

Recommendation 5:

Communications and stakeholder engagement

Health Canada has undertaken a number of initiatives to improve communications and stakeholder engagement. For example, since 2005, Health Canada has provided the public with information about review decisions through SBD documents. In 2012, in part to address concerns expressed by the OAG that it was not disclosing information related to NOC/cs, rejections, and withdrawals of new drugs, Health Canada introduced the post-authorization activity table (PAAT). PAATs provide ongoing information about approved products. They include a brief summary of activities that affect the safe and effective use of the product, such as information related to submissions for a new use of the product (whether Health Canada's decision was positive or negative), submissions filed in order to meet conditions (for products approved under the NOC/c Guidance), and regulatory decisions such as the cancellation of the DIN. Health Canada does not publish SBDs for negative decisions or otherwise provide information to the public about the reasons for negative decisions, unlike the FDA and the EMA.

To improve the quality and availability of easy-to-understand drug product labelling, Health Canada has enhanced the product monograph, developed a common monograph with the FDA for OTCdrugs under the RCC, and introduced regulatory amendments under the Plain Language Labelling Initiative. At present, Health Canada has limited authority to require manufacturers to modify product labelling once a product has received a Notice of Compliance. In contrast, the FDA has the authority to require product labelling to be updated as new safety information becomes available, including the authority to require a boxed warning containing new safety information. The EU's new pharmacovigilance legislation requires medicines subject to additional post-market monitoring to include an inverted black triangle on a drug product's leaflet.

Health Canada provides a variety of opportunities for stakeholder engagement, such as holding public consultations on proposed guidance, policies, and regulatory amendments and establishing advisory committees to guide regulatory and policy development. In addition, Health Canada consults with industry through pre-submission meetings and the BMP. While these specific opportunities are not available to health care practitioners and consumers/patients, Health Canada indicated that it does meet with pharmaceutical associations, hospital associations, and medical associations representing health care practitioners. Some external key informants expressed concern that the engagement and consultation process may favour industry over other stakeholders.

Performance - outcomes achieved

Immediate outcomes

In the short term, Health Canada's activities are also intended to produce increased safety and effectiveness of human drugs. There are pre-market and post-market processes in place that are designed to ensure that human drugs are safe and effective, but no concrete evidence of improvements in these areas. It could be argued that Health Canada's authority extends only to ensuring that products available on the Canadian market are safe and effective.

Finally, in the short term, Health Canada's activities are expected to lead to increased industry compliance with regulatory requirements. The available data suggest that serious non-compliance is relatively uncommon. Over the period of the evaluation, however, Health Canada has not reported regularly or consistently on the nature, seriousness, frequency, or prevalence of non-compliances related to human drugs, and has focused its reporting, instead, on quantifying activities and outputs. Furthermore, much of Health Canada's compliance data, including GMP, GCP, and GVP compliance data, is aggregated across multiple product categories, encompassing human drugs and biologic (and veterinary drugs, in the case of GMP).

Recommendation 6:

Intermediate outcomes

This finding is consistent with the 2011 report of the OAG, which noted that the Department's assessment of, and response to, potential safety signals is not timely. While most safety assessments examined by the OAG as part of its audit were completed within established timelines, the Department's approach to measuring its performance did not consider the amount of time a potential safety issue may wait before an assessment begins; the amount of time an assessment may be placed on hold; the amount of time needed to obtain additional information from external parties (such as the manufacturer); and the total number of calendar days, instead of working days, taken to complete the assessment. The OAG noted that when these factors were considered, Health Canada took at least one year to complete 34 of its 54 assessments and in some cases took significantly longer - two or three years.

Program representatives reported that developing the risk communication typically requires a considerable amount of negotiation with the market authorization holder, and posting of the communication may be delayed until appropriate changes have been made to the product labelling. As a result of these factors, the time elapsed from detection to posting of a risk communication can be quite lengthy. In 14 cases for which complete data were available (representing signal assessments completed between 2010 and 2012), the total time elapsed between signal detection and posting of the risk communication ranged from 232 days to 1,481 days (4 years), with a median of 1.4 years. As noted, Health Canada is currently reviewing existing performance standards for the content development of risk communications and the dissemination of risk communications.

Recommendation 7:

Health Canada should take steps to improve the timeliness of its response to safety signals (from signal detection to posting of a risk communication).

In the intermediate term, Health Canada expects to achieve increased international harmonization of regulatory requirements for human drugs. Among many other international engagements, Health Canada is an official observer to and participant in the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH); has developed MRAs on GMP compliance with a number of other jurisdictions; has formal information-sharing agreements with the FDA, Australia's TGA, and the European Community; and participates with the FDA in the RCC, the overall goal of which is to better align the two countries' regulatory approaches. As already noted, Health Canada is also a member of PIC/S. International key informants view Health Canada as a constructive participant in bilateral and multilateral engagement.

Long-term outcomes

Performance - efficiency and economy

While activity-based financial reporting has not taken place since 2008-2009, within TPD coding has been updated as of 2012-2013 so that functional activity information will be available in the future. Activity-based reporting is important to support activity-based costing, which in turn is important to analysing the efficiency with which program activities are carried out. HPFB undertook an activity-based costing exercise in 2007 to support the proposal for updated user fees, and used the results of this analysis to calculate unit costs for a variety of regulatory activities, including submission review. The available data indicate that the timeliness of submission review for most classes of human drugs has been improving under the new cost recovery framework. However, without analysing the unit costs of submission review and other cost-recovered activities, it is unclear if improvements in timeliness also represent improved efficiencies. In addition to enabling an assessment of the extent to which efficiencies may have been realized under the new cost recovery framework, such analysis would also assist the HDP in identifying where future adjustments to the framework may be necessary. To this end, HPFB is reviewing its costs, fees, and performance associated with the HDP, as per its commitment in the User Fee Proposal and the Regulatory Impact Analysis Statementassociated with the Fees in Respect of Drugs and Medical Devices Regulations.

Appendix A - Evaluation Matrix

Table 1: Evaluation Matrix
Summative Evaluation of the Human Drugs Program (Hdp) - Evaluation Matrix
Evaluation issues and questions	Indicators	Potential data sources
Section 1: Relevance
Issue #1: Continued need for the program
1. Is there a continued need for the HDP?	Need for program identified/documented	Document review: Official government documents
	Evidence of current/emerging human health and safety issues related to human drugs	Document review: HPFB Blueprint for Renewal, Regulatory Roadmap for Health Products and Food, and other related documents TPD strategic planning documents Health Canada environmental scan Literature review
	Expert/stakeholder assessment of ongoing need	Key informant interviews (internal and external)
	Responsiveness of HDP to needs of Canadians	Document review Literature review Key informant interviews (external) Surveys of industry, patients/consumers, and physicians
Issue #2: Alignment with government priorities
2. Is the HDP aligned with the priorities of the Government of Canada?	Extent to which the objectives of HDP are linked to Federal Government priorities	Document review: recent Speeches from the Throne/Budgets Therapeutic Access Strategy (TAS), Therapeutic Product Safety Initiative (TPSI), Canada's Access to Medicines Regime (CAMR), Food and Consumer Safety Action Plan (FCSAP), Cost Recovery Initiative (CRI)
	Extent to which the objectives of HDP are linked to the strategic outcomes/priorities of Health Canada/HPFB	Document review: recent Health Canada Reports on Plans and Priorities HPFB Blueprint for Renewal reports, Regulatory Roadmap for Health Products and Food, and other related documents HPFB, TPD, MHPD, Inspectorate strategic planning documents
Issue #3: Alignment with federal roles and responsibilities
3. Is the HDP consistent with federal roles and responsibilities?	Extent to which the objectives of the HDP are consistent with the legislative framework of the Federal Government	Document review: federal Acts and Regulations (Department of Health Act, Food and Drugs Act, etc., and relevant Regulations)
	Extent to which the objectives of the HDP are consistent with the legislative framework of Health Canada	Document review: federal Acts and Regulations (Department of Health Act, Food and Drugs Act, etc., and relevant Regulations) recent Health Canada Reports on Plans and Priorities
Section 2: Performance (Effectiveness, Efficiency, Economy)
Issue #4: Achievement of expected outcomes
4. Is the governance structure for the HDP likely to support the achievement of expected outcomes?
a) Is there an established governance structure to coordinate delivery of theHDP?	Extent to which internal and interdepartmental partners' roles, responsibilities, accountabilities, and decision making authorities are documented and understood	Document review: descriptions of the organizational structures, mandates, and activities of HDP partners, as available from: Health Canada website health product regulation roles and responsibilities framework documents (2005; 2008) other internal documents (as available) Key informant interviews (internal and external, i.e., other federal departments)
	Extent of collaboration among internal and interdepartmental partners, as evidenced by: existence of committees, working groups, and teams frequency of meetings of committees, working groups, and teams	Document review: TPD, MHPD, Inspectorate, and HPFB operational, strategic, and business plans and performance reports committee/working group Terms of Reference (as available) meeting agendas/minutes (as available) Key informant interviews (internal and external, i.e., other federal departments)
	Nature of industry and stakeholder involvement in HDP governance	Document review: minutes of meetings/reports of consultations with industry stakeholders Key informant interviews (internal and external)
b) Has a performance measurement framework been designed and implemented?	Existence of performance measurement framework(s)	Document review: FCSAP and TPSI RMAFs; CRI PMEP HDA Summative Evaluation Framework
	Extent to which performance data are collected	Document review: HPFB, TPD, MHPD, and Inspectorate performance reports (e.g., quarterly reports, annual reports, business transformation reports) FCSAP reports Health Canada DPRs Completed evaluations, e.g., Formative Evaluations of the TAS and the CAMR Key informant interviews (internal)
c) Is the performance measurement framework used to support decision making?	Extent to which performance data are used to support decision making	Document review: HPFB, TPD, MHPD, and Inspectorate operational, strategic, and business plans MRAPs for relevant evaluations (e.g., TAS) Key informant interviews (internal)
5. To what extent has the HDP been implemented as planned?
a) Has the Program effectively addressed challenges, emerging issues, and changing priorities?	Extent to which challenges, emerging issues, and changing priorities have been effectively addressed, for example: classification of therapeutic products combination products ability to keep pace with scientific/technological advances (e.g., nanotechnology) pharmacovigilance generic versus brand drugs timely access to new drugs drug shortages	Document review (all documents) Key informant interviews (internal and external)
b) Have activities been implemented as planned? c) Have the activities produced the expected outputs?	Extent to which HDA have been implemented as planned, including planned activities for CAMR, TAS, TPSI, CRI, FCSAP Enumeration of outputs (policies, guidelines, regulations, research, MOUs, etc.) produced, including outputs identified for CAMR, TAS, TPSI, CRI, FCSAP	Document review - For plans: Official government documents HPFB, TPD, MHPD, and Inspectorate operational and strategic plans RMAFS/PMEPs for FCSAP and TPSI; PMEP for CRI; TAS evaluation report; CAMR planning documents, and other planning documents as appropriate planned spending data	Document review - For actual: HPFB, TPD, MHPD, and Inspectorate performance reports (e.g., quarterly reports, annual reports, business transformation reports) FCSAP reports Health Canada DPRs TAS evaluation report actual spending data policies, guidelines, regulations, research, MOUs, etc.
d) Have requirements/ commitments to Central Agencies (i.e., Office of the Auditor General, Cabinet Directive on Streamlining Regulations, Policy on Public Consultation, Policy on Gender- Based Analysis) been addressed?	Extent to which requirements and commitments to Central Agencies have been addressed	Document review (all documents) Key informant interviews (internal and external)
6) To what extent has progress towards expected outcomes been achieved?
Immediate outcomes
a. To what extent is there increased awareness and understanding among external stakeholders of risks and benefits related to human drugs?	Number and nature of Health Canada communications to external stakeholders, by type and target group, including: Summary Basis of Decisions Product monographs Advisories, warnings, and recalls (for public and for health professionals) Canadian Adverse Reaction Newsletter Drug Recall Listings Canadian Adverse Reaction Database Products aimed at special or at-risk populations MedEffect subscription service Other information/communications	Document/administrative data review, including publicly accessible data on Health Canada's website
	Existence of stakeholder lists and use of technology to deliver information to stakeholders	Document review:
	Use of partnerships to develop/deliver information to stakeholders	Document review:
	Extent and nature of Health Canada consultations with external stakeholders regarding risks and benefits of human drugs	Document review Key informant interviews (internal and external)
	Proportion of post market isues where a public communication piece is released	Document/administrative data review
	Extent of stakeholder awareness and use of the information made available by Health Canada, by product type	Survey of health care providers Survey of patients/consumers Key informant interviews (external)
	Stakeholder assessment of quality of Health Canada information/communications, in terms of: timeliness accessibility ease of understanding usefulness	Survey of health care providers Survey of patients/consumers Key informant interviews (external)
	Extent to which external stakeholders rely on Health Canada for risk/safety information versus other sources	Survey of health care providers Survey of patients/consumers Key informant interviews (external)
	Stakeholder perceptions of their level of awareness and understanding of risks related to human drugs	Survey of health care providers Survey of patients/consumers Key informant interviews (external)
b. To what extent is there increased awareness and understanding among industry of Health Canada's regulatory framework for human drugs?	Number and nature of Health Canada communications, meetings, and consultations with industry stakeholders regarding the regulatory framework for human drugs, including: Pre-submission meetings Guidance documents Bilateral Meeting Program Consultations on proposed regulatory/policy changes and guidance for industry Other communications and consultations	Document/administrative data review
	Use of technology to deliver information to industry	Document review:
	Use of partnerships/collaboration to develop/deliver information to industry	Document review:
	Extent of industry awareness and use of the information made available by Health Canada, by product type	Survey of industry Key informant interviews (external)
	Industry assessment of quality of Health Canada information and communications, in terms of: timeliness accessibility ease of understanding usefulness	Survey of industry Key informant interviews (external)
	Extent to which industry relies on Health Canada for regulatory information versus other sources	Survey of industry Key informant interviews (external)
	Industry perceptions of their level of awareness and understanding of Health Canada's regulatory framework for human drugs	Survey of industry Key informant interviews (external)
c. To what extent is there increased safety and effectiveness of human drugs?	Description of current approach to human drug licensing, including extent to which process is risk-based	Document review Case study (clinical trials)
	Description of progressive licensing project, with emphasis on how progressive licensing is expected to improve safety and/or effectiveness	Document review Key informant interviews (internal)
	Extent and nature of pre-market activities, approaches, tools, and initiatives intended to increase safety and effectiveness, for example: Performance targets/standards Pre-submission meetings Changes to requirements for clinical trials Use of foreign reviews and post market data	Document/administrative data review Case study (clinical trials)
	Extent and nature of post market activities, approaches, tools, and initiatives intended to increase safety and effectiveness, for example: Performance targets/standards Pharmacovigilance plans/Risk management and mitigation plans Signal assessment and monitoring Adverse reaction reporting and analysis	Document/administrative data review Case study (post-market)
	(Pre-market) Proportion of pre-market reviews for human drugs conducted within service standards	Document/administrative data review, especially: TAS evaluation report TPD Annual Drug Submission Performance Reports
	(Pre-market) Trends in first decisions for human drug applications by outcome (Notice of Compliance, Notice of Deficiency, Notice of Non-Compliance, etc.)	Document/administrative data review: TPD Annual Drug Submission Performance Reports
	(Post market) Proportion of monitoring and surveillance activities addressed within service standards/targets	Document/administrative data review
	(Post market) Trends in number of post market monitoring and surveillance activities	Document/administrative data review Case study (post-market)
	(Post market) Trends in number and type of follow-up actions as a result of each type of monitoring and surveillance activity	Document/administrative data review Case study (post-market)
	Extent to which approval times are linked to safety (e.g., linkage between approval times and post market safety issues such as adverse reaction reports, withdrawals of drug from market)	Administrative data review Literature review
	Stakeholder satisfaction with timeliness of drug approval process and timeliness of access to approved drugs	Surveys of industry, health care providers, consumers/patients Key informant interviews (external)
	Stakeholder perceptions of safety and effectiveness of human drugs, including perceptions of adequacy of pre-market and post market processes in place to ensure safety and effectiveness	Literature review Document review Key informant interviews (external) Surveys of industry, health care providers and patients/consumers
d. To what extent is there increased industry compliance with Health Canada's regulatory requirements related to human drugs?	Description of new compliance and enforcement tools, approaches, and initiatives	Document review:
	Adequacy of information technology tools to support compliance tracking	Document review Key informant interviews (internal)
	Adequacy of training related to human drugs delivered to Inspectorate and RAPB staff	Document review Key informant interviews (internal)
	(Post market compliance) Number of regulated parties and proportion of regulated parties inspected and/or monitored (manufacturers, clinical trials)	Document/administrative data review: Inspectorate reporting
	(Post market compliance) Trends in number of inspections, compliance verifications, and other compliance/enforcement activities related to: Good Manufacturing Practices Establishment Licenses Clinical Trials/Good Clinical Practices Good Pharmacovigilance Practices Adverse drug reaction reporting Border integrity Controlled Drugs and Substances Post Market Reporting Compliance Etc.	Document/administrative data review: Inspectorate reporting
	(Post market compliance) Trends in number and type of non-compliances found through each type of inspections and compliance verifications	Document/administrative data review: Inspectorate reporting
	(Post market compliance) Trends in number and outcomes of compliance and enforcement actions, including: Education/information Voluntary actions by industry Health Canada initiated/ordered recalls Product removals/seizures Monetary penalties assessed Import alerts and interventions at the border Etc.	Document /administrative data review: Inspectorate reporting
	Proportion of regulatory/enforcement actions addressed within service standards/targets	Document /administrative data review: Inspectorate reporting
Intermediate outcomes
e. To what extent do external stakeholders adopt safe behaviours associated with human drugs?	Extent to which external stakeholders report using Health Canada risk communications for decision making	Key informant interviews (external, especially health care practitioners/facilities, patient/consumer groups) Surveys of health care providers and patient/consumer groups
	Number of stakeholders who report changes in behaviours due to Health Canada's risk communications, for example: physicians/health care providers who report changing their prescription behaviour	Survey of health care providers Key informant interviews (external) Literature review
	Trend data on improper or unsafe use of human drugs, including number and characteristics (type, severity, age, gender, etc.) of reported incidents	Literature review Document review (if data available) Key informant interviews (external, especially health care practitioners/facilities and patient/consumer groups Surveys of physicians and patient/consumer groups
f. To what extent is there increased use of scientific evidence and risk-benefit analysis by Health Canada to inform decision making?	Description of Health Canada's approach to decision making, including extent to which approach is risk-based	Document review Key informant interviews (internal)
	Extent and nature of Health Canada's post market activities to increase use of scientific evidence and risk-based analysis to inform decision making, for example: Pharmacovigilance Plans Risk Management and Mitigation Plans Periodic Safety Update Reports Canada Vigilance System Laboratory activities Mandatory Adverse Reaction Reporting by institutions Drug Safety and Effectiveness Network research activities	Document/administrative data review Case study (post-market)
	Number of safety signals generated through post market signal detection activities, including: Environmental scanning Evaluation of foreign post market data PSUR reviews Monitoring of adverse reaction reports DSEN research activities	Document/administrative data review Key informant interviews (internal) Case study (post-market)
	Extent to which information gathered through post market signal detection activities is used to inform decision making: number and nature of responses taken in response to risks identified through post market signal detection activities	Document/administrative data review Case study (post-market)
	Extent to which recommendations of expert/scientific advisory groups are used to inform and develop policy/regulatory responses	Document review: Recommendations of expert/scientific advisory groups compared against policies, guidelines, regulations developed
	Extent to which scientific research is used in regulatory framework development (regulations, guidances, Standard Operating Procedures)	Document review Key informant interviews (internal/external)
	Stakeholders' perceptions of extent to which use of scientific evidence and risk-based analysis to inform decision making has increased	Key informant interviews/consultations (external and internal) Surveys of industry, health care providers, and patients/consumers
g. To what extent is there a timely regulatory system response to identified risks?	(Overall regulatory/policy response) Description of regulatory process	Document review:
	(Overall regulatory/policy response) Elapsed time between initial identification of human drug-related risks and policy/regulatory response by Health Canada	Document review (if information is available) Case study (post-market)
	(Response to risks identified through post market activities) Proportion of post market actions undertaken within service standards	Document/administrative data review: Inspectorate/MHPD reporting Case study (post-market)
	Internal and external stakeholder perceptions of timeliness of Health Canada's response to identified risks associated with human drugs	Key informant interviews/consultations (external and internal) Survey of industry, health care providers, and patients/consumers
h. To what extent is Canada's regulatory framework for human drugs harmonized with international approaches?	Extent to which main features of Canada's regulatory framework for human drugs is harmonized with that of other jurisdictions	Literature review: comparison of main features of Canada's regulatory framework with that of selected other jurisdictions (EU, US, Australia, UK) Key informant interviews (internal and external) Case studies (post-market, clinical trials)
	Description of program's decision making process regarding harmonization (especially factors considered in decision whether to harmonize)	Key informant interviews (internal)
	Extent to which Health Canada participates in international bodies, initiatives, boards, etc.	Document review Key informant interviews (internal)
	Number of international standards, policies, and guidelines adopted by Canada	Document review:
	Extent of Health Canada contribution to development of international standards, policies, and guidelines	Document review Key informant interviews (internal)
	Extent to which Health Canada is recognized as a responsible human drug regulator and scientific expert (nationally and internationally), as evidenced by: Requests for information sharing by provinces/territories, other countries, and international organizations Level of acceptance of Canadian laboratory expertise Extent to which Health Canada experts participate on multilateral/international expert bodies	Document review Literature review Key informant interviews
i. To what extent is there reduced exposure to identified risks associated with the use of human drugs?	Trends in post market enforcement actions due to identified risks for authorized and unauthorized human drugs	Document/administrative data review: Inspectorate data
	Trends in ratio of number of serious adverse reaction reports to total number of adverse reaction reports	Document/administrative data review MHPD data
	Number of approved human drugs removed from the marketplace due to safety concerns	Document/administrative data review Inspectorate data
	Number of unlicensed and counterfeit drugs removed from the marketplace	Document/administrative data review Inspectorate rate
	Expert assessment of changes in exposure to health risks related to human drugs	Key informant interviews (internal and external) Literature review
Long term outcomes
j. To what extent have adverse events associated with the use of human drugs been reduced?	Trends in number and severity of adverse drug reaction reports over time	Document/administrative data review Key informant interviews (internal)
	Rate of morbidity/mortality related to human drugs	Document/administrative data review (if available)
k. To what extent is there increased public confidence in human drugs and the related regulatory system?	Level of public confidence in safety of human drugs and the related regulatory system	Document review: Health Canada public opinion research (if available) Health Canada DPRs Key informant interviews (external)
	Number of requests for implementation of a similar safety system internationally	Document review:
l. To what extent is there a sustainable, cost-efficient, responsive, and science-based regulatory system for human drugs in Canada?	Cumulative evidence from all outcome indicators	All data sources
7. Were there any unintended consequences, either positive or negative, of the HDP?	Unintended consequences identified by internal and external stakeholders	Key informant interviews/consultations(internal and external) Survey of industry Survey of stakeholders
Issue #5: Efficiency and Economy
8. Were HDP resources used as planned?What accounted for overruns or lower- than-planned expenditures?	Comparison of planned versus actual spending for components of TPD and explanations for variances	Administrative data review, for example: planned versus actual spending, SAP data, financial derivation reports, management variance reports (if available) Key informant interviews (internal)
9. Are there lower-cost approaches to producing outputs of HDP?	Extent to which existing resources could be used to produce outputs at lower cost Availability/accessibility of other, lower cost resources to produce outputs	Key informant interviews (internal) Document review
10. Are there alternate ways to achieve similar results at lower cost?	Approaches used in other jurisdictions and their costs Internal and external stakeholder assessment of other options	Literature review Key informant interviews/consultations (internal and external)

Appendix B - List of References

Abraham, C. (2005). Vioxx took deadly toll: study. The Globe and Mail. Retrieved from http://www.theglobeandmail.com/life/vioxx-took-deadly-toll-study/article1113848/

Canadian Paediatric Society. (n.d.). Canadian Paediatric Surveillance Program. Retrieved October 4, 2012, from http://www.web.cps.ca/English/surveillance/cpsp/index.htm

Cassels, A. (2012). Most of our prescription drugs are manufactured overseas - but are they safe? Canadian Medical Association Journal, 184(14), 1648.

CBC News. (2012). Sandoz drug shortage prompts Ottawa fast-track efforts. Retrieved September 10, 2013, from http://www.cbc.ca/news/health/story/2012/03/06/sandoz-drug-supply-concerns.html

CCSA. (2012). Prescription Drug Misuse. Retrieved October 29, 2012, from http://www.ccsa.ca/Eng/Priorities/Prescription-Drug-Misuse/Pages/default.aspx

CDC. (2012). Multistate Fungal Meningitis Outbreak Investigation. Website of the Centers for Disease Control and Prevention. Retrieved October 25, 2012, from http://www.cdc.gov/HAI/outbreaks/meningitis.html

CIHI. (2011). National health expenditure trends, 1975 to 2011. Retrieved from https://secure.cihi.ca/estore/productFamily.htm?locale=en&pf=PFC1671

Collier, R. (2013). Propecia lawsuits: the lasting effects of delayed drug warnings. Canadian Medical Association Journal, 185(10), E455-E456. doi:10.1503/cmaj.109-4488

Council Directive. (2001). Directive 2001/83/EC of the European Parliament and of the Council of 6 November 2001 on the Community Code Relating to Medicinal Products for Human Use. Official Journal of the European Union, 311. Retrieved from http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2001:311:0067:0128:en:PDF

Decima Research. (2003). Public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada.

Decima Research. (2006). Final report: 2006 General public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada.

Department of Finance. (2003). The Budget Plan 2003. Retrieved August 21, 2012, from http://fin.gc.ca/budget03/pdf/bp2003e.pdf

Dhalla, I., Mamdani, M. M., Sivilotti, M. L. A., Kopp, A., Qureshi, O., & Juurlink, D. N. (2009). Prescribing of opioid analgesics and related mortality before and after the introduction of long-acting oxycodone. Canadian Medical Association Journal, 181(12), 891-896. doi:10.1503/cmaj.090784

Duffin, J. (2012). Home: What is the drug shortage? Canadian Drug Shortage - A site for information about the situation in Canada. Retrieved October 25, 2012, from http://www.canadadrugshortage.com/

Eguale, T., Buckeridge, D. L., Winslade, N. E., Benedetti, A., Hanley, J. A., & Tamblyn, R. (2012). Drug, Patient, and Physician Characteristics Associated With Off-label Prescribing in Primary Care. Archives of Internal Medicine.

EMEA. (2009). Guideline on the Readability of the Labelling and Package Leaflet of Medicinal Products for Human Use. Retrieved July 8, 2013, from http://ec.europa.eu/health/files/eudralex/vol-2/c/2009_01_12_readability_guideline_final_en.pdf

EMEA. (2012). EU Clinical Trials Register - About Page | More information on EU-CTR search and website. Retrieved February 26, 2013, from https://www.clinicaltrialsregister.eu/about.html#whatsnew

EMEA. (2013a). European Medicines Agency - Pharmacovigilance - Risk-management plans. Retrieved July 4, 2013, from http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/document_listing/
document_listing_000360.jsp&mid=WC0b01ac058067a113

EMEA. (2013b). European Medicines Agency - Post-authorisation - Human post-authorisation Q&A: Introduction. Retrieved July 5, 2013, from http://www.emea.europa.eu/ema/index.jsp?curl=pages/regulation/general/
general_content_000166.jsp&mid=WC0b01ac0580023399

EMEA. (2013c). European Medicines Agency updates product-information template to label medicines subject to additional monitoring and encourage adverse-reaction reporting. Retrieved July 9, 2013, from http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2013/03/
news_detail_001740.jsp&mid=WC0b01ac058004d5c1

EMEA. (2013d). Guideline on good pharmacovigilance practices (GVP). Retrieved July 4, 2013, from http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/
2012/06/WC500129136.pdf

EMEA. (2013e). Human medicines - European public assessment reports. Retrieved September 9, 2013, from http://www.ema.europa.eu/ema/index.jsp?searchType=name&startLetter=
O&taxonomyPath=&keyword=opioid&alreadyLoaded=true&curl=pages%2Fmedicines%2
Flanding%2Fepar_search.jsp&status=Authorised&status=Withdrawn&status=Suspended&
status=Refused&mid=WC0b01ac058001d125&searchGenericType=generics&treeNumber=
&searchTab=&pageNo=2

Environics Research Group. (2007). Adverse Reaction Reporting - Survey with Health Professionals. Retrieved October 11, 2012, from http://epe.lac-bac.gc.ca/100/200/301/pwgsc-tpsgc/por-ef/health/2007/432-06/summary.pdf
European Parliament. (2010a). Directive 2010/84/EU of the European Parliament and of the Council of 15 December 2010. Official Journal of the European Union, 348. Retrieved from http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2010:348:0074:0099:EN:PDF

European Parliament. (2010b). Regulation (EU) No 1235/2010 of the European Parliament and of the Council of 15 December 2010. Official Journal of the European Union, 348. Retrieved from http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2010:348:0001:0016:EN:PDF

FDA. (2002). Compliance Program Guidance Manual Program: Drug Manufacturing Inspections. Retrieved July 5, 2013, from http://www.fda.gov/downloads/ICECI/ComplianceManuals/
ComplianceProgramManual/UCM125404.pdf

FDA. (2008). Sentinel Initiative - Transforming How We Monitor Product Safety > The Sentinel Initiative: A National Strategy for Monitoring Medical Product Safety. Retrieved July 5, 2013, from http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm089474.htm

FDA. (2009). Information for Consumers (Drugs) > OTC Drug Facts Label. Retrieved July 8, 2013, from http://www.fda.gov/Drugs/ResourcesForYou/Consumers/ucm143551.htm

FDA. (2010). Boxed warnings and other FDA communication tools. American Family Physician, 81(3). Retrieved from http://www.fda.gov/downloads/Safety/MedWatch/UCM201430.pdf

FDA. (2011a). Guidance for Industry Postmarketing Studies and Clinical Trials - Implementation of Section 505(o)(3) of the Federal Food, Drug, and Cosmetic Act. Retrieved July 5, 2013, from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatory
Information/Guidances/UCM172001.pdf

FDA. (2011b). Warning Letter. Novartis International AG 11/18/11. Retrieved September 10, 2013, from http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/ucm281843.htm

FDA. (2012a). FDA Adverse Event Reporting System (FAERS) (formerly AERS). Retrieved May 2, 2013, from http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/
AdverseDrugEffects/default.htm

FDA. (2012b). FDA Adverse Events Reporting System (FAERS) Electronic Submissions. Retrieved May 2, 2013, from http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/
AdverseDrugEffects/ucm115894.htm

FDA. (2012c). Risk Evaluation and Mitigation Strategy (REMS) for Extended-Release and Long-Acting Opioids. Retrieved November 16, 2012, from http://www.fda.gov/Drugs/DrugSafety/InformationbyDrugClass/ucm163647.htm

FDA. (2013a). Drugs@FDA: FDA Approved Drug Products. Retrieved September 9, 2013, from http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm

FDA. (2013b). FDA News Release. FDA approves abuse-deterrent labeling for reformulated OxyContin. Retrieved September 9, 2013, from http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm348252.htm

FDA. (2013c). Guidance for Industry Providing Postmarket Periodic Safety Reports in the ICH E2C(R2) Format (Periodic Benefit-Risk Evaluation Report). Retrieved July 4, 2013, from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/
Guidances/UCM346564.pdf

FDA. (2013d). Inspections, Compliance, Enforcement, and Criminal Investigations. Retrieved September 10, 2013, from http://www.fda.gov/iceci/enforcementactions/ucm222557.htm

FDA. (2013e). Laws, Acts, and Rules > PLR Requirements for Prescribing Information. Retrieved July 8, 2013, from http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/LawsActsandRules/
ucm084159.htm

FDA. (2013f). Warning Letter. Apotex Inc. February 21, 2013. Retrieved October 7, 2013, from http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2013/ucm344476.htm

Ferguson, B., & Lybecker, K. (2012). Economics of intellectual property protection in the pharmaceutical sector: Pills, patients, & profits II. Retrieved from http://www.macdonaldlaurier.ca/files/pdf/Economics-of-Intellectual-Property-Protection-in-the-Pharmaceutical-Sector-January-2012.pdf

Globe and Mail. (2012a). Sandoz Canada's production slows to a crawl after harsh criticism from U.S. regulators. Retrieved September 10, 2013, from http://m.theglobeandmail.com/globe-investor/sandoz-canadas-production-slows-to-a-crawl-after-harsh-criticism-from-us-regulators/article2343341/?service=mobile

Globe and Mail. (2012b, November 29). Tories block bid to make cheaper medicines for poor nations.

Globe and Mail. (2013a). Health Canada announces third birth-control recall this year. Retrieved September 10, 2013, from http://www.theglobeandmail.com/life/health-and-fitness/health/health-canada-announces-another-birth-control-recall-of-esme-28/article14120836/

Globe and Mail. (2013b, April 26). U.S. regulator warns Canadian drug maker Apotex about quality control. Retrieved October 7, 2013, from http://www.theglobeandmail.com/life/health-and-fitness/health/us-regulator-warns-canadian-drug-maker-apotex-about-quality-control/article11560463/

GoC. (2004). User Fees Act (S.C. 2004, c. 6).

GoC. (2006). Department of Health Act (S.C. 1996, c. 8).

GoC. (2008a). Bill C-51: An act to amend the Food and Drugs Act and to make consequential amendments to other Acts.

GoC. (2008b). Food and Drugs Act (R.S.C., 1985, c. F-27).

GoC. (2010). Regulations Amending the Food and Drug Regulations (743 - Non-Medicinal Ingredients) (Part II). Canada Gazette Part II, 144(11). Retrieved from http://gazette.gc.ca/rp-pr/p2/2010/2010-05-26/html/sor-dors105-eng.html

GoC. (2011). Fees in Respect of Drugs and Medical Devices Regulations. Canada Gazette Part II, 145(8). Retrieved from http://www.gazette.gc.ca/rp-pr/p2/2011/2011-04-13/html/sor-dors79-eng.html

GoC. (2012a). Cabinet Directive on Regulatory Management. Retrieved September 9, 2013, from http://www.tbs-sct.gc.ca/rtrap-parfa/cdrm-dcgr/cdrm-dcgrtb-eng.asp

GoC. (2012b). Communications Policy of the Government of Canada. Retrieved September 9, 2013, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12316&section=text

GoC. (2012c). Food and Drug Regulations (C.R.C., c. 870).

GoC. (2013). Canada Gazette - Regulations Amending the Food and Drug Regulations (Labelling, Packaging and Brand Names of Drugs for Human Use). Canada Gazette Part I, 147(25). Retrieved from http://www.gazette.gc.ca/rp-pr/p1/2013/2013-06-22/html/reg2-eng.html#reg

GoC. (n.d.). Common Electronic Submissions Gateway (ESG) Work Plan. Retrieved June 3, 2013, from http://actionplan.gc.ca/en/page/rcc-ccr/common-electronic-submissions-gateway-esg-work-plan

Grosh, L., Sharpe, J., Bergman, N., Bigge, D., Blanck, J., Cate, A., & Thornton, N. (2013). In the Arbitration Under Chapter Eleven of the North American Free Trade Agreement and the ICSID Arbitration (Additional Facility) Rules Between Apotex Holdings Inc. and Apotex Inc. (Claimants/Investors) and the United States of America (Respondent/Party) Case No. ARB(AF)/12/1. Retrieved September 16, 2013, from http://italaw.com/sites/default/files/case-documents/italaw1242.pdf

Health Canada. (2000). Health Canada decision-making framework for identifying, assessing, and managing health risks.

Health Canada. (2003). Notice: Common Technical Document - ICH Topic M4. Retrieved October 18, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/ctd_m4_notice-eng.pdf

Health Canada. (2006). Results-based Management and Accountability Framework - Therapeutic Products Safety Initiative submission.

Health Canada. (2008a). FCSAP Results-based Management and Accountability Framework (RMAF).

Health Canada. (2008b). Guidance document: Description of current risk communication documents for marketed health products for human use. Retrieved from http://www.hc-sc.gc.ca/dhp-mps/pubs/medeff/_guide/2008-risk-risques_comm_guid-dir/index-eng.php

Health Canada. (2008c). Introduction - Canada's Access to Medicines Regime. landing page. Retrieved August 28, 2012, from http://www.camr-rcam.gc.ca/intro/index-eng.php

Health Canada. (2009a). Guidance on Evidence to Demonstrate Drug GMP Compliance of Foreign Sites (GUI-0080). Retrieved July 24, 2013, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/gmp-bpf/docs/gui-0080-eng.php

Health Canada. (2009b). Notice: Implementation of risk management planning including the adoption of ICH guidance pharmacovigilance planning - ICH topic E2E.

Health Canada. (2009c). The Therapeutic Access Strategy (TAS) Formative Evaluation (Final Report).

Health Canada. (2010a). Draft Guidance Document: Labelling of Pharmaceutical Drugs for Human Use - 2010 Health Canada Consultation Document. Retrieved July 9, 2013, from http://www.hc-sc.gc.ca/dhp-mps/consultation/drug-medic/draft_ebauche_label_guide-eng.php#a31

Health Canada. (2010b). Guidance Document for Industry: Issuance of Health Professional Communications and Public Communications by Market Authorization Holders. Retrieved November 27, 2013, from http://www.hc-sc.gc.ca/dhp-mps/pubs/medeff/_guide/2010-guid-dir_indust_hppc-cpsp/index-eng.php

Health Canada. (2010c). Health Canada's Proposal to Parliament for User Fees and Service Standards for Human Drugs and Medical Devices Programs. Retrieved from http://www.hc-sc.gc.ca/dhp-mps/finance/costs-couts/fee-propo-frais-eng.php

Health Canada. (2011a). Canada's Health Care System. Website of Health Canada. Retrieved October 5, 2012, from http://www.hc-sc.gc.ca/hcs-sss/pubs/system-regime/2011-hcs-sss/index-eng.php

Health Canada. (2011b). Director General's Office - Health Products and Food Branch Inspectorate - Health Canada. organizational description. Retrieved August 20, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/hpfb-dgpsa/hpfi-ipsa/dgo-bdg-eng.php

Health Canada. (2011c). Draft Guidance Document - Post-Notice of Compliance (NOC) Changes: Framework Document. Retrieved July 3, 2013, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/applic-demande/guide-ld/postnoc_change_apresac/noc_pn_framework_ac_sa_cadre-eng.php

Health Canada. (2011d). Health Canada's Management Response and Action Plan (MRAP) - Audit of Chapter 4, Regulating Pharmaceutical Drugs.

Health Canada. (2012a). 2011-12 Departmental Performance Report. Retrieved September 9, 2013, from http://www.hc-sc.gc.ca/ahc-asc/performance/estim-previs/dpr-rmr/2011-2012/report-rapport-eng.php#a221

Health Canada. (2012b). 2011-12 Departmental Performance Report (DPR). Retrieved July 3, 2013, from http://www.hc-sc.gc.ca/ahc-asc/performance/estim-previs/dpr-rmr/2011-2012/report-rapport-eng.php

Health Canada. (2012c). Cost-Benefit Analysis: Regulations Amending the Food and Drug Regulations - Good Manufacturing Practices for Active Ingredients. Retrieved October 30, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/info-prod/drugs-drogues/actingred-cba-aca-eng.php

Health Canada. (2012d). Glossary of Fields in the Canada Vigilance Adverse Reaction Online Database. terminology. Retrieved October 22, 2012, from http://www.hc-sc.gc.ca/dhp-mps/medeff/databasdon/glossary_definition-eng.php

Health Canada. (2012e). Guidance Document: Considerations for Inclusion of Women in Clinical Trials and Analysis of Data by Sex. Retrieved February 26, 2013, from http://www.elsevierbi.com/~/media/Supporting%20Documents/The%20Gray%20Sheet/
38/4/Canada_draft_guidance.pdf

Health Canada. (2012f). Interpretation of Suspected Adverse Reaction Data. Retrieved September 9, 2013, from http://www.hc-sc.gc.ca/dhp-mps/medeff/databasdon/interpretation-eng.php

Health Canada. (2012g). Marketed Health Products Directorate. organizational description. Retrieved August 20, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/hpfb-dgpsa/mhpd-dpsc/index-eng.php

Health Canada. (2012h). Paediatric Expert Advisory Committee (PEAC) - Health Canada. resource list. Retrieved October 4, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/hpfb-dgpsa/opi-bip/peac-cceip/index-eng.php

Health Canada. (2012i). Performance Measurement and Evaluation Plan - EERC One-year Study.

Health Canada. (2012j). Regulatory Roadmap for Health Products and Food. Retrieved from http://www.hc-sc.gc.ca/ahc-asc/alt_formats/pdf/activit/mod/roadmap-feuillederoute-eng.pdf

Health Canada. (2012k). Statement on the authorization of generic OxyContin. Retrieved December 6, 2012, from http://www.hc-sc.gc.ca/ahc-asc/media/ftr-ati/_2012/2012_176-eng.php

Health Canada. (2012l). Summative evaluation framework: Human Drugs Activities.

Health Canada. (2012m). Use of international paediatric information by Health Canada.

Health Canada. (2013a). Draft Guidance for Industry: Submission and Information Requirements for Extraordinary Use New Drugs (EUNDs).

Health Canada. (2013b). Drugs removed from the market for safety reasons, 2004-2012, data extract.

Health Canada. (2013c). E-Communications: Interim Web Publishing Criteria and Process.

Health Canada. (2013d). Evaluating the Effectiveness of (Health Product) Risk Communications (EERC) in the Health Products and Food Branch: Project Charter.

Health Canada. (2013e). Federal Government Strengthens Drug Safety with New Standards - Health Canada News Release 2013-05-08. Retrieved July 5, 2013, from http://www.hc-sc.gc.ca/ahc-asc/media/nr-cp/_2013/2013-58-eng.php

Health Canada. (2013f). Health Canada's Clinical Trials Database. Retrieved June 3, 2013, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/databasdonclin/index-eng.php

Health Canada. (2013g). HPFB Risk Communication Process for Human Health Products (draft).

Health Canada. (2013h). Inspectorate Program: Inspectorate Annual Report 2012-2013.

Health Canada. (2013i). New standard for hospitals on reporting adverse drug reactions released today - Standard will be supported by marketing activities to boost reporting by health professionals. Retrieved March 21, 2013, from http://www.hc-sc.gc.ca/ahc-asc/media/nr-cp/_2013/2013-01-eng.php

Health Canada. (2013j). Notice - Adoption of the International Conference on Harmonisation (ICH) Guidance on Periodic Benefit Risk Evaluation Report - ICH Topic E2C(R2), as of March 1, 2013. Retrieved July 4, 2013, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/applic-demande/guide-ld/ich/efficac/e2cr2_notice_avis-eng.php

Health Canada. (2013k). Status Report on the Financial Management and Performance of the Human Drugs and Medical Devices Programs.

Health Canada. (2013l). Update and response to OAG recommendations for the Regulation of Pharmaceutical Drugs in Fall 2011: Health Canada's Action Plan as of March 28th, 2013.

Health Canada, & Spearhead Management Canada Limited. (2007). Costing model report - final deliverable.

Health Council of Canada. (2010). Decisions, Decisions: Family Doctors as Gatekeepers to Prescription Drugs and Diagnostic Imaging in Canada. Retrieved from http://www.healthcouncilcanada.ca/tree/2.33-DecisionsHSU_Sept2010.pdf

HECSB. (2009). Drug Strategy and Controlled Substances Programme - Healthy Environments and Consumer Safety Branch. Retrieved October 24, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/hecs-dgsesc/dscsp-psasc/index-eng.php

HESA. (2008). Report of the Standing Committee on Health: Post-market surveillance of pharmaceuticals. Retrieved from http://publications.gc.ca/collections/collection_2008/parl/XC62-392-1-1-04E.pdf

HESA. (2012). Report of the Standing Committee on Health: Drug supply in Canada: A multistakeholder responsibility. Retrieved from http://www.parl.gc.ca/content/hoc/Committee/411/HESA/Reports/RP5640047/
hesarp09/hesarp09-e.pdf

HPFB. (2005a). Draft guidance for industry: Marketed health product name assessment: Look-alike sound-alike (LA/SA) health product names.

HPFB. (2005b). HPFB public involvement framework.

HPFB. (2007a). Blueprint for Renewal II.

HPFB. (2007b). Review of regulated products: policy on public input.

HPFB. (2008). Guidance document for industry and practitioners: Special Access Programme for Drugs.

HPFB. (2009). Guidance document: Acetaminophen labelling standard. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/applic-demande/guide-ld/label_stand_guide_ld-eng.pdf

HPFB. (2011a). Cost recovery: Project summary.

HPFB. (2011b). Guidance document: Notice of compliance with conditions (NOC/c).

HPFB. (2011c). Guidance for industry: Management of drug submissions.

HPFB. (2012a). DSTS Premeeting submission types 2002-2012.

HPFB. (2012b). HPFB international activities: Inventory of significant international activities in HPFB (draft, updated April 2012).

HPFB. (2012c). HPFB mid-year report 2012-13: Summary of Successes and Challenges.

HPFB. (2012d). Pharmaceutical Drugs Program: 2012-13 mid-year review.

ICH. (2013). Quality Guidelines?: ICH. Retrieved July 5, 2013, from http://www.ich.org/products/guidelines/quality/article/quality-guidelines.html

Industry Canada. (2012). Pharmaceutical Industry Profile - Canadian Life Science Industries. Retrieved May 23, 2013, from http://www.ic.gc.ca/eic/site/lsg-pdsv.nsf/eng/h_hn01703.html

Inspectorate. (2002). Notice of Intent Published in Canada Gazette Part I, December 7, 2002 and Good Manufacturing Practices Guide for Active Pharmaceutical Ingredients (ICH Topic Q7A). Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/legislation/gazette1-q7a_tc-tm-eng.php

Inspectorate. (2007a). A2.2 Human Drugs - GMP Inspections operational planning (2007-08).

Inspectorate. (2007b). Inspectorate performance report (2006-07, year-end).

Inspectorate. (2009a). Inspectorate strategic plan (2009-12).

Inspectorate. (2009b). Policy on manufacturing and compounding drug products in Canada. Retrieved from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/compli-conform/pol_0051-eng.pdf

Inspectorate. (2010a). Border Integrity Approach (POL-0059). Retrieved July 20, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/compli-conform/import-export/pol_0059_BI_approach-eng.pdf

Inspectorate. (2010b). Inspectorate performance report (2009-10).

Inspectorate. (2010c). Policy on Counterfeit Health Products (POL-0048) [Health Canada, 2010]. Retrieved September 13, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/activit/pol_0048_counterfeit-contrefacon-eng.php

Inspectorate. (2012a). A2.10 PMRC - Drug Inspections operational planning (2011-12) (v2).

Inspectorate. (2012b). A2.2 Human Drugs - GMP Inspections operational planning (2011-12) ("Clean").

Inspectorate. (2012c). Compliance and Enforcement Risk Evaluation Guide: An approach to decision-making.

Inspectorate. (2012d). Guidance Documents, Directives and Policies. Retrieved November 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/info-prod/drugs-drogues/docs-eng.php

Inspectorate. (2012e). Guidance on Drug Establishment Licences and Drug Establishment Licensing Fees (GUI-0002). Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/compli-conform/licences/directives/gui-002-eng.pdf

Inspectorate. (2012f). Risk Classification of Good Manufacturing Practices (GMP) Observations (GUI-0023). Retrieved November 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/compli-conform/gmp-bpf/docs/gui-0023-eng.pdf

Inspectorate. (2012g). Summary report of inspections of clinical trials conducted from April 2004 to March 2011. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/compli-conform/clini-pract-prat/report-rapport/2004-2011-eng.pdf

Inspectorate. (2012h). Summary report of the Drug Good Manufacturing Practices (GMP) Inspection Program - April 1, 2006 to March 31, 2011. Retrieved November 26, 2013, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/gmp-bpf/2011-03-31-report-rapport-eng.php

Inspectorate. (2012i). Updates: Mutual Recognition Agreements. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/int/mra-arm/update-miseajour/index-eng.php

Inspectorate. (2013). Procedure or Work Instruction: Clinical Trial Site Selection for Inspection: SOP- 0548.

IOM. (2008). Forum on Drug Discovery, Development, and Translation: Addressing the Barriers to Pediatric Drug Development: Workshop Summary - Chapter 2: Regulatory Framework. Retrieved February 26, 2013, from http://www.ncbi.nlm.nih.gov/books/NBK3997/

Labrie, Y. (2012). How Can We Prevent Prescription Drug Shortages? Retrieved from http://www.iedm.org/files/note0912_en.pdf

Lemmens, T., & Bouchard, R. (2007). Regulation of pharmaceuticals in Canada. In J. Downie, T. Caulfield, & C. Flood (Eds.), Canadian Health Law and Policy (3rd ed., pp. 311-365).

Lexchin, J. (2012). New Drugs and Safety: What Happened to New Active Substances Approved in Canada Between 1995 and 2010? Archives of Internal Medicine, 1. doi:10.1001/archinternmed.2012.4444

Liu, H., Zhang, Z., & Linhardt, R. J. (2009). Lessons learned from the contamination of heparin. Natural Product Reports, 26(3), 313. doi:10.1039/b819896a

MHPD. (2008). MHPD retrospective: The first five years 2002-2007 (progress report).

MHPD. (2009). Towards Comprehensive Risk Management Planning: Health Canada's Proposed Approach. Retrieved July 3, 2013, from http://www.mclaughlincentre.ca/events/Parmaco2/Nashwa%20Irfan_UofO
%20workshop%20May%2028-29%202009.pdf

MHPD. (2010). Canada Vigilance_Signal Detection Business Transformation: Project overview - Office of the Auditor General.

MHPD. (2011a). Adverse Reaction Reporting and Health Product Safety Information - Guide for Health Professionals. guide. Retrieved September 18, 2012, from http://www.hc-sc.gc.ca/dhp-mps/pubs/medeff/_fs-if/2011-ar-ei-guide-prof/index-eng.php

MHPD. (2011b). Procedure - The Release to the Public of Information Obtained from Adverse Reaction and Medical Device Incident Reports [Health Canada, 2011]. guide. Retrieved September 14, 2012, from http://www.hc-sc.gc.ca/dhp-mps/pubs/medeff/_guide/2011-procedure/index-eng.php

MHPD. (2011c). Strengthening HPFB's Health Product Vigilance System into the Future - DRAFT.

MHPD. (2012a). Adverse reaction database: Canada Vigilance adverse reaction online database [2012]. Retrieved August 27, 2012, from http://www.hc-sc.gc.ca/dhp-mps/medeff/databasdon/index-eng.php

MHPD. (2012b). MedEffect Canada: Advisories, warnings and recalls. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/medeff/advisories-avis/index-eng.php

MHPD. (2012c). MHPD performance standards.

MHPD. (2012d). Standard Operating Procedure: Periodic Safety Update Report (PSUR) Screening-(Level I review).

MHPD. (2012e). Standard Operating Procedure: Prioritization and management of potential signal files in MHPD.

MHPD. (2012f). Standard Operating Procedure: Signal Assessment.

MHPD. (2012g). The Past and Future of Pharmacovigilance Activities at the Marketed Health Products Directorate: HPFB International Pharmacovigilance Symposium.

MHPD. (2013). Standard Operating Procedure: MHPD Signal Detection (MBBNHPB).

MHRA. (2009). Summary of Regulation on Me Dicines for Paediatric Use. Retrieved February 26, 2013, from http://www.mhra.gov.uk/home/groups/pl-a/documents/websiteresources/con2025602.pdf

Mini-Sentinel. (n.d). Background | About Mini-Sentinel | Mini-Sentinel. Retrieved July 5, 2013, from http://mini-sentinel.org/about_us/

Nanos Research. (2010). Results of drug safety survey of Canadians.

NIH. (2012). FDAAA 801 Requirements - ClinicalTrials.gov. Retrieved February 26, 2013, from http://clinicaltrials.gov/ct2/manage-recs/fdaaa

OAG. (2006). Report of the Auditor General of Canada to the House of Commons: Chapter 8 - Allocating funds to regulatory programs - Health Canada.
OAG. (2011). Report of the Auditor General of Canada to the House of Commons: Chapter 4 - Regulating pharmaceutical drugs - Health Canada.

Olson, M. (2002). Pharmaceutical Policy Change and the Safety of New Drugs. Journal of Law and Economics, 45(S2), 615-642.

Pew Health Group. (2011). After Heparin: Protecting Consumers from the Risks of Substandard and Counterfeit Drugs. Retrieved from http://www.pewtrusts.org/uploadedFiles/wwwpewtrustsorg/Reports/Health/
Pew_Heparin_Final_HR.pdf

Ramage-Morin, P. (2009). Medication use among senior Canadians (Health Report No. 82-003-X). Statistics Canada. Retrieved from http://www.statcan.gc.ca/pub/82-003-x/2009001/article/10801-eng.pdf

RCC Personal Care Products and Pharmaceuticals Working Group. (2012a). Good Manufacturing Practices (GMP) Work Plan. Retrieved July 24, 2013, from http://actionplan.gc.ca/grfx/BAP-RCC/GMP_Reformat.pdf

RCC Personal Care Products and Pharmaceuticals Working Group. (2012b). Over-the-Counter Therapeutic Products Approval and Licensing Work Plan. Retrieved July 25, 2013, from http://actionplan.gc.ca/grfx/BAP-RCC/Common_Monograph_OTC_Reformat.pdf

Risk Sciences International. (2012). Ensuring drug safety and effectiveness through pharmacovigilance. Retrieved from http://www.risksciencesint.com/app/wa/mediaEntry?mediaEntryId=11780

Sparrow, M. (2000). The Regulatory Craft: Controlling Risks, Solving Problems, and Managing Compliance. Washington, D.C.: Brookings Institution Press.

Srba, J., Descikova, V., & Vlcek, J. (2012). Adverse drug reactions: Analysis of spontaneous reporting system in Europe in 2007-2009. European Journal of Clinical Pharmacology, 68(7), 1057-1063. doi:10.1007/s00228-012-1219-4

SSCSAST. (2012). Canada's Clinical Trial Infrastructure: A Prescription for Improved Access to New Medicines. Retrieved November 26, 2012, from http://www.parl.gc.ca/Content/SEN/Committee/411/soci/rep/rep14nov12-e.pdf

TBS. (2009). Policy on Evaluation. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?section=text&id=15024

TGA. (2012). Therapeutic product vigilance. Retrieved July 3, 2013, from http://www.tga.gov.au/about/tga-therapeutic-product-vigilance.htm

The Antima Group, & TNS Canada. (2007). Underutilization of the Adverse Reaction Reporting System. Retrieved October 11, 2012, from http://epe.lac-bac.gc.ca/100/200/301/pwgsc-tpsgc/por-ef/health/2007/385-06/summary.pdf

Toronto Star. (2013a). Health Canada considers mandatory reporting of adverse drug reactions. Retrieved May 2, 2013, from http://www.thestar.com/news/canada/2013/04/16/conservative_government_aims_
to_improve_safety_of_food_childrens_products_and_prescription_drugs.html

Toronto Star. (2013b, September 6). Recalls of faulty birth control pills alarm Canadian doctors. Retrieved October 7, 2013, from http://www.thestar.com/news/canada/2013/09/06/recalls_of_faulty_birth_control_
pills_alarm_canadian_doctors.html

TPD. (2004). TPD business transformation progress report (2003-04).

TPD. (2007). Drug licensing process.

TPD. (2008a). Annual Drug Submission Performance Report - TPD - 2008.

TPD. (2008b). Frequently Asked Questions: Product Monographs Posted to the Health Canada Website. Retrieved September 4, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/pm_qa_mp_qr-eng.pdf

TPD. (2008c). Notice: To Market Authorization Holders: Health Canada's Decision on Labelling of Certain Paediatric (0 to under 12 years) Nonprescription Cough and Cold Products in Canada. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/notice_avis_decision_pedlscc_pednecr-eng.pdf

TPD. (2010a). Annual Drug Submission Performance Report - TPD - 2010. Retrieved August 31, 2012, from http://www.jptcp.com/files/tpd_dpt_annual_annuel_10_eng.pdf

TPD. (2010b). Introduction to the current system (drug regulation in Canada).

TPD. (2010c). Product monograph template (notice of revisions). Retrieved September 4, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/applic-demande/guide-ld/monograph/pmappe_mpanne-eng.pdf

TPD. (2011). Product information (labelling).

TPD. (2012a). Consultation notice: Draft Guidance Document - Acetylsalicylic Acid Labelling Standard. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/consultation/drug-medic/consult_asa_gd_draft_aas_ld_ebauche-eng.php

TPD. (2012b). Draft Guidance Document: Acetylsalicylic Acid Labelling Standard - 2012 Health Canada Consultation Notice. Retrieved September 12, 2012, from http://www.hc-sc.gc.ca/dhp-mps/consultation/drug-medic/consult_asa_gd_draft_aas_ld_ebauche-eng.php

TPD. (2012c). Frequently Asked Questions: Summary Basis of Decision (SBD) project: Phase II.
TPD, & BGTD. (2009). Evaluation of phase I of the Summary Basis of Decision project.

US Government. (1996). 21 CFR 314.80 Postmarketing reporting of adverse drug experiences. Retrieved July 5, 2013, from http://www.gpo.gov/fdsys/pkg/CFR-1996-title21-vol5/pdf/CFR-1996-title21-vol5-chap-id2.pdf

US Government. Public Law 110-85: An Act to Amend the Federal Food, Drug and Cosmetic Act to revise and extend the user-fee programs for prescription drugs and for medical devices, to enhance postmarket authorities of the FDA with respect to the safety of drugs and for other purposes. , Pub. L. No. 110-85 (2007). Retrieved from http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=85

US Government. (2012a). CFR - Code of Federal Regulations Title 21 - 201.57 - Subpart B-Labeling Requirements for Prescription Drugs and/or Insulin. Retrieved July 9, 2013, from http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm

US Government. (2012b). CFR - Code of Federal Regulations Title 21 - Sec. 201.56 Requirements on content and format of labeling for human prescription drug and biological products. Retrieved July 8, 2013, from http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=201.56&utm_
campaign=Google2&utm_source=fdaSearch&utm_medium=website&utm_term=
21%20CFR%20201.56&utm_content=1

Weise, M., Bielsky, M., De Smet, K., Ehmann, F., Ekman, N., Narayanan, G., Schneider, C. (2011). Biosimilars - why terminology matters. Nature Biotechnology, 29(8), 690-693.

Woo, J., Wolfgang, S., & Batista, H. (2008). The effect of globalization of drug manufacturing, production, and sourcing and challenges for American drug safety. Clinical Pharmacology & Therapeutics, 83(3), 494-497.

Appendix C - Supplementary Data Tables

Table 1: Number of CTAs received and approved, human drugs and biologics
Year	Human drugs		Biologics
Year	# received	# approved (NOLs)	# received	# approved (NOLs)
2004	1730	1677	258	229
2005	1732	1658	239	210
2006	1685	1621	272	245
2007	1724	1633	278	253
2008	1613	1579	267	232
2009	1400	1341	266	247
2010	1191	1162	272	263
% change, 2004-2010	-31%	-31%	5%	15%
Source: (BGTD, 2010; TPD, 2008a, 2008b, 2010a)

Table 2: Signal detection outcomes, human drugs - All MPMDB working groups and pre-working groups combined (2006-2012)
Dismissed
	Number	%	Combined %
Dismissed at 1st screening	1,209	61.1%	80.5%
Dismissed at 2nd screening	304	15.4%
Dismissed at signal coordination meeting	32	1.6%
Dismissed at signal prioritization or 3rd screening	49	2.5%
Total dismissed	1,594
Other outcome
Contact/involve TPD	77	3.9%	12.1%
PSUR	35	1.8%
Contact/involve ORMS	4	0.2%
Contact/involve MHPSEIB	4	0.2%
Other	35	1.8%
Blank	84	4.2%
Total other	239
Prioritized for signal assessment
Green	69	3.5%	7.4%
Yellow	59	3.0%
Red	2	0.1%
Prioritized	17	0.9%
Total prioritized	147
Total	1,980

Table 3: Signal detection outcomes, biologics - MBB SIC WG (2008-2012)
Dismissed
	Number	%	Combined %
Dismissed before preliminary review	6,288	96.7%	96.9%
Dismissed before signal prioritization	1	0.0%
Dismissed by signal prioritization committee	15	0.2%
Total dismissed	6,304
Other actions
Continue monitoring / monitor in future PSUR / no further assessment	133	2.0%	2.6%
Ad hoc request (AHR) / gap analysis	10	0.2%
PSUR	4	0.1%
Other (N/A, blank, forward to MHPSEIB, BGTD, other)	19	0.3%
Total other	166
Prioritized for signal assessment
Low	18	0.3%	0.5%
Medium	10	0.2%
High	-	-
Prioritized	7	0.1%
Total prioritized	35
Total	6,505
Sources: MAHSI WG tracking sheet.xls; PPSD WG tracking sheet.xls; Signal Detection Tracking Sheet.xls

Table 4: Sources for signal assessments, human drugs (2003-2012)
Sources	# of signal assessments	% of total	% (combined)*
International regulator or agency
US (FDA)	39	14.3%	26.1%
EMEA	15	5.5%
France (Affsaps)	6	2.2%
Italy (SIFA)	5	1.8%
UK (MHRA)	4	1.5%
New Zealand (Medsafe)	3	1.1%
WHO	2	0.7%
Australia (TGA)	1	0.4%
Health Canada (mechanisms)
Canada Vigilance Database	23	8.5%	16.5%
PSUR	15	5.5%
Signal assessment	7	2.6%
Health Canada (body)
TPD	19	7.0%	8.5%
MHPD	3	1.1%
Inspectorate	1	0.4%
Other sources
Literature	70	25.7%	-
MAH	18	6.6%	-
CNODES	2	0.7%	-
DSEN	1	0.4%	-
Other	4	1.5%	-
Source not indicated	40	14.7%	-
Total number of signal assessments	272	-	-
Source: MPMDB activity signal tracking (2006-12) (Excel Spreadsheet) (received February 2013). Filename: Copy of 2013-02 MPMDB Activity_Signal Tracking.xls Note: Column totals will not sum because signal assessments may have multiple sources. *Percent of total may not equal combined percentage because the latter counts only whether or not the signal assessment was sourced from an international regulator, a Health Canada mechanism, or a Health Canada directorate. Differences in percentages reflect the fact that some signal assessments were sourced from multiple international regulators or Health Canada directorates.

Table 5: Recommendations resulting from completed signal assessments, human drugs (2003-2012)
Recommendation	Count	Percent
Recommend changes to product labelling	148	54%
Standard monitoring	99	36%
Request issuance of a risk communication	66	24%
Request additional safety information	28	10%
Refer to MHPSEIB for future reassessment	24	9%
Request consideration by the DSEN	17	6%
Request submission of a pharmacovigilance and/or risk management plan (PvP/RMP)	11	4%
Enhance monitoring	7	3%
Request a benefit/risk assessment from the MAH	5	2%
Request information from another regulator	4	1%
Recommend issue analysis summary	3	1%
Recommend regulatory intervention based on C.01.013, C.01.014 and C.08.006 of the Food and Drugs Act and Regulations	3	1%
Information sharing with other directorate	2	1%
Other	5	2%
Closed	6	2%
Total	272
Source: Filename: Copy of 2013-02 MPMDB Activity_Signal Tracking.xls Note: Signal assessments may result in more than one recommendation; therefore, columns will not sum.

Table 6: RMPs, human drugs and biologics, 2005 to 2012
	2005	2006	2007	2008	2009	2010	2011	2012	Total
TPD	2	3	11	27	13	25	31	49	161
BGTD	0	5	4	13	24	13	13	26	98
MHPD	0	0	1	0	3	2	3	0	9
SIPD	0	0	0	1	0	0	0	0	1
Total	2	8	16	41	40	40	47	75	269
Source: Data provided by Health Canada. Notes: SIPD is the Submission Information and Policy Division within TPD; RMPs associated with SIPD relate to human drugs. It is unclear how many of the RMPs associated with MHPD relate to human drugs and biologics respectively. It is unclear if these data represent the number of RMPs requested, received, or reviewed by Health Canada.

Table 6: PSURs, human drugs and biologics, 2005 to 2012
	2005	2006	2007	2008	2009	2010	2011	2012	Total
TPD	51	24	21	8	14	48	29	19	214
BGTD	3	6	13	5	17	23	7	8	82
MHPD	0	2	4	7	3	0	3	1	20
SIPD	18	33	8	4	2	1	0	0	66
Total	72	65	46	24	36	72	39	28	382
Source: Data provided by Health Canada. Notes: SIPD is the Submission Information and Policy Division within TPD; PSURs associated with SIPD relate to human drugs. It is unclear how many of the PSURs associated with MHPD relate to human drugs and biologics respectively. It is unclear if these data represent the number of PSURs requested, received, or reviewed by Health Canada.

Table 8: Service standards for post-market activities
Post-market activity	Target days to completion
Adverse reaction reports
Priority initial report (death or life-threatening outcome)	15
Initial report	42
All initial and follow-up reports, standard and priority	84
Other post-market activities
Signal Assessment	130
RMP review	90
PSUR screening (Level I)	30
PSUR review (Level II)	90
Ad Hoc Reviews	60
Advertising Issue Assessment	15
Source:(Health Canada, 2011b; MHPD, 2012b) Note: Performance targets for adverse event/reaction reports are from time of receipt to time of completion, in calendar days. Performance targets for other activities are from time of assignment to time of completion, in working days.

Table 9: Summary of research findings relating to stakeholder awareness and understanding of risks related to human drugs and biologics
Survey	Target	Relevant findings
Public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada (Decima Research, 2003)	Public (n=1,500)	Perceptions of drug safety Most consumers (84%) are confident in the safety of prescription drugs, with 61% believing such drugs are generally safe, and 23% stating they are very safe. A similar proportion of consumers (85%) expressed confidence in the systems and safeguards in place to ensure the safety of prescription drugs for sale in Canada. Consumers are generally confident in the safety of non-prescription drugs (75%), with 62% believing such drugs are generally safe, and 14% believing they are very safe. Source of new drug safety information^{Table 8 footnote 1} Among those consumers who look for new safety information for health products they are already taking (n=1,171), very few (3%) consumers (unaided) identified Health Canada's website as a source for this information. When aided, 69% of all consumers said they were aware of Health Canada's Public Advisories and Warnings. Most of these consumers (62%) were aware of this information being issued through the media, while about 31% of all consumers were aware of new drug safety information on Health Canada's website (2003, p. 29,31). Among those consumers who had used Health Canada's website to look for new drug safety information in the previous 6 months (n=125 or 8% of total), the vast majority (91% or 114) were satisfied with Health Canada's website as a source of drug safety information (2003, pp. 30-31).
	Health professionals (n=551)	Familiarity with new drug safety information sources: Manufacturer-issued Dear Health Professional Letters (DHPLs) Just over half (54%) of the health professionals indicated they were either very familiar or somewhat familiar with manufacturer-issued Dear Health Professional Letters (DHPLs) (2003, p. 47). Among the health professionals who had used this source in the past 12 months (n=126 or 23%) most did so occasionally (57%) or rarely (25%), and when accessing the material, they tended to read it selectively (49%) or thoroughly (33%) (2003, p. 51). Among the health professionals who had used this source in the past 12 months, most (86%) indicated they were satisfied with the source because it was a good source of information (51%), contained current/up-to-date information (35%), and gave them the information/answers they wanted (27%) (2003, pp. 53-54). Health Canada-issued DHPLs 42% of health professionals were ether very or somewhat familiar with Health Canada-issued DHPLs (2003, p. 47). Among the health professionals who had used this source in the past 12 months (n=88 or 16%), most did so occasionally (52%) or rarely (31%). When doing so, they tended to read selectively (45%), read thoroughly (29%), or glance through it (26%) (2003, p. 51). Among the health professionals who had used this source in the past 12 months, most (84%) indicated they were satisfied with the source because it was a good source of information (55%), contained current/up-to-date information (35%), and gave them the information/answers they wanted (24%) (2003, pp. 53-54). Canadian Adverse Reaction Newsletter Just over half (53%) of the health professionals were very or somewhat familiar with the Canadian Adverse Reaction Newsletter (2003, p. 47). Among the health professionals who had used this source in the past 12 months (n=97 or 18%), most did so occasionally (52%) or rarely (31%). When doing so, they tended to read selectively (59%), glance through it (24%), or read it thoroughly (18%) (2003, p. 51). Among the health professionals who had used this source in the past 12 months, the vast majority (90%) indicated they were satisfied with the source because it was a good source of information (49%), contained current/up-to-date information (40%), and gave them the information/answers they wanted (30%) (2003, pp. 53-54). Health Canada's online drug safety advisories About 4 in 10 (38%) were familiar with Health Canada's online drug safety advisories (2003, p. 47). Among those who had used this source in the past 12 months (n=65 or 12%), most did so occasionally (52%) or rarely (39%). When doing so, they tended to read selectively (52%), read thoroughly (23%), or glance through it (23%) (2003, p. 51). Among the health professionals who had used this source in the past 12 months, most (82%) indicated they were satisfied with the source because it was a good source of information (49%), contained current/up-to-date information (26%), and gave them the information/answers they wanted (28%) (2003, pp. 53-54). Health Canada electronic mailing list Very few (11%) were aware of Health Canada's electronic mailing list, and even fewer (n=9 or 2%) had accessed this source in the past 12 months (2003, pp. 47, 51). ADRs 39% of health professionals consider Health Canada the best source of information on ADRs (2003, p. 61). 88% of health professionals perceive ADRs as a somewhat or very serious problem. 36% of health professionals said they would report ADRs in all situations, while 24% mentioned they would not report an ADR if it expected/well-known, or was too minor or trivial (23%). Overall, just over half (55%) of the health professionals were aware of how to report an ADR; higher levels of awareness were found among specific health professions, such as pharmacists (92%) and, to a lesser degree, physicians (63%) (2003, pp. 83-84). Among the health professionals who had filed an ADR in the past year (n=108), most (56%) mailed/faxed the form to Health Canada or contacted the drug manufacturer (48%). Most health professionals support the need for mandatory reporting of ADRs in Canada, though opinion is divided among physicians (Decima Research, 2003, p. 90).
Final report: 2006 General public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada (Follow-up to 2003 survey) (Decima Research, 2006)	Public (n=1,513)	Perceptions of drug safety Most respondents perceived prescription drugs as being safe (86%), including 20% who felt these drugs were very safe. Most respondents also felt non-prescription drugs were safe (75%), including 12% who felt they were very safe (2006, p. 15). About 78% of respondents expressed confidence in how the federal government monitors and regulates drug safety and effectiveness (2006, p. 16). New drug safety information 36% of respondents were aware of Health Canada's website as a source of new drug safety information. Among those who had accessed Health Canada's website in the past six months (n=147), most looked for drug information (57%) or ADR information (34%) (2006, pp. 29-30). More commonly (62%), respondents were aware of public advisories and warnings issued through the media (2006, p. 29). Very few respondents (1% or n=82) had subscribed to the MedEffect e-Notice. Among those who had not, most stated it was not very likely (29%) or not at all likely (38%) they would subscribe to it (2006, p. 31) About one quarter of respondents were aware Health Canada collects reports about adverse reactions from consumers (2006, p. 36).
Adverse Reaction Reporting - Survey with Health Professionals (Follow-up to 2003 survey) (Only executive summary is available online) (Environics Research Group, 2007)	Health professionals (n=1,108)	Sources for new drug safety information 89% of health professionals feel it is very important to stay current about new drug safety information, though fewer (56%) seek this type of information on a frequent basis. Unaided, 12% of health professionals listed Health Canada / MedEffect as a source for such information (2007, p. 2). 83% of health professionals report being very likely to read information received from Health Canada (2007, p. 3) ADRs About half the health professionals claim familiarity with how to report an adverse reaction; pharmacists were most likely to report being familiar with this process (87%), followed by physicians (51%). Overall, 37% of health professionals indicated they knew where to obtain an ADR reporting form (2007, p. 3).
Underutilization of the Adverse Reaction Reporting System (only executive summary is available online) (The Antima Group & TNS Canada, 2007)	Health professionals (n=48)	Sources for new drug safety information Mostly commonly, health professionals seek post-market drug information through the College of Physicians and Surgeons, journals, and interaction with colleagues. Very few health professionals were aware of the MedEffect website, and among those who were aware, only one had subscribed to the e-notice. Only three health professionals (two pharmacists and a medical doctor) indicated they had ever reported an ADR. There is a lack of awareness regarding how and why ADRs should be reported, and who should be making these reports to Health Canada.
Results of drug safety survey of Canadians Nanos, various years (Nanos Research, 2010)	Public	A 2007 survey of an unidentified number of Canadians showed most Canadians were either somewhat concerned (29%) or concerned (48%) about the safety of medicines. A 2010 survey of 1,008 Canadians showed that about 65% feel medications approved by Health Canada are safe.
Canadians' Awareness of Health and Safety Issues Related to Consumer Products (Phoenix Strategic Perspectives Inc, 2011)	Public Survey (n=1,357) Focus groups (x8) (n=64)	Research was conducted on awareness of health and safety issues related to a wide range of consumer products, which included "medications." However, the survey report does not describe results by product type, and, therefore, data from this survey could not be used. The survey examined topics such as sources of safety information; awareness and use of safety information such as advisories, warnings, and recalls; and subscription to Health Canada RSS feeds.
Table 8 footnotes Table 8 footnote 1 Includes natural health products (NHPs). Reported data are drawn from the survey report and not the survey's dataset, and, therefore, it is not possible disaggregate prescription and non-prescription drugs from NHPs. Return to table 1 footnote 1 referrer

Table 10: NDS and ANDS as percentage of human drug submissions
	2004	2005	2006	2007	2008	2009	2010	2011	2012	Total/ average
Total submissions	286	328	389	359	387	407	368	400	506	3,430
NDS as % of total	16.4%	15.5%	13.6%	14.2%	13.7%	15.7%	14.7%	13.3%	13.6%	14.5%
ANDS as % of total	40.9%	41.8%	31.9%	39.6%	41.6%	44.0%	50.3%	52.0%	42.4%	42.6%
Sources: (TPD, 2008a, 2010a). Data for 2011 and 2012 provided by Health Canada. Total and calculations for 2012 based on 215 ANDS.

Table 11: Domestic GMP inspections for human drugs (including biologics) and veterinary drugs
Fiscal Year	Beginning backlog	Annual inspection target	Inspections*	Complete to annual target	Non-compliant	Compliancy rate	Backlog to carry over
2005-06	N/A	567	420	74%	21	95%	165
2006-07	165	543	356	66%	6	98%	187
2007-08	165	484	446	92%	9	98%	170
2008-09	210	689	440	64%	8	98%	57
2009-10	117	475	406	86%	15	96%	78
Source: (Inspectorate, 2007, 2010b) *This is presumed to refer to inspections initiated.

Table 12: PMRC inspections for human drugs and biologics
Fiscal Year	Target	Inspections completed	Complete to annual target	Non-compliant	Compliancy rate
2004-05	-	25	-	0	100%
2005-06	243	75	31%	0	100%
2006-07	256	120	47%	0	100%
2007-08	242	153	63%	0	100%
2008-09	94	76	81%	0	100%
2009-10	64	72	113%	0	100%
Source: (Inspectorate, 2004, 2006, 2007, 2010b)

Table 13: Human drug and biologics shipments inspected by the Border Integrity Unit
Fiscal year	Personal shipments			Commercial shipments			Total
	Refusal		Releases	Refusal		Releases
	(Suspected) counterfeit	All other	Releases	(Suspected) counterfeit	All other	Releases
Biologics (Schedule C)
2010-2011			2				2
2011-2012			17			8	25
Biologics (Schedule D)
2010-2011		1	9		1	4	15
2011-2012			17		14	15	46
Schedule F, prescription drugs
2010-2011	3,789	20,577	935	86	585	201	26,173
2011-2012	6,007	7,202	1,011	244	872	160	15,496
unscheduled, OTC
2010-2011		10	1,178		150	66	1,404
2011-2012		3	1,945		347	113	2,408
Source: (Inspectorate, 2010a, 2011, 2012a)

Table 14: Advisory Committees associated with the Human Drugs Program
Name	Description/mandate	Documented activity
Expert Advisory Committee on the Vigilance of Health Products (EAC-VHP)	The Committee's mandate "is to provide HPFB with ongoing external expert broad strategic policy advice on the safety and therapeutic effectiveness of marketed health products for human use. It will also provide a mechanism to involve the public, providing them with a forum to have their views heard by experts who can discuss their input and incorporate it into the advice provided" (MHPD, 2006).	There is record of nine meetings being held from 2007 to 2009 The Committee engaged in discussion and provided advice on 47 presented topics. The Committee's advice was used in developing the Recommendations for the Appropriate Use of Cough and Cold Products in Children document (MHPD, 2008). The Committee developed a Public Involvement Plan. Of the 450 stakeholders who were invited to provide input on three questions, 58 contributed prior to the meeting and 16 did at the meeting.
Paediatric Expert Advisory Committee	The Committee, which was formed in 2009, has the objective "to provide a way for Health Canada to seek expert advice and public involvement in the development, licensing, and continued vigilance for health products - pharmaceuticals, medical devices, biologics including vaccines and natural health products - on the market destined for children, and pregnant and nursing women." (Health Canada, 2012b) The Committee also provides input related to food safety and nutrition. It consists of 15 experts, including paediatric specialists, university professors, pharmacists, researchers, industry/patient groups, not-for-profit organizations, and parents (Health Canada, 2012b).	In May 2010, the Committee discussed "off-label" use of drugs in the paediatric population. Documents suggest that this led to a "Mind the Gap" study initiated by the Office of Paediatric Initiatives to address issues related to off-label drug use in paediatrics (MHPD, 2011). Sponsored an initiative to provide "an evidence-based and authoritative assessment on the state of therapeutic products for children in Canada and abroad" (GoC, 2012, p. 16:10)
New Drug Committee (NDC)	The Committee was formed at the request of Health Canada and the pharmaceutical company Shire BioChem Inc.,manufacturer of Adderall XR, to provide recommendations relating to the decision to suspend said drug formulation (Health Canada, 2005).	The Committee produced the Report of the Adderall XR New Drug Committee, which provided recommendations on the decision to suspend the Notice of Compliance for Adderall XR. This recommendation was subsequently adopted by Health Canada (the NOC was suspended in February 2005, but later reinstated)
Scientific Advisory Panel on Opioid Analgesic Abuse (SAP-OAA)	The panel "acts as a forum of input and a sounding board for management and scientist of Health Canada. The SAP-OAA will be consulted on issues concerning safety and abuse of new formulations of prescription opioid analgesics and proposals for the types of information which should be included in the Product Monograph of such drug products, as well as advice about risk management strategies" (TPD, 2011f).	There is record of one meeting taking place on March 29, 2011. The closing remarks of the meeting indicate that the participants felt the discussion was productive and that there was no need for a second meeting (TPD, 2011e).
Scientific Advisory Panel on Nonprescription Drugs (SAP-NPD)	The Committee "acts as a forum of input and a sounding board for management and scientists of Health Canada. The Committee provides scientific, technical and medical advice to assist the TPD in the regulations of nonprescription drugs" (TPD, 2011c).	There is record of one meeting taking place on November 24, 2012 (TPD, 2012a).
Scientific Advisory Panel on Oncology Therapies (SAC-OT)	The Committee "provides Health Canada with timely scientific, technical, and medical advice related to the regulation of oncology therapies. Involvement of the scientific, medical and consumer communities in the regulatory review process is expected to enhance transparency and provide opportunity for proactive external guidance, thus facilitating the drug review process. The SAC-OT provides Health Canada with advice and recommendations, but the decision-making responsibility remains with Health Canada" (TPD, 2010b).	There is record of four meetings taking place on September 13, 2007; February 26, 2009; April 15, 2010; and December 6, 2011. At these meeting the answers and recommendations on six topics posed to the Committee were provided (TPD, 2011b).
Scientific Advisory Committee on Pharmaceutical Sciences and Clinical Pharmacology (SAC-PSCP)	The Committee "will provide Health Canada with timely scientific, technical and clinical advice on pharmaceutical science and clinical pharmacology issues affecting data requirements for new formulations of approved drugs, for example, pharmaceutical equivalence for generic drug products, new salts, complex formulations, etc."(TPD, 2011d).	The only documented activity is a nomination call for members and a meeting announcement (TPD, 2011a) (TPD, 2012c). The terms of reference were not finalized.
Scientific Advisory Committee on Respiratory and Allergy Therapies (SAC-RAT)	The Committee "will provide Health Canada with ongoing and timely scientific, technical, and medical advice on the evaluation of safety, efficacy, and effectiveness of drugs used for the treatment of respiratory diseases and allergic conditions as well as policy issues"(TPD, 2010c).	There is record of eight meetings taking place since 2006 (TPD, 2012b). The Committee has developed two draft documents which are yet to be finalized: Submission Requirements for Subsequent Market Entry Inhaled Corticosteroid Products for Use in the Treatment of Asthma; and Submission Requirements for Subsequent Market Entry Steroid Nasal Products for Use in the Treatment of Allergic Rhinitis (TPD, 2012b).
Scientific Advisory Panel for Bioequivalence of Gender-Specific Drug Products (GSDP)	The Committee "provides advice to Health Canada on appropriate bioequivalence study requirements for pharmaceutical drug products intended for administration either only to females or only to males. Of particular interest are those drug products where concerns have been raised that existing study requirements may not be adequate, for example, combination products containing pyridoxine and vitamin B6 for the treatment nausea in pregnancy" (Health Canada, 2011c)	There is record of one meeting taking place on June 2, 2011 (Health Canada, 2011d).
Scientific Advisory Panel for Modified-Release Dosage Forms (MRDF)	The panel "provides advice to Health Canada on appropriate bioequivalence standards for modified-release dosage forms of pharmaceutical drugs. Of particular interest are those drugs where concerns have been raised that existing standards may not be adequate, for example, methylphenidate and nifedipine" (TPD, 2010d).	There is record of one meeting taking place on June 11, 2010. At this meeting the panel provided final recommendations on two questions posed by Health Canada (TPD, 2010e). The terms of reference were not finalized (TPD, 2010d).
Expert Advisory Panel on the Special Access Programme (EAP-SAP)	The Panel "acts as a forum of advice and a sounding board for management and scientists of [HPFB]. Panel members meet to discuss options for improving the regulatory framework under which the Special Access Programme will function" (Health Canada, 2007)	Nothing regarding the Panel's activity was available in the documents provided to PRA. According to Health Canada's website, this Panel's mandate has been completed (Health Canada, 2012c).
Scientific Advisory Committee on Human Reproductive Therapies (SAC-HRT)	The Committee "provides [HC] with timely scientific, technical and medical advice related to the evaluation of safety and efficacy data for human reproductive therapies that are submitted as part of the drug/medical devices review process and/or from post-market surveillance activities." The Panel will "enhance transparency and provide the opportunity for proactive external guidance" (TPD, 2003).	Nothing regarding the Panel's activity was available in the documents provided to PRA. According to Health Canada's website, this Panel's mandate has been completed (Health Canada, 2012a).
Scientific Advisory Committee on Metabolic and Endocrine Therapies (SAC-MET)	The Committee provides HC with "advice related to the evaluation of safety and efficacy data for metabolic and endocrine drug products that are submitted as part of the drug/medical device assessment process and/or from post-market surveillance activities. Involvement of [the Committee] in the regulatory review process is expected to enhance transparency and provide opportunity for proactive external guidance, thus facilitating the drug/medical device assessment process" (Health Canada, 2006)	There is record of one meeting taking place on November 15, 2007(Health Canada, 2006)
Scientific Advisory Committee on Musculoskeletal Therapies (SAC-MST)	No information available.	Committee was cancelled (Health Canada, 2011a)
Scientific Advisory Committee on Neurological Therapies (SAC-NT)	No information available.	Committee was cancelled (Health Canada, 2011a)

Table 15: Timeliness of signal assessment activities
	Human drugs 2003-2012	Biologics 2005-2012
Number of signal assessments completed	272	113
Signal receipt to signal assignment, median working days	65 days	14 days
Signal receipt to signal assignment, mean working days	157.7 days	70 days
Completed within 130 working days, pre-April 2007	10%	N/A
Completed within 130 working days, post-April 2007	51%	92%
Low priority assessments meeting target of 200 working days, post-April 2007	68%	97%
Medium priority assessments meeting target of 130 working days, post-April 2007	44%	99%
High priority assessments meeting target of 80 working days, post-April 2007	75%	100%
Median processing time in working days, pre-April 2007	348 days	N/A
Median processing time in working days, post-April 2007	128 days	46 days
Mean processing time in working days, pre-April 2007	481 days	N/A
Mean processing time in working days, post-April 2007	161 days	70 days

Table 16: Signal assessments that recommended a risk communication, human drugs (2010-2012) (n=38)
Priority level	Activity period	Calendar days			Total cases for this priority level and activity period
Priority level	Activity period	Min*	Max*	Med*	Total cases for this priority level and activity period
Green (low) (n=15)	Detected to received	1	91	25	10
	Received to assigned	1	1,310	199	15
	Assigned to approved	51	538	238	15
	Approved to posted	115	382	200	10

Yellow (medium) (n=19)	Detected to received	0	96	21	13
	Received to assigned	0	223	31	19
	Assigned to approved	54	835	199	19
	Approved to posted	32	507	176	17

Red (high) (n=1)	Detected to received	37	37	-	1
	Received to assigned	6	6	-	1
	Assigned to approved	106	106	-	1
	Approved to posted	203	203	-	1

Prioritized (no priority level assigned) (n=3)	Detected to received	42	42	-	1
	Received to assigned	0	12	6	2
	Assigned to approved	94	276	185	2
	Approved to posted	98	98	-	1
*Excludes negative values.

Table 17: HPFB participation in international fora and activities
Activity/forum	Description
American Association of Tissue Banks	This group publishes standards, accredits tissue banks internationally, interacts with regulatory agencies, and conducts educational meetings and training. BGTD is a member of the Standards Committee responsible for standard development, including 13th edition of the American Association of Tissue Banks (AATB) standards, which was published in March 2012.
Asia Pacific Economic Cooperation (APEC) through the Regulatory Harmonization Steering Committee (RHSC)	TPD chairs the RHSC, the mandate of which is to promote harmonization and identify projects of highest value in partnership with the APEC Harmonization Center, other harmonization initiatives, training organizations, and international players such as the WHO. In 2011, the APEC ministers endorsed the Strategic Framework on Regulatory Convergence of Medical Products by 2020. Health Canada currently chairs APEC's Life Sciences Working Group.
APEC GCP Inspector's Working Group	The Inspectorate participates in this group, which promotes harmonization of GCP and provides opportunities for networking and sharing of best practices with international colleagues. The group also supports development of regulatory frameworks of countries with less-developed regulatory systems.
African Medicines Registration Harmonization (AMRH)	Assists African countries to harmonize medicines registration, and reduce the time spent to register priority essential medicines. AMRH partners assist Regional Economic Communities in Africa in their development of project proposals for medicines registration harmonization projects. The kick-off stakeholders meeting was scheduled for March 28, 2012, in Tanzania.
Canadian HIV Vaccine Initiative (CHVI)	This is a five-year collaborative initiative between the Government of Canada and the Bill & Melinda Gates Foundation. Through this initiative, BGTD provides training to emerging national regulatory authorities (NRAs) from low- and middle-income countries to strengthen regulatory frameworks in relation to clinical trials and market authorization for vaccines. The initiatives below are supported through the CHVI.
a. African Vaccines Regulatory Forum	BGTD participates in this group, which provides regulators with expert advice on the evaluation of vaccines. The forum helps decision making with respect to clinical trial authorization, evaluation of registration dossiers, or other issues in vaccine evaluation.
b. CVHI Regulatory Capacity Building Mentorship Program	This mentorship program is intended to provide one-on-one learning opportunities for NRAs to strengthen regulatory capacity in biologics and vaccines. BGTD initiated a mentorship program with Malawi in 2012 under the Program.
c. Developing Countries' Vaccine Regulators Network	BGTD participates in this network, which promotes regulatory capacity development of NRAs through expertise and exchange of information about evaluation of clinical trial proposals and clinical trial data.
d. Second International Congress on Pharmacology of Vaccines	BGTD participated with PAHO countries in this congress (VacciPharma 2012) to discuss the use of prophylactic vaccines in relation to HIV.
Heads of Agencies Consortium	The Heads of Agencies Consortium also includes the TGA, Swissmedic, and Singapore's Health Sciences Authority with the goal of leveraging expertise and knowledge and identifying work-sharing opportunities. The group has developed a plan of action on generic drugs and has created a standardized template and process for sharing applications in queue and identifying common applications. Efforts are also currently underway for the partner agencies to work together in the review of PSURs.
International Laboratory Forum on Counterfeit Medicines (ILFCM)	Health Canada participates in the ILFCM, which is an informal information sharing forum for experts to share experiences about important operational issues such as the interception and analysis of counterfeit therapeutic products. The forum meets semi-annually.
International Generic Drug Regulators' Group (IGDRG)	Health Canada participates in the IGDRG, which is a forum for regulatory authorities to explore collaborative opportunities in the area of generic drugs. Collaborative opportunities include work sharing and increased regulatory alignment. At least two face-to-face meetings have taken place, resulting in a consensus on goals, objectives, and priority areas for collaboration.
International Regulatory Forum	An annual forum that is intended to contribute to African and Asian development and collaboration within the Americas. The fourth annual International Regulatory Forum was planned for September 24-28, 2012, in Ottawa, with approximately 100 participants from over 40 countries
OMCL Network	The Inspectorate Laboratories participate in this network. Annual meetings permit exchange on best practices and topics of interest among managers of medicines control laboratory in Europe, Australia, and Canada.
Pan American Health Organization (PAHO)	PAHO has requested that the HPFB undergo a WHO/PAHO regulatory assessment as part of the PAHO Directing Council resolution on strengthening National Regulatory Authorities. This would allow Health Canada to serve as a model for other authorities within the hemisphere and improve its own regulatory programs (BGTD's vaccines program has already been qualified).
Pan American Network for Drug Regulatory Harmonization (PANDRH) Cooperation	This network supports the processes of pharmaceutical regulatory harmonization in the Americas. TPD participates on the Steering Committee and BGTD participates in two PANDRH working groups (biotechnological products and vaccines), which have produced international guidance on these topics.
Pharmaceutical Inspection Cooperation Scheme (PIC/S) and PIC/S-MRA Special collaboration	PIC/S has multiple objectives, including: facilitate the networking and cooperation between participating Regulatory Authorities; maintain mutual confidence; promote quality assurance of inspections; exchange information and experience in the field of GMP and related areas; coordinate mutual training of GMP inspectors; and improve and harmonize technical standards and procedures related to GMP with a view to contributing to global harmonization. HPFB participates in various PIC/S working groups. Health Canada is also currently involved in a special collaboration to assess the feasibility of a closer collaboration with the Pharmaceutical Inspection Convention/Scheme (PIC/S) in a joint evaluation program for countries that want to join the MRA program and PIC/Ss. This fiscal year, some PIC/Ss representatives are working with one MRA Officer with the VMD authority as a pilot assessment of this eventual collaboration. The team of evaluators is composed of the Czech Republic as a lead, as well as Switzerland and Health Canada.
International Organization for Standardization	HPFB participated in the development of ISO - TC215, the purpose of which is to enable open and free sharing of international safety data, promote, interoperability between independent systems, and enable compatibility and consistency for health information and data. MHPD participates in the TC215, Health Informatics Working Group 6 - Pharmacy and medicines business.
Four-way pharmacovigilance teleconference	HPFB participates in a four-way pharmacovigilance teleconference, which began in 1998 and consists of the MHPD/HPFB, Centre for Drug Evaluation and Research/FDA, Adverse Drug Reaction Unit/TGA, Medsafe/New Zealand, and Health Sciences Authority/Singapore. The teleconference acts as a forum for exchange and discussion of health product vigilance and promotes coordination of pharmacovigilance efforts between jurisdictions by creating awareness of emerging safety issues in each regulatory authority. The meeting is held every two months, with the chair rotating between agencies.
Cooperation with WHO on biological standardization and evaluation of biologics	BGTD cooperated with the WHO on biological standardization and evolution of biologics, including work in key areas of vaccines, and HIV/AIDS vaccines. Activities include provision of technical advice and assistance in to develop biologic regulatory standards, provision of assistance to the WHO pre-qualification program, and implementation of WHO guidelines into regulatory practice.
WHO International Drug Monitoring Programme	Intended to enhance collaboration between regulatory agencies through discussion and information sharing. The Forums allow Health Canada to share its pharmacovigilance expertise to help countries developing pharmacovigilance programs. As part of their participation in the Forum, Health Canada uploads adverse reaction reports into the WHO Vigibase database.
WHO CIOMS	CIOMS aims at improving the methodology and the standardisation in the field of biomedical science (i.e., risk minimization). The reports produced by CIOMS have become the basis of several international guidance documents related to health product review and surveillance. MHPD participates in several working groups.
WHO Pre-Qualification Programme for Pharmaceuticals	This program assesses pharmaceutical applications submitted to the WHO's expressions of interest. These assessments ensure the quality of medicines procured by UN funding agencies to be sent to eligible countries, and contribute to the UN's goal of addressing widespread diseases in countries with limited access to quality medicines. Up to six review sessions are conducted each year.
International Medical Products Anti-Counterfeiting Task Force (IMPACT)	IMPACT works to determine appropriate actions to take in response to the availability of compromised medical products
ExL Pharma Conference on Proactive GCP Compliance	This three-day conference in Arlington, Virginia provided an opportunity to educate international stakeholders about Canadian regulatory requirements for clinical trials and network with international regulators.

Recall data

To derive the data on product recalls, PRA compiled information on recalls from the online Drug Recall Listing. The analysis included data up to and including August 23, 2012. A table listing all of the recalls identified in this way was circulated to TPD and BGTD with a request to identify those pertaining to human drugs and biologics, respectively. The analysis excluded 181 of the originally identified recalls for the following reasons: 106 recalls involved products that the TPD could not identify, often because the listing had no market authorization number; 40 recalls involved natural health products that contain pharmaceutical ingredients; and 35 recalls involved products that could not be definitively identified by the TPD as either a pharmaceutical or a natural health product. A total of 666 recall notices for human drugs and biologics were included in the analysis. Given the steps involved in completing this analysis, the analysis was not updated to include the entire 2012 year.

Table 18: Drug recall listings, by year and hazard classification
Year*	Type I			Type II			Type III			Other/unspecified**			Total
Year*	Total	Biol.	Pharm.	Total	Biol.	Pharm.	Total	Biol.	Pharm.	Total	Biol.	Pharm.	Total	Biol.	Pharm.
2005	11	0	11	12	0	12	33	2	31	7	0	7	63	2	61
2006	11	1	10	53	0	53	17	0	17	4	0	4	85	1	84
2007	4	0	4	10	3	7	15	0	15	3	0	3	32	3	29
2008	13	5	8	37	1	36	20	0	20	1	0	1	71	6	65
2009	5	0	5	57	1	56	29	1	28	0	0	0	91	2	89
2010	7	0	7	63	6	57	34	2	32	6	0	6	110	8	102
2011	17	0	17	71	4	67	52	2	50	0	0	0	140	6	134
2012	3	0	3	51	3	48	19	0	19	0	0	0	73	3	70
Unspecified	0	0	0	1	0	1	0	0	0	0	0	0	1	0	1
Total	71	6	65	355	18	337	219	7	212	21	0	21	666	31	635
Source: (Inspectorate, 2012b). Data for 2012 are current to August 23, 2012. Notes: Table excludes the following cases: 106 recalls involving products that the TPD could not identify, often because the listing had no market authorization number; 40 recalls involving NHPs that contain pharmaceutical ingredients; and 35 recalls involving products that could not be definitively identified by the TPD as a pharmaceutical or NHP. A recall is included in a given year if the recall start date occurs in that year *This field includes cases that were categorized by the Inspectorate as: N/A, Unacceptable Risk to Health, a market withdrawal order from Health Canada, or were blank.

Table 19: Serious adverse reaction reports as a proportion of all reports (public dataset)
Year	Not serious		Serious		Total
Year	#	%	#	%	Total
1999	2,628	47.0%	2,960	53.0%	5,588
2000	3,251	45.7%	3,862	54.3%	7,113
2001	1,976	27.2%	5,292	72.8%	7,268
2002	2,561	30.5%	5,828	69.5%	8,389
2003	2,605	29.0%	6,392	71.0%	8,997
2004	3,184	31.6%	6,904	68.4%	10,088
2005	3,134	30.5%	7,147	69.5%	10,281
2006	3,454	33.0%	7,007	67.0%	10,461
2007	4,169	33.6%	8,227	66.4%	12,396
2008	4,918	31.6%	10,646	68.4%	15,564
2009	5,210	28.0%	13,414	72.0%	18,624
2010	6,604	29.7%	15,631	70.3%	22,235
2011	8,064	28.1%	20,595	71.9%	28,659
2012	2,286	26.6%	6,313	73.4%	8,599
Total	54,044	31.0%	120,218	69.0%	174,262
Source: (MHPD, 2012a) Data for 2012 are current to June 30, 2012.

Table 20: Drugs inactivated by manufacturer for safety
Brand name	Active ingredient	Schedule
Anzemet/Anzemet Injection	Dolestron mesylate	F
Apo-Phenylbutazone	Phenylbutazone	F
Apo-Thioridazine/Dom Thioridazine/Novo-Rizadine/PMS-Thioridazine/Thioridazine	Thioridazine hydrochloride	F
Apo-Sibutramine/Meridia	Sibutramine hydrochloride	F
BCI Pravastatin/Riva-Pravastatin	Pravastatin sodium	F
Bextra	Valdecoxib	F
Climacteron Injection	Estradiol benzoate, estradiol dienanthate, testosterone enanthate	G (CDSA IV)
Darvon-N	Destropropoxyphene	Narcotic (CDSA I)
Depakene	Valproic acid	F
Fluotic	Sodium fluoride	F
Fucidin	Fusidate sodium	F
Levo-T	Levothyroxine sodium	F
Palladone XL	Hydromorphone hydrochloride	Narcotic (CDSA I)
Permax/Shire-Pergolide	Pergolide mesylate	F
Prexige	Lumiracoxib	F
Raptiva	Efalizumab	D, F
Thelin	Sitaxsetan sodium	F
Vioxx	Rofecoxib	F
Xigris	Drotrecogin Alfa	D, F
Zelnorm	Tegaserod maleaste	F
Source: (Health Canada, 2013)

List of references for Appendix C - Supplementary Data Tables

BGTD. (2010). Drug submission performance annual report (2010).

Decima Research. (2003). Public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada.

Decima Research. (2006). Final report: 2006 General public opinion survey on key issues pertaining to post-market surveillance of marketed health products in Canada.

GoC. (2012). Proceedings: SSCSAST, Issue No. 16 (May 9-10, 2012).

Health Canada. (2005). Report of the "Adderall XR New Drug Committee." Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/ndca_rep_cnma_rap_2005-08-25-eng.pdf

Health Canada. (2006). Metabolic and Endocrine Therapies TOR. Retrieved April 4, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/sacmet_tor_ccstme_att-eng.pdf

Health Canada. (2007). Expert Advisory Panel on the Special Access Programme: Terms of Reference.

Health Canada. (2011a). Completed or Dormant Scientific Advisory Bodies (2011). Retrieved March 19, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-com/sab_ocs_comm_list-eng.php

Health Canada. (2011b). Performance measurement and evaluation plan for the proposed fees in respect of Human Drugs and Medical Devices Regulations (version 8).

Health Canada. (2011c). Scientific Advisory Panel on Bioequivalence Requirement for Gender-Specific Drug Products - Terms of Reference. Retrieved October 4, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/gender-sexe/sapgsdp_tor_gcsmss_att-eng.pdf

Health Canada. (2011d). Scientific Advisory Panel on Bioequivalence Requirements for Gender-Specific Drug Products (SAP-GSDP): Record of proceedings 2011-06-02. Retrieved September 28, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/gender-sexe/sapgsdp-rop_gcsmss-crd_2011-06-02_ati-eng.pdf

Health Canada. (2012a). Completed or Dormant Scientific Advisory Bodies (2012). Retrieved October 18, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-com/sab_ocs_comm_list-eng.php

Health Canada. (2012b). Paediatric Expert Advisory Committee (PEAC) - Health Canada. resource list. Retrieved October 4, 2012, from http://www.hc-sc.gc.ca/ahc-asc/branch-dirgen/hpfb-dgpsa/opi-bip/peac-cceip/index-eng.php

Health Canada. (2012c). Scientific/Expert Advisory Panels - Drugs and Health Products - Health Canada. Retrieved October 18, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-consult/sab_ocs_pan_gr_list-eng.php

Health Canada. (2013). Drugs removed from the market for safety reasons, 2004-2012, data extract.

Inspectorate. (2004). A2.10 PMRC - Drug Inspections performance report (2003-04, year-end).

Inspectorate. (2006). A2.10 PMRC - Drug Inspections performance report (2005-06, year-end) (Final, v2).

Inspectorate. (2007). Inspectorate performance report (2006-07, year-end).

Inspectorate. (2010a). B1.5 Regional Border Integrity Unit performance report (2009-10) (Excel Spreadsheet).

Inspectorate. (2010b). Inspectorate performance report (2009-10).

Inspectorate. (2011). B1.5 Regional Border Integrity Unit performance report (2010-11) (Excel Spreadsheet).

Inspectorate. (2012a). B1.5 Regional Border Integrity Unit performance report (2011-2012)(Excel Spreadsheet).

Inspectorate. (2012b). Drug and medical device recall listing (August 2012). Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/compli-conform/recall-retrait/_list/index-eng.php

MHPD. (2006). Terms of reference of the Expert Advisory Committee on the Vigilance of Health Products (EAC-VHP).

MHPD. (2008). Meeting summary report - Expert Advisory Committee on the Vigilance of Health Products (EAC-VHP) - February 21-22, 2008.

MHPD. (2011). Meeting summary report - Expert Advisory Committee on the Vigilance of Health Products (EAC-VHP) - May 31, June 1-2, 2011.

MHPD. (2012a). Adverse reaction database: Canada Vigilance adverse reaction online database [2012]. Retrieved August 27, 2012, from http://www.hc-sc.gc.ca/dhp-mps/medeff/databasdon/index-eng.php

MHPD. (2012b). MHPD performance standards.

Nanos Research. (2010). Results of drug safety survey of Canadians.

Phoenix Strategic Perspectives Inc. (2011). Canadians' Awareness of Health and Safety Issues Related to Consumer Products (HC POR 10-09).

TPD. (2003). Scientific Advisory Committee on Human Reproductive Therapies (SAC-HRT) Terms of Reference.

TPD. (2008a). Annual Drug Submission Performance Report - TPD - 2008.

TPD. (2008b). Drug submission performance annual report (2008).

TPD. (2010a). Annual Drug Submission Performance Report - TPD - 2010. Retrieved August 31, 2012, from http://www.jptcp.com/files/tpd_dpt_annual_annuel_10_eng.pdf

TPD. (2010b). Scientific Advisory Committee on Oncology Therapies: Terms of Reference. Retrieved September 26, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/onco/sacot_tor_ccsto_att-eng.pdf

TPD. (2010c). Scientific Advisory Committee on Respiratory and Allergy Therapies (SAC-RAT): Terms of reference. Retrieved September 28, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/resp_allerg/sacrat_tor_ccstrta_att-eng.pdf

TPD. (2010d). Scientific Advisory Panel on Bioequivalence Requirements for Modified Release Dosage Forms (SAP-MRDF): Draft terms of reference. Retrieved September 28, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/mrdf-fplm/mrdf_tor_dr_fplm_att_eb-eng.pdf

TPD. (2010e). Scientific Advisory Panel on Bioequivalence Requirements for Modified Release Dosage Forms (SAP-MRDF): Record of proceedings 2010-06-11. Retrieved September 28, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/mrdf-fplm/sapmrdf_rop_gscfplm_crd_2010-06-11-eng.pdf

TPD. (2011a). Notice: Nomination call for members - Scientific Advisory Committee on Pharmaceutical Sciences and Clinical Pharmacology (SAC-PSCP). Retrieved September 27, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/pharma/sacpscp_ccssppc_inv-eng.pdf

TPD. (2011b). Oncology Therapies - Scientific/Expert Advisory Committees. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-com/onco/index-eng.php

TPD. (2011c). Scientific Advisory Committee on Nonprescription Drugs (SAC-NPD): Terms of reference. Retrieved September 26, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/nonpres/sacnpd_tor_ccsmvl_att-eng.pdf

TPD. (2011d). Scientific Advisory Committee on Pharmaceutical Sciences and Clinical Pharmacology (SAC-PSCP): Draft terms of reference. Retrieved September 27, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/pharma/sacpscp_tor_ccssppc_att-eng.pdf

TPD. (2011e). Scientific Advisory Panel on Opioid Analgesic Abuse (SAP-OAA): Record of Proceedings 2011-03-29. Retrieved September 26, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/opioid/sapoaa_rop_gcsaao_crd_2011-03-29-eng.pdf

TPD. (2011f). Scientific Advisory Panel on Opioid Analgesic Abuse (SAP-OAA): Terms of reference. Retrieved September 26, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-consult/opioid/sapoaa_tor_gcsaao_att-eng.pdf

TPD. (2012a). Nonprescription Drugs - Scientific/Expert Advisory Committees. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-com/nonpres/index-eng.php

TPD. (2012b). Respiratory and Allergy Therapies - Scientific/Expert Advisory Committees. Retrieved August 21, 2012, from http://www.hc-sc.gc.ca/dhp-mps/prodpharma/activit/sci-com/resp_allerg/index-eng.php

TPD. (2012c). Scientific Advisory Committee on Pharmaceutical Sciences and Clinical Pharmacology (SAC-PSCP) Announcement of meeting and invitation to participate - 2012-06-26. Retrieved September 27, 2012, from http://www.hc-sc.gc.ca/dhp-mps/alt_formats/pdf/prodpharma/activit/sci-com/pharma/sacpscp_ccssppc_anno-eng.pdf

List of Acronyms

ADM: Assistant Deputy Minister
ANDS: Abbreviated New Drug Submission
API: Active pharmaceutical ingredient
BGTD: Biologics and Genetic Therapies Directorate
BMP: Bilateral Meeting Program
CBSA: Canada Border Services Agency
CCSA: Canadian Centre on Substance Abuse
CDC: Centres for Disease Control
CDSA: Controlled Drugs and Substances Act
CFIA: Canadian Food Inspection Agency
cGMP: Current Good Manufacturing Practices
CIHI: Canadian Institute for Health Information
CIHR: Canadian Institutes of Health Research
CPAB: Communications and Public Affairs Branch
CPSP: Canadian Paediatric Surveillance Program
CRI: Cost Recovery Initiative
CTA: Clinical Trial Application
CTA-A: Clinical Trial Application Amendment
CTD: Common Technical Document
CTO: Cells, Tissues and Organs
DEL: Drug Establishment License
DHPL: Dear Health Professional Letter
DIN: Drug Identification Number
DPD: Drug Product Database
DPR: Departmental Performance Report
DSEN: Drug Safety and Effectiveness Network
EAC: Expert Advisory Committee
EAP: Expert Advisory Panel
EBP: Employee Benefit Plan
eCTD: electronic Common Technical Document
EMA/EMEA: European Medicines Agency
EPAR: European Public Assessment Report
ER/LA: Extended release/long-acting
EU: European Union
EUND: Extraordinary Use New Drug
FCSAP: Food and Consumer Safety Action Plan
FDA: Food and Drug Administration (United States)
FTE: Full-time equivalent
GCP: Good Clinical Practices
GGP: Good Guidance Practices
GMP: Good Manufacturing Practices
GoC: Government of Canada
GRP: Good Review Practices
GVP: Good Pharmacovigilance Practices
HDP: Human Drugs Program
HECSB: Healthy Environments and Consumer Safety Branch
HESA: House of Commons Standing Committee on Health
HPFB: Health Products and Food Branch
ICH: International Conference on Harmonization on Technical Requirements for Registration of Pharmaceuticals for Human Use
IOM: Institute of Medicine
LA/SA: Look alike/sound alike
MAH: Market authorization holder
MHPD: Marketed Health Products Directorate
MHRA: Medicines and Healthcare Products Regulatory Agency
MOU: Memorandum of Understanding
MRA: Mutual Recognition Agreement
NAS: New active substance
NC: Non-compliant
NDS: New Drug Submission
NIH: National Institutes of Health (US)
NMI: Non-medicinal ingredient
NOC: Notice of Compliance
NOC/c: Notice of Compliance with Conditions
NOL: No Objection Letter
OAG: Office of the Auditor General of Canada
OCS: Office of Controlled Substances
ORS: Office of Research and Surveillance
OTC: Over the counter
PAA: Program Activity Architecture
PAAT: Post-authorization activity table
PBRER: Periodic Benefit Risk Evaluation Report
PDP: Pharmaceutical Drugs Program
PEAC: Paediatric Expert Advisory Committee
PHAC: Public Health Agency of Canada
PIC/S: Pharmaceutical Inspection Cooperation Scheme
PLR: Physician Label Rule
PMRC: Post Market Reporting Compliance
PMS: Performance measurement strategy
PPIAD: Policy, Planning and International Affairs Directorate
PSUR: Periodic Safety Update Report
PVP: Pharmacovigilance plans
Q&A: question and answer
RAPB: Regions and Programs Bureau
RCC: Regulatory Cooperation Council
REB: Research ethics board
REMS: Risk Mitigation and Evaluation Strategy
RMOD: Resource Management and Operations Directorate
RMP: Risk management plan
SAC: Scientific Advisory Committee
SANDS: Supplemental Abbreviated New Drug Submission
SAP: Special Access Program
SBD: Summary Basis of Decision
SEB: Subsequent entry biologic
SIMS: Stakeholder Information Management System
SNDS: Supplemental New Drug Submission
SOP: Standard Operating Procedure
SSCSAST: Senate Standing Committee on Social Affairs, Science and Technology
TAS: Therapeutic Access Strategy
TGA: Therapeutic Goods Administration
TPD: Therapeutic Products Directorate
TPSI: Therapeutic Product Safety Initiative
WHO: World Health Organization

Footnotes

Footnote 1

This MRAP has been developed by participating organizations [i.e., Therapeutic Products Directorate (TPD), Marketed Health Products Directorate (MHPD), Biologics and Genetic Therapies Directorate (BGTD), HPFB Inspectorate (Inspectorate), Policy, Planning, and International Affairs Directorate (PPIAD) of the Health Products and Food Branch (HPFB); the Regions and Programs Bureau (RAPB)] in response to the recommendations made in the Evaluation of the Medical Devices Program. All responsibility for reporting on key activities rests at the Director General level.

Return to footnote1 Referrer

Footnote 2

It is important to note that the GCP, GMP, and GVP inspection programs encompass both pharmaceutical and biologic drug products.

Return to footnote2 Referrer

Footnote 3

To obtain a copy of the Logic Model graphic please use the following e-mail "Evaluation Reports HC - Rapports Evaluation@hc-sc.gc.ca".

Return to footnote3 Referrer

Footnote 4

Calculated out of 365 valid email addresses.

Return to footnote4 Referrer

Footnote 5

Calculated out of 622 valid email addresses.

Return to footnote5 Referrer

Footnote 6

The literature does not always distinguish between pharmaceutical drugs and biologic drugs. Therefore, the remainder of this discussion describes general trends and issues, and makes distinctions between pharmaceutical and biologic drugs, where possible.

Return to footnote6 Referrer

Footnote 7

Of these, two were biologics (Schedules D and F), two were narcotics (CDSA I), one was Schedule G (CDSA IV), and the remainder were prescription drugs (Schedule F).

Return to footnote7 Referrer

Footnote 8

Heparin is currently regulated as a biologic, but at the time was regulated as a pharmaceutical.

Return to footnote8 Referrer

Footnote 9

In addition, compounding "can involve raw materials or the alteration of the form and strength of commercially available products," and "can include reformulation to allow for a novel drug delivery," but excludes "mixing, reconstituting, or any other manipulation that is performed in accordance with the directions for use on an approved drug's labelling material" (Inspectorate, 2009b, p. 7). In Canada, drug manufacturing falls under federal jurisdiction, while drug compounding falls primarily under provincial/territorial jurisdiction; distinctions between compounding and manufacturing are made on a case-by-case basis (Inspectorate, 2009b).

Return to footnote9 Referrer

Footnote 10

More specifically, in 2006, the OAG recommended establishing baseline information for performance measurement, setting user fees based on clear and measurable service standards, and reviewing core funding to determine if it is sufficient (OAG, 2006).

Return to footnote10 Referrer

Footnote 11

A cost sharing ratio of 50% means that 50% of program costs are covered by user fees.

Return to footnote11 Referrer

Footnote 12

Prior to 2009 there is no evidence of regular meetings involving senior management within the partner directorates to provide overall strategic direction to the HDP. This may be because Health Canada's activities related to human drugs have not historically been conceptualized as a "program." This designation is relatively recent and appears to have occurred partly in response to the requirement for evaluation and the restructuring of Health Canada's PAA in 2007-2008.

Return to footnote12 Referrer

Footnote 13

To streamline the presentation of information, this organizational scheme does not align precisely with the program logic model. Nevertheless, all relevant program activities are covered.

Return to footnote13 Referrer

Footnote 14

According to Health Canada, few new pharmaceutical molecules were successful between 2005 and 2011, which contributed to the decrease in clinical trials for pharmaceuticals and the concurrent increase in biologic CTAs. Health Canada also indicated that there was a large increase in CTA-As in the same time period.

Return to footnote14 Referrer

Footnote 15

The clinical trial inspection program is described in detail in Section 5.4.5.

Return to footnote15 Referrer

Footnote 16

Health Canada promised to enhance public access to clinical trial information in the 2007 Blueprint for Renewal, and since then has encouraged sponsors to register their clinical trials of therapeutic products within 21 days of the trial's onset with one of three publicly-accessible registries of the World Health Organization's (WHO) Register Network. Both the OAG and the Senate Standing Committee on Social Affairs, Science and Technology (SSCSAST) have criticized the Department in recent years for not making more clinical trial information available to Canadians, despite its commitment to do so (OAG, 2011; SSCSAST, 2012).

Return to footnote16 Referrer

Footnote 17

Health Canada has taken some steps to encourage sponsors to conduct paediatric studies. In 2006, regulatory incentives for paediatric studies came into force, providing a six-month data protection extension to sponsors submitting paediatric clinical trial information as part of an innovative drug submission (HPFB, 2007a, p. 27). Health Canada also recently released a discussion paper on the use of international paediatric information, which laid out three possible approaches to addressing Health Canada's current lack of a regulatory mechanism or policy to encourage or require drug sponsors to submit new paediatric clinical trial data that are generated internationally (Health Canada, 2012m).

Return to footnote17 Referrer

Footnote 18

A DIN is an eight digit numerical code that identifies a drug product's brand name, manufacturer, ingredients, strength, pharmaceutical form, and route of administration. With the exception of radiopharmaceuticals, every drug product marketed in Canada (under the Food and Drugs Act and Regulations) has a DIN.

Return to footnote18 Referrer

Footnote 19

The User Fees Act provides that "if a regulating authority's performance in a particular fiscal year in respect of a user fee does not meet the standards established by it for that fiscal year by a percentage greater than ten per cent, the user fee shall be reduced by a percentage equivalent to the unachieved performance, to a maximum of fifty per cent of the user fee" (GoC, 2004, sec. 5.1). The penalty is applied when the average performance is more than 110% of the associated time target.

Return to footnote19 Referrer

Footnote 20

The implications of this initiative for Health Canada's current electronic submission infrastructure, which has been a work-in-progress for several years, are unknown.

Return to footnote20 Referrer

Footnote 21

The Use of Foreign Reviews project is one aspect of a larger Use of Foreign Regulatory Information project that also includes a Foreign Regulatory Scientific Committee, scientific advice, parallel reviews, joint reviews, and inter-regulatory discussion groups (HPFB, 2012b). A Foreign Regulatory Scientific Committee is in the pilot phase, and Health Canada also engages real-time scientific advice and discussions with other regulatory agencies (FDA, EMA) through Oncology, Paediatric, and Blood "clusters."

Return to footnote21 Referrer

Footnote 22

Data on the percentage of review cycle completions within targetis presented. This indicator was selected in consultation with Health Canada and includes all submission classes except Labelling only within these submission types and all review cycles. This is not the only possible way to analyse review performance. The OAG's 2011 report considered only Review 1, as do Health Canada's departmental performance reports (DPRs).

Return to footnote22 Referrer

Footnote 23

Examples include drug-drug interactions and adverse reactions with long latency or a high background rate in the population.

Return to footnote23 Referrer

Footnote 24

These data were extracted by Health Canada from the Canada Vigilance Database.

Return to footnote24 Referrer

Footnote 25

The outreach and education activities target consumers as well as health care professionals.

Return to footnote25 Referrer

Footnote 26

A more detailed discussion of Health Canada's service standards and performance targets for post-market activities is found later in this section.

Return to footnote26 Referrer

Footnote 27

In 2011, Health Canada published its procedure for the public release of information obtained from adverse reaction reports (MHPD, 2011b). The procedure specifies that personal and confidential third-party information is protected and that requests for information not available on the website or in standard summary format, as well as information from clinical trial adverse reaction reports, must be made through the Access to Information and Privacy Division of Health Canada.

Return to footnote27 Referrer

Footnote 28

For human drugs, the time period covered by the analysis was 2006 to 2012. For biologics, the time period was 2008 to 2012.

Return to footnote28 Referrer

Footnote 29

BGTD currently requests RMPs to be submitted as part of applications for subsequent entry biologics, or SEBs.

Return to footnote29 Referrer

Footnote 30

The US FDA recently announced that it would accept periodic safety reports in the PBRER format(FDA, 2013c).

Return to footnote30 Referrer

Footnote 31

Health Canada originally published its Notice of Intent to adopt ICH Q7 and proceed with developing a regulatory framework for APIs in 2002 (Inspectorate, 2002). Health Canada representatives explained that the 10-year delay between the initial announcement of intent and the regulatory amendment was due to the complexity of the regulation and changes in HPFB priorities with respect to regulatory development over time.

Return to footnote31 Referrer

Footnote 32

The FDA refers to GMP as Current Good Manufacturing Practice, or cGMP. As in Canada, inspections of manufacturing sites occur on a two-year cycle, but because the FDA "lacks the resources to audit every aspect of cGMP during every inspection visit," the depth of the inspection is determined by the firm's compliance history, the technology employed and the characteristics of the products manufactured (FDA, 2002). An abbreviated inspection may be done for firms with a satisfactory past history of cGMP compliance, while a full inspection may be done when little or no information is known about a firm's cGMP compliance, such as for new firms, or firms where there is doubt about compliance, based on their history of compliance/recidivism (FDA, 2002).

Return to footnote32 Referrer

Footnote 33

According to information provided by Health Canada, product monographs are posted on the DPD generally 24 hours after an NOC is granted. However, they are visible to the public only once the company notifies Health Canada that it intends to market the product in Canada. A company has 30 days to notify Health Canada once they market a product. Once Health Canada receives notification, the product monograph is made visible to the public. This is typically done within 24 to 48 hours of receipt of notification.

Return to footnote33 Referrer

Footnote 34

Look-alike sound-alike (LA/SA) health product names are "names of different health products that have orthographic similarities and/or similar phonetics (i.e. similar when written or spoken)" (HPFB, 2005a, p. 1). LA/SA health products create risks due to the possibility of confusion and errors in prescribing, dispensing, or administering the product. The proposed regulatory amendment would require manufacturers to submit an assessment of the drug's brand name, showing it is not likely to be mistaken for another drug (GoC, 2013).

Return to footnote34 Referrer

Footnote 35

This information includes product name, boxed warning, recent major changes, indications and usage, dosage and administration, contraindications, warnings and precautions, adverse reactions, drug interactions, and use in specific populations.

Return to footnote35 Referrer

Footnote 36

Section C.01.004(c)(iii) of the Food and Drug Regulations requires that that the inner and outer labels show "adequate directions for use of the drug" (Health Canada, 2010a), which is interpreted to include warnings or cautions specifically required by the Regulation. In some cases, a specific warning or precaution will be required for an individual product or product category, but the warning may not, in itself, be a regulated statement. Rather, the warnings are considered an expression of adequate directions for use based on medical or pharmacological grounds (Health Canada, 2010a).

Return to footnote36 Referrer

Footnote 37

Until recently, all drug recalls, regardless of type, were posted on the Drug Recall Listing, and some resulted in a risk communication posted on MedEffect. Health Canada guidance states that "[n]otices/advisories from companies may be posted on MedEffect when Health Canada deems that it is important to have the detailed safety information available for everybody who may need to know it" (Health Canada, 2008b, p. 29). Attempts by the evaluation to quantify the Type I recalls in the Drug Recall Listing that resulted in a MedEffect communication were ultimately abandoned, due to methodological difficulties

Return to footnote37 Referrer

Footnote 38

These initiatives are in response to a Federal Court ruling that all federal government sites must meet international standards for accessibility, as well as a directive from Treasury Board to reduce the number of pages on the website by half by July 31, 2013 (Health Canada, 2013c).

Return to footnote38 Referrer

Footnote 39

That being said, a review of the postings shows that some communications describe recalls and use the word "recall" in the title, but nonetheless are classified as "advisories" instead of "recalls". The criteria or rationale used by Health Canada for classifying risk communications as either recalls or advisories is unknown to the evaluation.

Return to footnote39 Referrer

Footnote 40

Health Canada does not pay participants' travel expenses.

Return to footnote40 Referrer

Footnote 41

The studies examined public and medical professionals' perceptions of drug safety in general, and Health Canada's drug safety information in particular, assessing respondents' information-seeking practices, awareness of information sources, and satisfaction with these information sources.

Return to footnote41 Referrer

Footnote 42

According to the Performance Measurement and Evaluation Plan for the project, evaluation methods will include analysis of the timeliness and readability of Health Canada's risk communications; website analytics; comparisons of the degree to which Health Canada's approach (timeliness and messaging) is consistent with that of other regulators; a pop-up survey of users of the MedEffect website; analyses of prescribing trends and adverse reaction reporting trends; focus groups with users to explore attitudes, opinions, and satisfaction levels; and a survey of Canadians to examine similar issues (Health Canada, 2012i).

Return to footnote42 Referrer

Footnote 43

In the US, Olson (2002) found that a one-month reduction in a drug's approval time through the FDA is associated with a 1% increase in expected reports of adverse drug reaction hospitalizations and a 2% increase in expected reports of adverse drug reaction deaths (p. 27). In Canada, Lexchin (2012) found that 23.7% (84 out of 434) of the new active substances (NASs) approved by Health Canada between January 1, 1995, and December 31, 2010, had safety issues, defined as having been withdrawn from the market and/or receiving a serious safety warning. This included 68 products with serious safety warnings; 9 with serious safety warnings which were then withdrawn from the market; and 7 that were withdrawn without previous safety warnings. The study estimated that NASs receiving a standard review had a probability of 19.8% of experiencing a safety issue, versus a probability of 34.2% for NASs receiving a priority review. For those NASs that were not major therapeutic advances (n=81), the probability of experiencing a safety issue was 36.0% (Lexchin, 2012, p. E1).

Return to footnote43 Referrer

Footnote 44

The FDA also indicated that the development of abuse-deterrent formulations is a "public health priority for the FDA," and in July 2012, approved a class-wide REMS for all extended release (ER)/long acting(LA) opioids, the specific goal of which is to reduce "serious adverse outcomes resulting from inappropriate prescribing, misuse and abuse" of ER/LA opioid analgesics (FDA, 2012c). In contrast to the FDA, in November 2012, Health Canada authorized six manufacturers to produce generic oxycodone for the Canadian market in November 2012, indicating that the drug "is considered safe and effective under the recommended conditions of use" (Health Canada, 2012k). Health Canada also asked manufacturers to submit RMPs that outline their proposed strategies to monitor, respond to, and educate health care professionals and the public on known and potential risks; and announced additional licensing conditions specifically on controlled release-oxycodone products.

Return to footnote44 Referrer

Footnote 45

As previously described, a "compliant" rating does not mean that there were no noted observations of deviation from GMP requirements; in fact, firms that are rated "compliant" often have observations and are required to implement corrective actions.

Return to footnote45 Referrer

Footnote 46

Although the evaluation framework developed by Health Canada refers to "risk-benefit analysis," many Health Canada documents refer to risk analysis, risk-based analysis, and risk-based approaches. These are distinct concepts, but the discussion here treats them as largely similar.

Return to footnote46 Referrer

Footnote 47

The NOC was subsequently reinstated.

Return to footnote47 Referrer

Footnote 48

There are three main stages involved in the process of developing regulations in Canada: preliminary research and decision to regulate; pre-publication of a proposed regulation; and final approval, publication, and registration of the proposed regulation. While Stages Two and Three combined typically require between 6 to 24 months to complete, there are no timelines associated with the preliminary research stage.

Return to footnote48 Referrer

Footnote 49

A pre- and post-April 2007 comparison could not be performed for biologics, since all biologics signal assessments with an assigned date recorded were assigned after April 2007, specifically between November 2008 and October 2012.

Return to footnote49 Referrer

Footnote 50

The evaluation examined all human drugs signal assessments completed in 2010, 2011, and 2012 that produced a recommendation for a risk communication (n=38). The evaluation originally planned to analyse whether, and the timeliness with which, the recommendations stemming from all completed signal assessments were implemented by Health Canada. However, the data necessary to complete this analysis were unavailable, as internal Health Canada resources would have been required to compile it from a variety of sources. The evaluation therefore restricted its analysis to signal assessments completed in 2010, 2011, and 2012 that recommended the issuance of a risk communication, since the information necessary to support this analysis could be compiled from public sources (i.e., the MedEffect website). A similar analysis could not be undertaken for biologics due to a limited number of completed signal assessments in this time period that produced a recommendation for a risk communication.

Return to footnote50 Referrer

Footnote 51

The evaluation was able to piece together a "full record" for only 14 of the 38 human drugs signal assessments that resulted in a risk communication. That is, it was able to match a signal detection record with confidence to a signal assessment record and to a posted risk communication for only 14 completed assessments. The evaluation was unable to complete a similar analysis for biologics due to a limited number of cases.

Return to footnote51 Referrer

Footnote 52

In one high-profile example, Health Canada updated the product labels for Propecia and Proscar to include a warning that sexual dysfunction could continue after discontinuation of treatment three years after regulatory authorities in Sweden, Italy and the United Kingdom (Collier, 2013).

Return to footnote52 Referrer

Footnote 53

Health Canada indicated that its current guidance on Clinical Trial Applications specifically targets academics and is intended to be more responsive to the needs of small scale rese.

Return to footnote53 Referrer

Footnote 54

According to data provided by HPFB, the number of FTEs allocated to the HDP within HPFB increased from 576.53 in 2009-2010 to 712.91 in 2011-2012.

Return to footnote54 Referrer

Page details

Date modified:: 2014-08-29

Evaluation of the Human Drugs Program 1999-2000 to 2011-2012

Table of Contents

Executive Summary

Findings

Relevance

Performance - program implementation

Clinical trials

Recommendation 1:

Recommendation 2:

Recommendation 3:

Recommendation 4:

Recommendation 5:

Performance - outcomes achieved

Recommendation 6:

Recommendation 7:

1.0 Introduction

1.1 Organization of the report

2.0 Profile of the Human Drugs Program

2.1 Roles and responsibilities of program partners

2.1.1 Therapeutic Products Directorate

2.1.2 Marketed Health Products Directorate

2.1.3 Inspectorate and RAPB

2.1.4 Other partners

2.2 Program activities and logic

2.3 Description of the Logic ModelFootnote 3

2.4 Resources

2.5 Program context

3.0 Evaluation methodology

3.1 Evaluation design and methods

3.2 Limitations of the methodology and mitigation strategies

4.0 Findings - relevance

4.1 Ongoing need

4.2 Alignment with federal priorities

4.3 Consistency with federal roles and responsibilities

5.0 Findings: governance and implementation

5.1 Program governance

5.2 Collaboration with external partners

5.3 Performance measurement

5.4 Program implementation

5.4.1 The drug regulatory process

5.4.2 Clinical trials

Strengthening the regulatory framework for clinical trials

5.4.3 Submission review and market authorization

Main types of drug submission

Review and market authorization

Post-market changes

Special Access Program

Improving the submission review and market authorization process

5.4.4 Post-market surveillance and monitoring

Enhancing adverse reaction reporting and analysis

Adverse reaction reporting in Canada for human drugs

Addressing under-reporting by health care professionals

Implementing electronic submission of adverse reaction reports

Enhancing analysis of adverse reaction reports

Enhancing signal detection and assessment

Implementing risk management and pharmacovigilance planning

Requirements for RMPs in Canada

Enhancing post-market safety reporting by manufacturers

Implementing service standards for post-market activities

Expanding partnerships and use of existing databases

5.4.5 Compliance and enforcement

Available compliance and enforcement measures

Implementing new authorities and tools for compliance and enforcement

Strengthening clinical trial inspections

Enhancing drug establishment licensing and GMP inspections

GMP compliance of domestic sites

GMP compliance of foreign sites

Other initiatives to strengthen compliance and enforcement

5.4.6 Communications and stakeholder engagement

Enhancing the transparency of the drug review process

Enhancing drug product labelling

Enhancing post-market safety and risk communications

Opening the regulatory system to stakeholder input and involvement

6.0 Findings - outcomes achieved

6.1 Stakeholder awareness and understanding

External key informant perceptions of Health Canada's communications and information

6.2 Industry awareness and understanding

6.3 Safety and effectiveness

Pre-market stage

Post-market stage

2.3 Description of the Logic Model^{Footnote 3}