Final Audit Report - Audit of the Use of the Standards-Based Approach in Regulations - Biologics and Genetic Therapies Directorate

September 2009

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Management Response and Action Plan

Table of Contents

• Executive Summary
• Introduction
  • Background
  • Objective
  • Scope and Approach
• Findings, Recommendations and Management Responses
  • Project Management
    • Decision to revise regulations
    • Decision to use a standards-based approach
    • Development of action plans
  • Measuring and reporting results and impacts
    • Development of a performance management framework
• Appendix 1 - Lines of Enquiry and Audit Criteria

Executive Summary

Health Canada's Biologics and Genetic Therapies Directorate, which is part of the Health Products and Food Branch, is the regulatory authority that regulates, among other things, blood and blood components (intended for transfusion); cells, tissues and organs for transplantation; viral and bacterial vaccines; biological drugs and genetic therapies. These regulations outline the legal requirements for areas such as, clinical trials, licensing, and product safety standards.

In recent years, the Directorate has adopted a standards-based regulatory framework for developing regulations for two of its biologics. While the standards-based approach had been used by Federal departments to develop regulations, the approach received a sizable support in 1993 when the House of Commons' Standing Committee on Finance issued a report on "Regulations and Competitiveness". In that report, the Committee indicated that when identifying and assessing alternative approaches to the regulatory process, regulatory authorities should review existing standards (Canadian, foreign, and international) to ascertain which ones might be suitable for use in the Canadian context.

The objective of this audit was to determine whether the Directorate efficiently managed and implemented the standards-based approach for Safety of Human Cells, Tissues and Organs for Transplantation; and the Safety and Quality of Human Blood and Blood Components. The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, and has examined sufficient, relevant evidence and obtained sufficient information and explanations to provide a reasonable level of assurance in support of the audit conclusion.

The audit focussed on the Directorate's project management activities for adopting and implementing the standards-based approach and was aimed at identifying lessons learned and making recommendations to improve the future use of this approach.

For example, one of the "lessons learned" in the early stages of both projects was that the Directorate would have benefited from profiling and assessing the risks associated with adopting and implementing the standards-based approach. A second lesson relates to the projects length of time. Given the complexity of the two biologics identified for this approach, it is not surprising that the projects took much longer to complete than first anticipated. As such, if during the planning process for any similar project in future, it becomes clear that the project will take several years to complete; this fact should signal the need to ensure that adequate measures are in place to mitigate any risks to health and safety in the interim.

Another lesson that arose in the "Blood Project" was in the area of roles and responsibilities. Initially the Directorate did not provide sufficient direction to Health Canada scientists who represented the Department at the Technical Committee. With such a large change management exercise, it is important to explain key aspects to employees who may be affected by them. When the Directorate recognized this issue, it took timely action by issuing clear guidance on the role of Health Canada's Technical Committee representatives and their obligation to convey the Department's views as the regulatory authority of Canada's blood supply. Furthermore, the Directorate gave presentations to all members of Technical Committees on the Department's role as regulator and how it would eventually be referencing part of the standards in the Regulations.

A final lesson relates to performance management. While the Directorate did begin to strengthen its performance management framework for Cells, Tissue and Organs as early as 2001, and began to develop an equivalent framework for Blood in 2004, at the time of the audit, the evaluations for both biologics had not yet been completed. Accordingly, the Directorate has not been able to provide management with adequate performance information.

Since none of the "lessons learned" had yet been incorporated into the Directorate's main policy tool for the standards-based approach,the audit makes one recommendation to review and update the Policy Development Process, to reflect all lessons identified (both from the audit report and other sources) as they relate to managing projects for use of the standards-based approach.

There is one recommendation in the report. Management has agreed with it and has provided a comprehensive action plan to address the recommendation.

Introduction

Background

Responsibility for regulating health products

One of Health Canada's core roles is that of a regulator. It carries out various legislative and statutory activities aimed at protecting Canadians' health and safety, and ensuring that products vital to our health are available. The Department's Biologics and Genetic Therapies Directorate (hereafter referred to as the Directorate), which is part of the Health Products and Food Branch (HPFB), is the regulatory authority that regulates, among other things, blood and blood products; cells, tissues and organs for transplantation; viral and bacterial vaccines; biological drugs and genetic therapies. The regulatory requirements for these products are stated in the Food and Drug Regulations. These regulations outline the legal requirements for areas such as, clinical trials, licensing, and product safety standards.

In 2008-09, the Directorate had approximately 350 employees, with a budget of approximately $37 million.

The two approaches to regulating

In developing and implementing regulations, regulators follow either the "traditional" or "standards-based" approach.

Under the traditional approach, the regulatory authority writes detailed product and manufacturing standards into the regulations themselves. It leads the development of the regulations, which tend to be prescriptive, by following a formal regulatory process. This process involves consulting with interested parties, but it does not necessarily involve trying to obtain consensus with respect to the regulations.

Under the standards-based approach, regulations are developed by using standards created by Standards Development Organizations. These are developed through Technical Committees whose membership is made up of experts and typically includes those who will be subject to the regulations (e.g., manufacturers and medical practitioners). The standards are written following an accredited process based on consensus. Once they are approved and published, the regulatory authority may reference or incorporate them in whole or in part in the regulations, making them mandatory by law. When this approach is followed, the government relies on the Standards Development Organization to periodically review and update the standards.

Regardless of the approach (traditional or standards-based), the regulatory authority is accountable for ensuring compliance with the regulations. In addition, policies, guidelines and procedures (referred to as "non-regulatory instruments") support the regulations. The regulations and non-regulatory instruments are collectively referred to as a regulatory framework.

While the standards-based approach had been used by Federal departments to develop regulations, the approach received a sizable support in 1993 when the House of Commons' Standing Committee on Finance issued a report on "Regulations and Competitiveness". In that report, the Committee indicated that when identifying and assessing alternative approaches to the regulatory process, regulatory authorities should review existing standards (Canadian, foreign, and international) to ascertain which ones might be suitable for use in the Canadian context. If an acceptable standard exists, then reference to it should be made when drafting the regulations. If no acceptable standard exists, regulatory authorities should arrange for the development of a standard through the National Standard System.

The National Standard System is Canada's network of people and organizations involved in developing, promoting and implementing standards. It is coordinated and overseen by the Standards Council of Canada, a Crown Corporation established in 1970 to foster and promote voluntary standardization in Canada.

The need for new regulations

In the past, the regulatory frameworks for biologics have been largely based on the traditional approach for developing and maintaining regulations. While this approach is a well established practice, the Directorate has indicated that it is not sufficiently flexible to accommodate today's scientific advances in molecular biology and manufacturing technology. As such, in recent years Health Canada has adopted a standards-based regulatory framework for two of its biologics that it considered best suited for this latest approach to developing regulations.

First, whole blood and blood components intended for transfusion has been regulated under the Food and Drugs Regulations since 1989. However, the regulations were difficult to follow, did not contain provisions specific to blood collection and components manufacturing, and were being applied only to the blood system operators (the Canadian Red Cross Society up to 1998).

The need to develop regulations covering blood collection became even more pressing in February 1995 when the Commission of Inquiry on the Blood System in Canada (also known as the "Krever Commission") issued its interim report.

The second area of biologics considered was Cells, Tissues and Organs for Transplantation. This area was not regulated by a specific section of the Food and Drug Regulations and like other countries, Canada needed to develop and apply a more comprehensive regulatory framework.

Development of standards by the Canadian Standard Association

Health Canada started to develop standards for Cells, Tissues and Organs for transplantation and blood in 1995 and 1996 respectively. The first versions of these standards were developed in-house by the Department with the support of expert working groups, comprised of external members. However, in order to comply with the National Standard System, the development of these standards was completed by Technical Committees created by the Canadian Standards Association (CSA), a non-for-profit, voluntary membership organization engaged in standards development and certification activities and accredited by the Standards Council of Canada.

The new regulations for Cell, Tissues, and Organs came into force in December 2007 and the Blood Regulations are scheduled for publication in the Canada Gazette in the Fall 2009.

Objective

The objective of the audit was to determine whether the Biologics and Genetic Therapies Directorate (BGTD), efficiently managed and implemented the standards-based approach for two regulatory frameworks (Blood and Blood Products and Cells, Tissues and Organs).

Scope and Approach

The audit was undertaken by the Audit and Accountability Bureau as per the Health Canada Risk-Based Audit Plan for 2008-2009 which was approved by the Departmental Audit Committee on April 3, 2008 and was conducted in accordance with the Internal Auditing Standards for the Government of Canada, and has examined sufficient, relevant evidence and obtained sufficient information and explanations to provide a reasonable level of assurance in support of the audit conclusion.

The audit focussed on the Directorate's project management for adopting and implementing the standards-based approach to develop two regulatory frameworks, Safety of Human Cells, Tissues and Organs for Transplantation (CTO Regulations); and the Safety and Quality of Human Blood and Blood Components (Blood Regulations).

The audit focussed on the management of the development of CTO and Blood Regulations. For this reason, audit criteria (see Appendix A) were derived from expectations found in the Treasury Board Project Management Policy. The audit also covered compliance with requirements stated in the Treasury Board Regulatory Policy.

The audit did not examine regulations currently being updated using the traditional approach (i.e. clinical trials, vaccines and radiopharmaceuticals). Furthermore, the audit did not attempt to determine whether the Directorate's decision to adopt the standards-based approach to developing these two regulations was appropriate.

Audit evidence was obtained from centres of the Biologics and Genetic Therapies Directorate and included interviews with management teams and staff members, literature review and document review.

Except for a study commissioned by the Standards Council of Canada in 1997, which included Health Canada among other federal departments, the use of the standards-based approach by the Department to develop regulations has never been audited internally. Thus, the audit was aimed at identifying lessons learned and making recommendations to improve the use of this approach.

To present a comprehensive view of the regulatory development process, the audit covered activities that took place as early as the mid-1990s, when the decisions to proceed with new regulations for Blood and Cells, Tissues and Organs were made, up to 2008.

Findings, Recommendations and Management Responses

Project Management

Audit Criteria

The decision to adopt a standards-based approach should be documented and in-line with expectations in the Treasury Board Project Management Policy and Regulatory Policy.

Decision to revise regulations

As early as 1994, the Directorate had begun to explore the need to develop new regulations covering cells, tissues, and organs used for transplant purposes. In deciding initially whether to develop more comprehensive regulations, the Directorate drew on the sources of existing scientific information. It consulted with scientific experts across Canada and internationally to get advice in this area. In particular, the Directorate requested a not-for-profit organization to provide a report on the status of organ and tissue transplantation in Canada.

The report covered, among other things, the prevalence and type of transplants that were being carried out. The Directorate also asked the organization to recommend measures to prevent disease from being transmitted from one person to another through tissue and organ transplants. Moreover, the Directorate consulted with a wide range of stakeholders. These consultations promoted buy-in among stakeholders and created a body of essential information that the Directorate used in making its key decision to regulate Cells Tissues and Organs. The information was also used later on when the Directorate was faced with the decision on "how best" to regulate in this area.

In the case of Blood, the same level of consultation was not required because the work of the Commission of Inquiry on the Blood System in Canada provided comprehensive information on all aspects of the blood system in this country.

The consultations combined with the Commission's Report well-positioned the Directorate to be able to move forward into the next phase of their project and ensured compliance with the Treasury Board Regulatory Policy.

Decision to use a standards-based approach

In 1995, the Directorate responded to renewed interest across the Federal Government for the standards-based approach by making the decision to use it for developing new regulations for both Blood, and Cells Tissues and Organs for Transplant.

Health Canada, like other regulatory authorities, had used this approach to develop regulations, however, using the "standards-based approach" to develop regulations in areas as complex as these two had not been done.

Given the factors of subject-matter complexity combined with piloting the standards-based approach, it would have been critical for the Directorate to have identified and assessed the many inherent risks associated with both projects. However, no documented evidence was found that demonstrated that the Directorate had rigorously profiled and assessed the project management risks associated with adopting a standards-based approach. As a result, this had some downstream implications which affected the overall project management for both biologics.

The examples noted in the remainder of the report, while not exhaustive, are intended to highlight those key areas of potential risks that the Directorate should manage to better ensure better efficiency with any future projects involving the standards-based approach to regulation.

Development of action plans

Audit Criteria

Action plans should be developed to mitigate identified risks involved in developing standards and incorporating them into the Food and Drug Regulations.

Timelines

It is important to set realistic timelines in order to avoid the risks of creating unreasonable expectations and cost overruns. Both these projects took far more time and resources to complete than originally planned. The proposal indicated a start-to-finish timeline for developing standards-based regulations for Cells Tissues and Organs at approximately 16 months. In fact this project spanned several years. Moreover, the proposal did not include a key element of the project related to referencing the standards into the regulations which required more time than originally planned. However, past experiences within the Department in developing regulations have also shown lengthy development times.

If, during the planning process for any similar project in future it becomes clear that the project will take several years to complete, this fact should signal the need to ensure that adequate measures are in place to mitigate any risks to health and safety in the interim.

Change Management

When introducing significant change to an organization, it is important to explain the key aspects to those employees who may be affected by them. Doing so reduces the risk of resistance.

Initially, the Directorate did not provide direction in a key area to Health Canada scientists who represented the Directorate on the Technical Committee involved in developing new Blood standards. Interviews indicated that the scientific representatives had not received sufficient guidance on their role vis-à-vis the Committee. Specifically, their main concern was the extent to which they, as regulators, could express reservations or disagreement with particular requirements to be incorporated into the blood standards without standing in the way of consensus required by the standard-based approach.

When the Directorate recognized this issue, it then took timely action by issuing clear guidance on the role of Health Canada's Technical Committee representatives and their obligation to convey the Department's views as the regulatory authority of Canada's blood supply.

The Directorate also gave presentations to all members of Technical Committees on the Department's role as regulator and how it would eventually be referencing part of the standards in the Regulations.

Lesson-learned exercise

In 2005, the Directorate completed a lesson-learned exercise on the use of the standards-based approach for Cells, Tissue and Organs and Blood Regulations. This document offers valuable considerations and advice on the use of the standards-based approach to regulations. For this reason, the Health Products and Food Branch should consider sharing this document across the Department and with other federal regulatory authorities. At the time of the audit, the lessons learned had yet to be incorporated into the Directorate's Policy Development Process.

Measuring and reporting results and impacts

Criteria

A Performance Management Framework should be developed that specifies performance expectations and performance information should be routinely reported to management.

Development of a performance management framework

In 1996, soon after the Directorate had decided to use the standards-based approach to regulate Cells, Tissues and Organs, it developed a performance management framework that contained high-level outcomes. These included a decreased risk of transmitting disease through transplantation and increased availability of tissues and organs for transplant. Also included, was better analysis of health outcomes associated with transplants to be supplied back into the health system with a view to improving transplant practices and procedures. However, the Directorate had not yet defined its outcomes in measureable terms.

Through interviews and document reviews, it was determined that the primary reason for this was insufficient data on transplant activities in Canada. As well, during the mid-1990's the policy and self-assessment guidance for the standards-based approach did not require departments to develop quantifiable performance expectations.

It was not until 2004 that the report of the External Advisory Committee on Smart Regulation recommended a more rigorous approach to measuring performance. This approach stressed that regulators specify the expected results of the regulations, how they intend to measure these results, and when and how often they will report on them. The Committee's report also indicated that regulators must measure their progress in achieving results and be prepared to change their approach, if needed.

To its credit, the Directorate did begin to strengthen its performance management framework for Cells, Tissue and Organs as early as 2001, and began to develop an equivalent framework for Blood in 2004. However, at the time of the audit, the evaluations for both biologics had not yet been completed. Accordingly, the Directorate has not been able to provide management with adequate performance information in a number of key areas.

Recommendation

It is recommended that the Assistant Deputy Minister of the Health Products and Food Branch further strengthen its policy development process as it relates to managing complex projects for use of the standards-based approach, including the requirements to:

• systematically profile risks;
• develop action plans with reasonable timelines;
• formalize change management action; and
• routinely report on results.

Management Response

Management accepts the recommendation.

The Health Products and Food Branch acknowledges that complex regulatory projects present special challenges and that processes must be in place to manage these projects such that continued progress is made and the objectives of the project remain relevant as time on the project elapses. In the time since the standards-based regulatory framework projects for Blood and Cells, Tissues and Organs were initiated in the mid-1990s, numerous new tools and provisions to support the regulatory process have been implemented, many of which address this recommendation.

Specifically, the creation of the Policy, Planning and International Affairs Directorate whose mandate is to provide leadership in developing and advancing HPFB's agenda as well as the creation of a Legislative and Regulatory Modernization Steering Committee. This committee steers HPFB's work to modernize the Food and Drugs Act.

Conclusion

Overall, the Biologics and Genetic Therapies Directorate was able to adopt and implement the standards-based approach for regulating these two key biologics despite a few issues related to the way the projects were managed. The Directorate should be recognized for embracing an important government-wide initiative to use alternate methods for developing regulations.

The Directorate acknowledges that there were several lessons learned throughout the life-cycle of the two projects that should be reflected in its policy development process thereby ensuring that future users will benefit from the Directorates knowledge.

Appendix 1 - Lines of Enquiry and Audit Criteria

Lines of Enquiry	Audit Criteria
Development and the implementation of regulations	The decision to adopt a standards-based approach should be documented and in-line with expectations in the Treasury Board Project Management Policy and Regulatory Policy. Action plans should be developed to mitigate identified risks involved in developing standards and incorporating them into the Food and Drug Regulations.
Performance Measurement	A Performance Management Framework should be developed that specifies clear, concrete performance expectations; and Information on performance should be reported to management.