Final Audit Report - Tobacco Control Directorate: Audit of Management Systems and Practices

October 2008

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Management Response and Action Plan

Table of Contents

• Executive Summary
• Introduction
  • Background
  • Objective
  • Scope and Approach
• Findings, Recommendations and Management Responses
  • Planning and Risk management
  • Monitoring, Measuring and Reporting
• Appendix A: Lines of Enquiry and Audit Criteria
• Appendix B: TCD Surplus History

Executive Summary

The Tobacco Control Directorate (TCD) is responsible for supporting the Federal Tobacco Control Strategy (FTCS) which is intended to reduce disease and death due to tobacco use. Health Canada plays a leadership role in implementing the Strategy along with five other federal departments. This report summarizes the main findings of an audit of the Directorate's management systems and practices.

The audit was conducted as part of the Audit and Accountability Bureau's Risk-Based Audit Plan (RBAP) 2006-07 to 2008-09, approved by the Departmental Audit and Evaluation Committee (October 2006). The audit focused on activities within TCD's control and primarily those that took place during the fiscal year 2007-08. The objective of the audit was to determine whether the Directorate's management and control systems and practices were adequate to support informed decision-making in an effective manner.

The audit was conducted by the Audit and Accountability Bureau in accordance with the Government of Canada's Policy on Internal Audit.

While acknowledging the TCD's efforts to reduce tobacco consumption, the current audit report underlines the need to strengthen the planning process, risk management framework and performance measurement, monitoring and reporting systems and practices by:

• Identifying and clearly linking work plans with the Directorate's objectives
• Establishing measurable performance indicators and targets at the work plan level to facilitate monitoring, reporting and decision making
• Implementing a formal process to document and respond to risks relating to TCD as part of their planning process
• Routinely monitoring activities and documenting the results achieved against performance targets, and adjusting activities if required

Many of TCD's management systems and practices have recently been reviewed, and the Directorate has developed a management action plan to address most of the deficiencies that have been identified. However, it will be important to follow-up on progress in this direction.

Introduction

Background

The Tobacco Control Directorate supports the Federal Tobacco Control Strategy (FTCS), which was approved by Cabinet in 2001, and renewed in 2007 with an end date of 2011. The FTCS was introduced in 2001 as a 10-year strategy to reduce disease and death due to tobacco use. It builds on the 1994 Tobacco Demand Reduction Strategy and the 1997 Tobacco Control Initiative. Health Canada shares federal responsibility for implementing the Strategy with five other federal departments. The goal or strategic objective of the FTCS (and, by extension, the Tobacco Control Directorate) is to reduce the number of Canadians who smoke from 19% (2005) to 12% by 2011.

The Tobacco Control Directorate consists of five Offices which are responsible for ensuring that initiatives are implemented to support the achievement of TCD's objectives as articulated in the FTCS. These Offices include: Management Services; Research, Surveillance and Evaluation; Policy and Strategic Planning; Programs and Knowledge Exchange; and Regulations and Compliance.

Annual expenditures are approximately $42 million, including approximately $15 million in Grants and Contributions (G&Cs). Approximately 100 persons work in the National Capital Region (NCR), and another 75 work in the regional offices. G&Cs were outside the scope of this audit since this area has recently been reviewed by AAB in the audit of the Healthy Environments and Consumer Safety Branch's Quality Assurance Process Related to Recipient Auditing.

Objective

The audit objective was to determine if appropriate governance (i.e., management) and control systems and practices were in place to support informed decision-making in an effective manner.

Scope and Approach

To fulfill the audit objective, the team focused on two Lines of Enquiry (LOE). These were: to determine if the planning systems and practices of TCD were adequate; and to determine if the systems for monitoring, measuring and reporting on the Directorate's performance were effective. These areas were assessed against generally accepted criteria, best practices, and relevant requirements embedded in TBS's Integrated Risk Management Policy, Integrated Risk Management Framework, Management Accountability Framework, and its Policy on Active Monitoring.

Please see Appendix A for detailed audit criteria for each LOE.

The audit was conducted by the Audit and Accountability Bureau in accordance with the Government of Canada's Policy on Internal Audit. Audit methodology included interviews with key individuals and the review and analysis of relevant documentation and management practices.

Findings, Recommendations and Management Responses

Treasury Board's Management Accountability Framework (MAF) clearly sets out the Treasury Board's expectations for senior managers at all levels of an organization with respect to both improving the functioning of that organization, and strengthening accountability. This Framework contains expectations relating to the areas covered by our two lines of enquiry. These areas include planning and the linkages between the Tobacco Control Directorate's strategic and operational objectives and the activities that it carries out to achieve them; the Directorate's risk management framework; and its performance measurement, monitoring and reporting systems and practices.

Planning and Risk management

TCD's planning process

Setting and linking objectives. We expected that the Directorate would have a clearly articulated high-level, strategic objective, and that its operational objectives would flow from, or support, that objective. In addition, objectives should be clearly communicated for coordination purposes, so that people understand the context and direction for the Directorate's decisions and actions.

We found that TCD's strategic and operational objectives were logically linked, and that TCD had revised and communicated its objectives to reflect its accomplishments in the first years of the 10-year mandate.

In 2001, Treasury Board Secretariat (TBS) originally approved five objectives for the Tobacco Control Directorate, which were to be met within 10 years. Within five years, TCD had met the majority of its 10-year objectives. Accordingly, in 2006 TCD proposed a new, measurable strategic objective against which to determine the success of the Directorate. The goal was to reduce the percentage of Canadians who smoke from 19% (2005) to 12% by 2011. In its proposal, the Directorate included several operational objectives, which clearly linked with the larger strategic objective.

In 2006 TCD consulted with several stakeholders to discuss and receive feedback on the Directorate's proposed objectives. These stakeholders included non-governmental organizations, researchers and academics, the general public and provincial and territorial governments.

Developing work plans and activities. A key element in an effective planning process is to develop work plans for carrying out specific activities. These activities should enable a program to meet its operational objectives and, in turn, ultimately achieve the strategic objective. In effect, planning should translate objectives into strategies, action plans and activities with targets against which progress can be monitored and measured.

As noted above, TCD has established operational objectives with measurable objectives. The Directorate follows the Department's top down planning process and reported against "Key Outcomes" which aligned with the TB documentation. However, we found that the Directorate's work plans did not explicitly identify specific activities that would contribute to meeting either its operational or strategic objectives. TCD has 1-year work plans for each Office within the Directorate. These work plans cascade from the Branch's Key Strategic Areas (KSAs), and there was not a clear cross-walk to the TCD objectives as approved by Treasury Board. Therefore, if TCD monitors and reports against its work plans, it will be difficult for them to demonstrate whether or not their initiatives have facilitated the achievement of their objectives and if not, why. The recent Capacity Assessment of TCD reinforces this observation.

In response, TCD has started to develop "crosswalks" linking performance indicators to key strategic areas. TCP has also outlined their 3-5 year strategic plan in the Treasury Board submission and a 1-3 year business plan, called an Integrated Planning Table, which links the strategic objectives of the program with the work activities. However, this table has not been updated since January 2007. While specific performance indicators have been developed, effort needs to be taken to make them more measurable. We encourage TCP to continue updating this document as a planning and monitoring tool to support management decisions taken.

This issue is discussed further under the audit's second Line of Enquiry(Monitoring, Measuring and Reporting).

Recommendation No. 1

It is recommended that the Assistant Deputy Minister of the Healthy Environments and Consumer Safety Branch ensure that individual Office work plans identify, and are clearly linked to, the specific activities which contribute to achieving the Directorate's objectives.

Management Response

TCD accepts this recommendation and will incorporate standardized work/project plans for each office that link to the operational plans for the Directorate as part of the 2009-10 Operational Planning cycle (Nov 2008 to Jan 2009). The work plans will include, wherever appropriate, indicators/targets that link to the Results-based Management Accountability Framework (RMAF) for the Federal Tobacco Control Strategy.

Planning for performance measurement. An effective planning function is evidenced by an approved organizational strategy to integrate strategic and work planning and performance monitoring. All targets should be measurable and align with the objectives of the program. Performance indicators should be used to measure performance against targets and to provide early warnings if targets have been exceeded or have not been met.

In a 2004 Evaluation of the Tobacco Control Initiative, it was recommended that strengthened performance monitoring was needed through the development and implementation of related results-based management and accountability frameworks. In a 2007 Evaluation of the Tobacco Control Program it was noted that this recommendation still required progress and the Management Action Plan to the Evaluation stated that implementation should occur April 2008. They recommended the need for clearer goals for tobacco-control infrastructure development, which will help improve performance monitoring and evaluation. They also recommended strengthened performance monitoring and evaluation through the development and implementation of related results-based management and accountability frameworks (RMAFs) across all aspects of tobacco control programs, including strategic, measurable, attainable and realistic indicators that are linked to a logic model and anticipated outcomes, with appropriate baseline information.

Additionally, we found that work plans do not always contain targets or timelines. For example, performance indicators may state "Level of uptake" or "Number of requests for information", without providing a specific target or indication of when this should occur to facilitate monitoring and reporting. The lack of specific measurable performance indicators and targets at the work or "Office" level makes it difficult to monitor, measure and report progress and make appropriate management decisions.

In response to the recent Capacity Assessment which was in response to findings by the OAG in 2006, TCD committed to participating in Branch-led initiatives to develop performance indicators and a data collection strategy with a target date of completion for March 2009. With respect to performance measurement, the Branch aims to establish meaningful indicators and targets and use performance data to support resource allocation decisions.

Recommendation No. 2

It is recommended that the Assistant Deputy Minister of the Healthy Environments and Consumer Safety Branch ensure that measurable performance indicators and targets be established at the work plan level to facilitate monitoring, reporting and decision making.

Management Response

TCD accepts this recommendation. In addition to the work committed to as part of Recommendation #1 (incorporating measurable indicators/targets to work plans for each Office linked to the operational plans for the Directorate commencing in the 2009-10 planning year), TCD will update the Integrated Planning Table for 2007-11 (a 1 to 3 year business planning document that links strategic objectives to work activities) and will continue to actively participate in the Branch-led project to develop and implement integrated planning and reporting incorporating indicators and targets to support resource allocation and reallocation decisions.

TCD's risk-management process

Risk management is another key element that should be addressed during the strategic and work plan planning processes. Therefore, we expected to find that the Directorate had identified both the key risks that could prevent or hinder it from achieving its objectives, and strategies for managing those risks. However, this was not always the case, and improvements are needed in order to comply with TBS's Integrated Risk Management Framework and Health Canada's own risk-management policy.

In 2003, Health Canada developed an integrated risk management (IRM) framework, which was based on the 2001 TBS guide Integrated Risk Management Framework. Health Canada recognizes the importance of linking IRM to the planning cycle and outlines the responsibilities of the ADMs and DGs in fulfilling IRM expectations. These senior managers are expected, among other things, to identify key risk areas for their respective areas of responsibility, and incorporate risk management in their management frameworks and planning.

We found that risk identification occurs with relation to the long term strategy perspective. TCD examined the risks associated with tobacco control using a variety of mechanisms including: meetings with the Ministerial Advisory Committee, Tobacco Control Liaison Committee, consultations with Non-Governmental Organizations and stakeholders and a roundtable with academics and researchers. There was also an environmental scan that identified major drivers for change which constituted risk areas.

We found that IRM, as it relates to the Directorate, was carried out at a high level to support the development of the Branch Risk Profile. It also led to the development of a TCD HR action plan. However, IRM had not been adequately integrated into the work planning process. Our main concern was the informal nature of risk management within the Directorate. Although the TCD management team does regularly discuss risk, we found that it had not always formally documented either the risks that had been identified, or strategies for dealing with them. In the absence of formal documentation outlining the Directorate's approach to risk-management, we have no comprehensive documented evidence that high-risk areas and mitigating steps have been identified. Given the lack of documentation demonstrating otherwise, low risk areas may be receiving unwarranted attention.

We note that the recent Capacity Assessment of TCD concluded that "little evidence of formal, sound risk assessment and analysis protocols and response strategies [were] found within TCD ... approaches to risk identification, analysis and related decision-making appears to be somewhat sporadic and not well integrated across TCD."

With the support of the Branch, TCD has developed a management action plan to ensure that the identification and assessment of risks are incorporated into TCD's planning process so that risks can be appropriately managed. TCD also plans to document its risk-assessment process to facilitate communication and consistency among the management team.

Recommendation No. 3

It is recommended that the Assistant Deputy Minister of the Healthy Environments and Consumer Safety Branch ensure that a formal process is implemented for documenting and responding to risks relating to TCD as part of the planning process.

Management Response

TCD accepts the recommendation and will ensure that risk assessment and management will be documented as a standardized part of every Directorate Management Committee and Weekly Analysis of Risk discussion, that a summary of corporate and environmental risks will be completed for the strategic planning cycle and that operational and work planning templates will include a section on risk. Further, the Functional Review indicated the need for greater capacity for integrated risk management, performance and strategic planning and business planning. A business case is under development and approved results will be implemented.

Monitoring, Measuring and Reporting

Routine monitoring is an important tool that enables management to recognize changes in the environment that may necessitate changing or adjusting program activities in response. Accordingly, TBS requires departments and agencies to implement systems that will provide for appropriate monitoring of activities.

We expected to find that activities were routinely monitored, and that their results would be documented and reported. However, we found that a formal process for routinely monitoring the results that activities produce is lacking, and that attributing results to specific activities remains problematic.

Monitoring and linking results to activities

As noted previously, TCD has developed general performance indicators for activities. However, the Directorate needs to do more work to identify both targets, and the timelines for reaching them. The targets and timelines are central to enabling managers to monitor and, subsequently, measure and report on the extent to which the results of activities have contributed to achieving the Directorate's work objectives, and hence its strategic objective, i.e., to reduce the prevalence of smoking in Canada.

The recent TCD Capacity Assessment noted that an appropriate monitoring function was lacking for the Directorate. It noted that "performance reporting appears to be largely restricted to the annual Departmental Performance Results against RPP." They also found that "except for compliance, formal workload measurement and performance measurement systems are not in place to monitor on a regular basis actual work performed and evaluation of performance against service level standards."

TCD continues to record large surpluses of funds at the end of each fiscal year. This has been an ongoing issue with an average surplus of 16% of the total TCD budget between the fiscal years 2002-03 and 2007-08. We noted that TCD does conduct monthly budget variance analyses and has taken some corrective actions on this issue. In addition, some of the reasons for surpluses are beyond its control. However, as a result of these lapses, a significant amount of money that has been earmarked for activities relating to reducing smoking has either remained unspent, or been redirected to activities outside the Directorate.

Recommendation No. 4

It is recommended that the Assistant Deputy Minister of the Healthy Environments and Consumer Safety Branchensure that TCD routinely monitors and documents the results achieved against performance targets, and that work plans and priorities are adjusted, as appropriate.

Management Response

TCD accepts this recommendation and will complete development of its Financial Decision Making Strategy incorporating sound financial principles, quarterly review of expenditures against approved operational and work plans with documented reallocation based on risk and annual or as updated risk management framework by October 2008. Tied to the quarterly review of expenditures against work plans will be the quarterly review and update of the Human Resources Plan and forecasts. TCD will continue to produce an annual performance report as per its requirements under the RMAF, including new information arising from the recommendations in this audit.

Responding to evaluations and reviews

TCD has experienced extensive coverage over the last year. In 2007 a summative evaluation of the first five years of the Federal Tobacco Control Strategy (FTCS) was completed. This evaluation was carried out to support the renewal of the Terms and Conditions of FTCS Grants and Contributions. Treasury Board had also requested a review of management practices since the Strategy had not been subject to a formative evaluation since 2001.

In 2007 the HECS Branch initiated a Branch-wide comprehensive review in response to the OAG November 2006 Chapter on Allocating Funds to Regulatory Programs: Health Canada. As a result, a Capacity Assessment of TCD and a Compliance Strategy for TCD were produced in early 2008.

The three reports relating to these exercises concluded that improvements were needed in key areas--including those discussed above--that directly affect the integrity of the Directorate. At the time of the audit, TCD had management action plans for responding to the findings in the three respective reports. For certain areas, such as performance measurement and reporting, TCD has recently started working with the Branch and has engaged consultants to assist them in improving its Performance Management Framework.

In our view, as TCD moves forward, responding to the concerns noted in this report will help to ensure that its efficiency and effectiveness are maximized.

Appendix A: Lines of Enquiry and Audit Criteria

Lines of Enquiry: To determine if the purpose and planning systems and practices of TCD support informed decision-making in an effective manner.

Audit Criteria:

• Objectives should be established and communicated.
• The significant internal and external risks faced by an organization in the achievement of its objectives should be identified and assessed.
• Policies designed to support the achievement of an organization's objectives and the management of its risks should be established, communicated and practiced so that people understand what is expected of them and the scope of their freedom to act.
• Plans to guide efforts in achieving the organization's objectives should be established and communicated.
• Objectives and related plans should include measurable performance targets and indicators.

Lines of Enquiry: To determine if the monitoring, learning and reporting systems and practices of TCD support informed decision-making in an effective manner.

Audit Criteria:

• External and internal environments should be monitored to obtain information that may signal a need to re-evaluate the organization's objectives or control.
• Performance should be monitored against the targets and indicators identified in the organization's objectives and plans.
• The assumptions behind an organization's objectives should be periodically challenged.
• Follow-up procedures should be established and performed to ensure appropriate change or action occurs.
• Management should periodically assess the effectiveness of control in its organization and communicate the results to those to whom it is accountable.

Appendix B: TCD Surplus History

TOBACCO CONTROL PROGRAM FINANCIAL INFORMATION* (unaudited)

Fiscal year	Funds Budgeted $	Actuals $	Funds Surplus (deficit) $	Lapsed $	Released to Branch $
2002-2003	65,513,803	61,152,974	4,360,829	4,360,829	N/A
2003-2004	57,398,389	59,049,895	(1,651,506)	(1,651,506)	N/A
2004-2005	53,970,446	40,325,615	13,644,831	1,022,831	12,622,000
2005-2006	49,576,335	40,442,200	9,134,135	2,511,313	6,622,822
2006-2007	46,167,788	41,942,201	4,225,587	639,937	3,585,650
2007-2008	48,318,796	35,147,517	13,171,279	7,949,056	5,222,223

Note: Total Funds Surplus / Total actuals = 16%
*Source: Branch and Departmental Financial Data (SAP)