Development of a Logic Model and an Evaluation Framework of the Canadian Food Inspection Agency's Modernized Poultry Inspection Program

Health Canada Food Safety Assessment Program

Our mission is to help the people of Canada maintain and improve their health

Ashwani Wadhera, Acting Director
Bureau of Food Safety Assessment
Sir Frederick Banting Building
4 East - 2204D
Ottawa, Ontario K1A 0L2

Telephone: (613) 954-2996
FAX: (613) 954-0149
E-mail: BFSA_BESA@hc-sc.gc.ca

This document can be made available in alternate format(s), upon request.

© Her Majesty the Queen in Right of Canada, represented by the Minister of Health, 2003

Cat. No.: H44-53/2003E-HTML
ISBN: 0-662-35160-6

For further information or to obtain additional copies, please contact:

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Executive Summary

The Canadian Food Inspection Agency's Modernized Poultry Inspection Program (MPIP) is an initiative to address the hazards associated with raw poultry leaving federally registered processing plants. The goals of MPIP are to control microbial pathogens, such as Salmonella, in raw poultry products and to enhance the use of science and risk-based management in Canada's poultry inspection system. The Canadian Food Inspection Agency's Modernized Poultry Inspection Program was chosen for assessment by a committee comprised of senior Health Canada and Canadian Food Inspection Agency representatives because of the potential risks to human health posed by these hazards.

Raw poultry products may contain microbiological, chemical, or physical hazards, which if not controlled, may result in health risks for the consumer. Hazards of microbiological origin are of greatest concern in poultry products because poultry is known to be a reservoir of food borne pathogens, particularly serotypes of Salmonella and Campylobacter. In the production of food there is a chain or continuum from the farm level, through processing, to final preparation and cooking. Poultry slaughter establishments are one link in this chain and the mitigation measures MPIP provides can influence the prevalence and/or concentration of organisms on poultry and poultry products. All federal poultry inspection programs require producer farms to submit flock information sheets that provide the processor with information regarding potential chemical and biological hazards associated with live poultry.

MPIP is a voluntary program being implemented currently in 10 federally registered poultry slaughter establishments out of 64 across Canada. The other federally registered plants are either using traditional modes of inspection (with government inspectors performing a sensory-based carcass-by-carcass inspection) or the Canadian Poultry Inspection Program (CPIP) which is a shared inspection program where industry operators perform carcass defect detection at pre-evisceration stage of the process and examine the cavity of each and every poultry carcass for defects while the CFIA inspectors inspect the carcass exterior and corresponding viscera for defects.

Health Canada and the Agency jointly decided that a logic model and an evaluation framework be developed before initiating a full assessment. The logic model is a diagram that captures information about the main elements of the program and describes in concise terms how the program operates, as well as the outcomes or results that the program is intended to produce. The evaluation framework outlines key evaluation questions, examines the type of data that needs to be collected to answer those questions including its availability and provides an evaluation strategy or options for an assessment.

Under MPIP, the industry operator is responsible for the examination of all poultry carcass exterior, cavities and viscera and for removing any defective carcasses from the evisceration line for subsequent inspection and eventual disposition by a CFIA veterinarian. The roles and responsibilities of CFIA inspectors have moved away from hands-on inspection to an inspection regime that involves continuous monitoring of the operator's defect detection performance which is a component of the establishment's Hazard Analysis Critical Control Point (HACCP) Plan. This allows resources that were previously allocated to on-line inspection to be re-directed to higher risk areas, based on need.

A MPIP logic model was developed in collaboration with the CFIA. The logic model, as well as interviews held with stakeholder groups, were assets in developing key evaluation questions and potential indicators for evaluating the effectiveness of MPIP. These questions and indicators are organized in a matrix using broad evaluation issues of Rationale, Design, Program Delivery and Impact/Success. The evaluation questions include ones that may interest CFIA Program administrators along with more science-based questions to address the health and safety elements of the program. The summary of the stakeholder interviews and the document on international practices are attached as appendices.

The data necessary to conduct a comprehensive formative evaluation were identified, provided to and discussed with the CFIA. In this process, the CFIA has received feedback on the positive aspects of MPIP and information on current limitations in measuring the program's effectiveness.

CFIA Management Response

Background

Section 11(4) of the Canadian Food Inspection Agency Act provides the Minister of Health with the responsibility for assessing the effectiveness of CFIA's food safety activities. Health Canada's (HC) Bureau of Food Safety Assessment conducts these assessments with the objectives of providing advice and guidance to CFIA on its food safety activities; and providing feedback to Health Canada to assist in carrying out its roles of developing food safety and nutrition policies and standards.

In December 2001, the Bureau selected the Canadian Food Inspection Agency's Modernized Poultry Inspection Program (MPIP) for evaluation. At that time, MPIP was a voluntary program, implemented as a pilot in eight of the 64 poultry plants across Canada. The desired result of the evaluation was to determine the effectiveness of MPIP, and to assist in establishing direction for the MPIP program prior to its mandatory implementation. As the assessment planning proceeded, the Assessment Committee determined that there was not enough information available to complete a full assessment. In September 2003 a decision was made to prepare an Evaluation Framework document, in anticipation of a full evaluation at a later stage in the Program's development.

Evaluation Challenges

In the Evaluation Framework, Health Canada identified five evaluation challenges that may warrant a delay of the assessment. First, the CFIA's current baseline survey data (Canadian Microbiological Baseline Survey of Chicken Broiler and Young Turkey Carcasses, June 1997-May 1998) would not be useful tool to determine how well MPIP plants were performing when compared to other modes of inspection. The CFIA agrees that it would be desirable to conduct a new national baseline survey to update the current baseline data regarding microbiological contamination levels (e.g. Salmonella and E. coli) in poultry. Pathogen level data from MPIP establishments could then be compared to the new baseline information and to alternate methods of poultry inspection to demonstrate the effectiveness of MPIP. CFIA is exploring options to fund a new baseline study.

The CFIA has also considered conducting another baseline survey in the "traditional" slaughter establishments to update acceptable quality level (AQL) standards. AQL standards refer to non food safety related defects in poultry such as bruises, fractures, localized areas of skin disease. Post-mortem inspection to detect defects is carried out by CFIA inspectors in non-MPIP establishment, and by industry defect detectors in MPIP-establishments. The results of defect detection from the two types of establishments could then be compared to evaluate the detection performance of industry compared to CFIA.

The second evaluation challenge identified by HC was "the manner in which salmonella samples are collected". While this is primarily a technical issue, the CFIA agrees with Health Canada's suggestion regarding the extension of sample collection period over the entire year to account for possible seasonal variations. The Agency will re-evaluate its sampling methods as it proceeds with the development of the Canadian Pathogen Reduction Program.

The third evaluation challenge identified by HC was the limited available surveillance information that could link raw poultry to food borne illnesses. The CFIA agrees that this presents a challenge with respect to measuring the outcomes of the Agency's food safety activities. The Agency supports Health Canada's ongoing efforts to collect surveillance information related to food borne illness.

The fourth evaluation challenge indicated that the small number of establishments currently participating in the MPIP program prevents the comparison of the establishments' performance results. Currently, 14 plants are participating in the Program of which 6 have fully implemented MPIP. The number of establishments participating in MPIP is increasing. We expect that this increase will facilitate the CFIA's ability to compare the performance of MPIP plants that differ, for example, in the type of species slaughtered (turkey versus chicken), in the volume of production (high versus low), or in the number of shifts employed (single versus double shifts).

The fifth evaluation challenge identified was the limited use of flock information sheets in the development of "farm to table" linkages. The CFIA plans to partner in various research projects to transform the flock sheet information into electronic data, and to analyze the data to provide information. For example, information may be generated on the effects of various feed withdrawal regimes on levels of carcass contamination.

Next Steps

The CFIA consults, on an ongoing basis, with partners and stakeholders in the development of science-based inspection methodologies. The Agency supports the assessment of the Modernized Poultry Inspection Program by Health Canada as a means to further validate the scientific basis for the program and to help establish its future direction.

While a full food safety assessment of the Program was not completed, the development of an evaluation framework for the Modernized Poultry Inspection Program has had a number of useful outcomes. A logic model, linking program activities to food safety outcomes was developed and validated by CFIA program officials. Key evaluation challenges were identified and potential evaluation questions were proposed. Finally, the evaluation framework identifies options for a future assessment of the Modernized Poultry Inspection Program.

The activities described above will assist the CFIA in development of performance measures to assess the effectiveness of the Program and to further promote the poultry industry's adoption of science-based, risk management practices.