ARCHIVED - Evaluation Of the Fish, Seafood and Production Program's Quality Management Program of the Canadian Food Inspection Agency

For further information or to obtain additional copies, please contact:

Ashwani Wadhera, Director
Bureau of Food Safety Assessment
Sir Frederick Banting Building
4 East - 2204D
Ottawa, Ontario K1A 0L2

Telephone: (613) 954-2996
FAX: (613) 954-0149
E-mail: BFSA_BESA@hc-sc.gc.ca

This document can be made available in alternate format(s), upon request.

© Her Majesty the Queen in Right of Canada, represented by the Minister of Health, 2005

Cat. H44-83/2005E-HTML
ISBN 0-662-39980-3

Executive Summary

This report is the result of an evaluation of the Canadian Food Inspection Agency's (CFIA) Fish, Seafood and Production Program's Quality Management Program (QMP) by the Bureau of Food Safety Assessment (BFSA) of Health Canada. The evaluation stems from the Minister of Health's responsibility under the Canadian Food Inspection Agency Act to assess the effectiveness of CFIA's activities with respect to food safety and nutritional quality.

QMP is a mandatory inspection program for federally registered fish and seafood establishments that was re-engineered in 1999. It is based on internationally recognized Hazard Analysis Critical Control Point (HACCP) principles which identify specific hazards and the measures to control them rather than reliance on end product testing to provide a reasonable assurance of food safety.

QMP was selected for evaluation by a joint Health Canada and CFIA committee. The scope of the evaluation included QMP activities related to fish and seafood products processed in federally registered establishments and sold in Canada and covered the period of 1999 to 2003. This evaluation focussed on the design and implementation of QMP and the extent to which it is effective in meeting its food safety objectives. Seven evaluation questions were developed to assess program relevance, design and delivery, and the extent to which the program achieves its intended food safety outcomes. Multiple lines of evidence were used to answer these evaluation questions using both quantitative and qualitative methodologies.

The evaluation found that QMP is generally effective in enhancing the safety of fish and seafood products in Canada. QMP is internationally recognized and contributes to the fish and seafood continuum (i.e. from aquaculture / harvesting to final preparation by the consumer and food service industry). We found the objectives and approach of QMP to be relevant, based on sound science, and that program priorities are planned based on risk to human health. The program was re-designed with the consultation and collaboration of industry stakeholders, and continues to evolve and incorporate stakeholder engagement. In general, the QMP stakeholders understand and are committed to regulations and policies. Results from our industry survey indicated that the vast majority of respondents felt QMP addresses food safety risks in their establishments. QMP activities such as compliance verifications (CV), incident investigations and enforcement actions contribute to addressing the food safety needs of consumers and industry.

QMP has also begun to collect performance measurement data to measure its effectiveness in meeting stated outcomes. Based on a review of this, and other program data, we found that industry compliance with regulations is satisfactory. Our data analysis found enforcement actions to be low based on the number of warning letters, suspensions and revocations of certificates of registration in relation to the overall number of QMP registered establishments. We also found that the lot traceability requirements of QMP were effective in facilitating recalls in the majority of cases we examined. However, we were not able to say with certainty, due to gaps in available program data, the extent to which emergencies and incidents were managed effectively.

In our examination of 21 recalls, which includes the three in-depth case studies, we found there were prompt responses to unsatisfactory laboratory results and/or assessments from the CFIA technical specialists. We also found that most QMP recalls were handled effectively in that the firms that recalled product were able to meet the 24 hour requirement to submit their lists of first shipping destinations. However several information gaps did exist. Because of these information gaps, we were not able to conclude if all appropriate steps in the investigation and follow-up were taken, but not recorded, or not taken at all.

While we found the design and implementation of QMP to be generally effective and meeting its food safety objectives, our evaluation has noted a number of areas for program enhancements that could assist QMP in continuous improvement. These recommendations are listed below in the order that they appear in the body of the report. The BFSA evaluation team has prioritized these recommendations, by health and safety criteria, where first priority is number 3, second priority is number 4, followed by numbers 1, 2, 5, and 6. We recognize that CFIA may implement these recommendations in a different order, according to their priorities.

1. An additional tool for work planning and risk prioritization of compliance verifications (CVs), such as a general guideline for risk ranking based on commodity / hazard concerns by operation type, should be considered to enhance risk management strategies, in support of Bulletin 24. (Bulletin 24 is a guidance document to supplement the Facilities Inspection Manual of the Fish, Seafood and Production program. The fundamental risk prioritization of Bulletin 24 is based on the existence of the establishment's HACCP plan. Those establishments that have identified significant hazards are required to develop a HACCP plan, and therefore a higher frequency of verification is required to ensure that they have addressed their identified hazards).
   
2. CFIA managers responsible for QMP should continue their performance measurement activities through the Performance Measurement Framework (PMF) and enhance performance indicators to identify areas and trends for continuous improvement. Some further refinements of PMF are suggested in the report.
   
3. CFIA should consider ongoing quality assurance of the Issue Management System (IMS) to ensure pertinent information is reported and appropriate corrective actions are recorded during an incident. CFIA should also improve linkages between relevant informatics systems, and improve data retrieval capabilities to assist decision making in the management of emergencies and incidents.
   
4. CFIA managers responsible for QMP should conduct trend analysis of complaints and incident data, which would enhance the management of emergencies and incidents.
   
5. CFIA managers responsible for QMP should examine program data to identify whether there are challenges relating to small and medium sized establishments in terms of understanding and meeting QMP requirements. This might prove useful to identify where activities could be focussed to respond to issues that may be specific to these segments of the industry.
   
6. Lastly, CFIA managers responsible for QMP should collect more data on consumer confidence related to fish and seafood safety, which would be useful for program performance measurement and continuous improvement.

The Bureau of Food Safety Assessment would like to acknowledge the cooperation and assistance of CFIA, the QMP registered establishments, and industry associations in responding to information requests that were essential to the completion of this report.

CFIA Management Response

The Canadian Food Inspection Agency (CFIA) has reviewed this report on the "Evaluation of the Fish, Seafood and Production Program's Quality Management Program of the Canadian Food Inspection Agency" and is pleased to provide a response to the report. The recommendations provided by Health Canada (HC) have been carefully considered and the Agency will take appropriate action and implement steps to enhance our food safety activities.

The CFIA, under the mandate of the Fish Inspection Regulations (FIR), requires all federally registered establishments processing fish and seafood for export and inter-provincial trade to develop and implement a Quality Management Program (QMP). HC's assessment evaluated the design and implementation of QMP and the program's effectiveness in meeting its food safety objectives. CFIA was satisfied with the process used to conduct the evaluation, including the multiple lines of evidence that were implemented to assess the program's relevance, design and delivery. The assessment made several recommendations to enhance the design and implementation of QMP, listed below.

The report found QMP's objectives and overall approach to be relevant, based on meeting the ongoing food safety needs of industry and consumers, and its strong scientific base. The QMP's design is based on the Hazard Analysis Critical Control Point (HACCP) model to ensure food safety and a risk-based approach to fish and seafood safety. While the current tools used by QMP for prioritization of activities based on risk to human health are adequate, HC recommends the implementation of an additional tool.

Recommendation #1

An additional tool for work planning and risk prioritization of compliance verifications (CVs), such as a general guideline for risk ranking based on commodity/hazard concerns by operation type, should be considered to enhance risk management strategies, in support of Bulletin 24.

CFIA'S Response:
The CFIA agrees that additional tools be considered for work planning and risk prioritization. With the development and implementation of the Performance Management Framework (PMF) there will be opportunities for further data analysis and review to enable managers to better track program delivery.

HC recognized that the Agency's Performance Measurement Framework (PMF) is an effective tool for QMP improvements, however, the report noted that enhancements are needed to PMF indicators chosen for QMP (e.g. establishment size, commodity and operation type).

Recommendation #2

After analysing the PMF indicators chosen for QMP and data collected for a nine month period since its implementation, we would recommend the following additional enhancements:

1. For the compliance verification (CV) Delivery Rate, where CV coverage is less than planned, an analysis of the risk level for those establishments that did not receive the expected number of verifications could be useful in modifying program priorities in the event that there is a need to concentrate on higher risk establishments.
   
   
2. In the Enforcement Profile, the tracking of those enforcement activities related health and safety could be advantageous.
   
   
3. Both the Domestic and Export Incident Count could include the volume of product shipped which would put these statistics into perspective. It would also be helpful to extract the number of recalls (due to failure in the establishments' QMP plan) from the Domestic Incident Count that resulted in or potentially could result in food-borne illness.
   
   
4. The tracking of PMF indicators by establishment size, commodity and operation type might prove useful to identify where activities could be focussed to respond to issues that may be specific to a segment of the industry.
   
   
5. The current Product Compliance Rate-Export measures health and safety issues separately. It would also be advantageous to have a similar measure for products shipped to Canadian markets, for a domestic product compliance rate. This could be achieved by using data such as laboratory results obtained from the Fish, Seafood and Production Division's (FSPD) Product Verification Program (PVP).
   
   
6. Some other possible performance indicators to consider could include: tracking the time it takes recalling processors to notify their first shipping destinations, tracking unsatisfactory lab results based on the failures in the establishments' QMP plan and linking compliance results with CV delivery rate, when year-to-year data becomes available.

CFIA'S RESPONSE:
The CFIA appreciates the constructive and useful recommendations for further enhancements to PMF. Each of the recommendations included in the report will be considered as the PMF is further developed and enhanced. The decision to incorporate each recommendation will be based on the ability to collect the required data and the resources necessary for review and analysis.

The assessment identified areas where the CFIA could improve data quality to further enhance the overall management of emergencies and incidents. HC found that several information gaps existed, therefore, they were not able to conclude if all appropriate steps in the investigation and follow-up were taken, but not recorded, or not taken at all.

Recommendation #3

CFIA should consider ongoing quality assurance of the Issue Management System (IMS) to ensure pertinent information is reported and appropriate corrective actions are recorded during an incident. CFIA should also improve linkages between relevant informatics systems, and improve data retrieval capabilities to assist decision making in the management of emergencies and incidents.

CFIA's RESPONSE:
The CFIA agrees there is a need to continue verifying that appropriate corrective actions are taken by recalling firms and documenting those actions in the IMS. Currently, some recall data is captured outside of the IMS. Consideration, while taking into account resource implications, will be given towards consolidating the tracking component in the IMS, enhancing the IMS reporting capabilities, and exploring opportunities of linking IMS with other database systems.

The report found that the legislation is relevant and guidelines and manuals are adequate, and support QMP's capacity to effectively manage emergencies and incidents. There are monitoring tools in place, such as PMF, which aid in the capacity to effectively manage emergencies and incidents. However, the report identified the need to improve data retrieval capabilities and trend analysis of this performance data to assist decision making in the management of emergencies and incidents.

Recommendation #4

CFIA managers responsible for QMP should conduct trend analysis of complaints and incident data, which would enhance the management of emergencies and incidents.

CFIA'S RESPONSE:
The CFIA, as a component of the PMF, will continue to individually review and analyze complaint and incident data. Potential areas for enhancement of the management of emergencies, which are identified in the course of this analysis, will be considered as part of the ongoing program improvement.

HC conducted an industry survey, interviews, and a review of program data, and concluded that industry stakeholders generally have a good understanding and strong commitment to CFIA's regulations and policies. However, the assessment noted that some segments of the industry find it challenging in meeting QMP requirements.

Recommendation #5

CFIA managers responsible for QMP should examine program data to identify whether there are challenges relating to small and medium sized establishments in terms of understanding and meeting QMP requirements. This might prove useful to identify where activities could be focussed to respond to issues that may be specific to these segments of the industry.

CFIA'S RESPONSE:
The CFIA is cognizant of the challenges facing small and medium sized establishments and will continue to consider the impact of existing and proposed regulatory requirements on this industry segment. As the PMF is further developed, the ability of the CFIA to track and analyze relevant data in this area will expand.

The Fish, Seafood and Production Division is updating a guide for processors to use in writing and implementing their QMP. This document, along with other tools being developed, will be useful for communicating to industry the regulatory requirements. In addition, the National Manager for Inspection Systems is currently participating in a United Nations' Food and Agriculture Organization (FAO) expert working group on the implementation of HACCP in small and less developed businesses. This will provide CFIA with the opportunity to share our experiences and to help develop tools for these businesses to use in order to meet the challenges.

The assessment found that the level of consumer confidence, international confidence, and industry confidence in QMP can provide some indication of the extent of public and market confidence in the safety of Canadian fish and seafood. As part of an industry survey HC conducted, 40 % of the respondents felt that the buyer/consumer confidence related to food safety and quality of their products has increased, while 45% indicated that it stayed the same since the implementation of QMP. However, more data would be useful to determine the extent of consumer confidence in Canadian fish and seafood products.

Recommendation #6

CFIA managers responsible for QMP should collect more data on consumer confidence related to fish and seafood safety, which would be useful for program performance measurement and continuous improvement.

CFIA'S RESPONSE:
The CFIA agrees that more data on consumer confidence related to fish and seafood safety would be useful for program performance measurement and continuous improvement. However, the CFIA is of the opinion that data of this sort specific to QMP would be difficult and impractical to collect. Approximately two thirds of the fish and seafood consumed in Canada is imported and therefore it would not be possible to extrapolate this data to performance measurement of QMP without a considerable investment in time and resources for analysis. The CFIA would prefer to continue to collect general consumer confidence data and analyse overall trends to measure the effectiveness of CFIA inspection programs.

In summary, the CFIA confirms its commitment to responding to the recommendations of the assessment and implementing improvements to the food safety activities related to QMP. The Agency looks forward to continued work with HC on future food safety assessments.

Table of Contents

1.0 Introduction

2.0 Description of the Program

3.0 Findings, Conclusions and Recommendations by Evaluation Question

Appendix 1 - Consumption and Hazards associated with fish and seafood products

Appendix 2 - Evaluation Questions and Methodology

Appendix 3 - Program Logic Model

Appendix 4 - List of Acronyms

Appendix 5 - Glossary of Terms

Appendix 6 - Survey Questions

Exhibits

Exhibit 1

1.0 Introduction

1. This report is the result of an evaluation of the Canadian Food Inspection Agency's Fish, Seafood and Production Program's Quality Management Program (QMP) by the Bureau of Food Safety Assessment (BFSA) of Health Canada. The BFSA's mandate is to evaluate the effectiveness of the Canadian Food Inspection Agency's (CFIA) activities with respect to food safety and nutritional quality.
   
2. QMP was chosen for evaluation by a joint Health Canada and CFIA committee. The scope of the evaluation included fish and seafood products processed in federally registered establishments and sold in Canada and covered the period of 1999 to 2003. The objective of this assessment is to evaluate the effectiveness of QMP's food safety activities and, if necessary, provide recommendations for improvements. A description of the consumption and hazards associated with fish and seafood products can be found in Appendix 1.
   
3. This evaluation focussed on seven evaluation questions which were developed to assess the program relevance, design and delivery, and the extent to which the program achieves its intended food safety outcomes. Multiple lines of evidence were used to answer these evaluation questions using both quantitative and qualitative methodologies. Evaluation methods (see Appendix 2) included an extensive document review of program data, literature reviews, interviews with CFIA personnel and stakeholders, case studies of recalls and follow-up actions, and a survey questionnaire to federally registered fish and seafood processing establishments.
   
4. The evaluation was conducted during a transition period of QMP as it implemented its Performance Measurement Framework (PMF). Therefore, some of the performance data used in this report is preliminary and based on information collected by the program during the period of January to September 2003.

2.0 Description of the Program

• 5. The Fish Inspection Act and Regulations enable the Government of Canada to regulate the safety and quality of fish and seafood products processed in federally registered establishments in Canada. These establishments are legally required under the Fish Inspection Regulations to adhere to the QMP. Currently, there are approximately 940 federally registered fish and seafood establishments in Canada.
  
• 6. QMP, as revised in April 1999, is a fish inspection and control system, based on Hazard Analysis Critical Control Point (HACCP) principles, which identifies specific biological, chemical, and physical hazards and the specified measures to control them rather than reliance on end product testing to provide reasonable assurance of food safety. Each federally registered fish and seafood establishment must develop and maintain an individual QMP plan^{Footnote 1}  which includes procedures, inspections, and record keeping to verify and document the safety and quality of fish and seafood processed in Canada. CFIA's QMP places a greater responsibility on the industry for monitoring and controlling their processes in compliance with the regulations, which has allowed CFIA to rely more on audit techniques in carrying out their inspection activities. For the purposes of this evaluation, "QMP plan" refers to the individual establishment's plan, and QMP refers to CFIA's Fish, Seafood and Production Program's Quality Management Program.
  
• 7. The fish and seafood products continuum includes many stages where potential food safety hazards can be introduced. QMP's role in the food continuum is primarily at the processing stage in federally registered fish and seafood establishments. Outside of QMP, food safety responsibilities in regulating fish and seafood are assumed by other CFIA programs and federal departments as well as provincial, territorial and municipal governments. A QMP logic model, developed for the purposes of this evaluation and focussing on food safety aspects of domestically processed fish and seafood products destined for the Canadian market, can be found in Appendix 3. This logic model identifies the key activities of QMP, as well as the immediate, intermediate and long-term intended outcomes of these activities.

Program Objectives

• 8. The objectives of QMP, as indicated in the 1998 Regulatory Impact Analysis Statement (RIAS)^{Footnote 2}, are:
  • to provide reasonable assurance that fish products are safe, of acceptable quality, and fairly traded;
  • to enhance food safety and consumer protection by preventing product and process defects from occurring rather than spending time attempting to inspect defects out of finished products;
  • to address fish quality minimum acceptability and fraudulent presentation of fish products;
  • to maintain market access by the incorporation of all HACCP principles into QMP;
  • to foster a more collaborative approach between industry and government; and
  • to encourage the utilization of modern audit or regulatory techniques and methods.

Program Design and Management

• 9. QMP's design activities include the development of regulations, policies, procedures and standards. Policy directives and interpretations are provided to CFIA Area personnel to support operational activities. QMP works in close collaboration with other areas of the CFIA and Health Canada for the development of regulatory policies and for information gathering activities.
  
• 10. Program management activities include the tracking of program results. During implementation of QMP, CFIA developed and implemented a monthly tracking report. In January 2003, QMP was included in the Performance Measurement Framework (PMF)^{Footnote 3} pilot, in order to track performance indicators that measure the effectiveness of the program. The information will be used to report on program results, and contribute to policy and procedure development and continuous improvement of the program. Since April 2003, the PMF pilot has moved towards full implementation in regards to QMP.
  
• 11. In addition, CFIA staff involved in QMP activities scan domestic and international issues related to food safety, and identify emerging issues and risks for work planning and program improvements. They also participate in international committees that develop science-based standards related to fish processing and regulatory verification systems, as well as internal working groups and committees to facilitate program changes and updates.

Program Delivery

• 12. Program delivery includes activities surrounding the registration and operation of fish and seafood establishments and ensuring compliance with regulatory requirements. The registration of establishments includes a facilities inspection and an assessment of the establishment's written QMP plan, called a systems verification (SV).
  
• 13. Another key element of program delivery is compliance verification (CV) activities to verify that a federally registered fish processing establishment has implemented their QMP plan as designed and that it meets the requirements set out in the Fish Inspection Regulations and QMP Reference Standard. This includes a combination of audit and inspection activities to ensure industry is implementing their QMP plans as designed and approved through the SV.
  
• 14. Compliance verification requires the collection of both qualitative and quantitative evidence, including test results, inspection data, and interview results. When non-conformities are identified, they are documented and the results are communicated to the establishment's management. A critical non-conformity which could result in the production of unsafe product is dealt with immediately. Corrective action plans developed by the establishment are reviewed by CFIA inspectors for acceptance and subsequent follow-up.
  
• 15. QMP provides a compliance regime which includes a range of enforcement activities that are employed when necessary to ensure compliance with federal regulations and standards. These include warning letters, product detentions, and suspension or revocation of certificates of registration. These activities may lead to product recalls or prosecutions, which are coordinated respectively by CFIA's Office of Food Safety and Recall (OFSR) and Enforcement and Investigations Services.
  
• 16. Inspector training is integral to the program and a requirement prior to conducting QMP activities. Following training, inspectors are mentored by more experienced personnel (i.e. the mandatory process of coaching and assessing new inspectors prior to them qualifying to participate in compliance verifications).

Stakeholder Engagement

• 17. For the purposes of this report, stakeholders will refer to the federally registered fish and seafood industry and its associations. CFIA and industry staff involved in QMP activities have established several committees, including the Seafood Inspection Policy Advisory Committee (SIPAC). This national joint industry and government committee meets twice a year and provides broad policy advice on inspection issues. At the CFIA Area level, one on one communication with industry occurs on a regular basis. Broader communication with industry associations also occurs where these groups exist. The purpose of consultation with stakeholders includes the sharing of information and providing stakeholders with a mechanism to voice their food safety and inspection concerns.

3.0 Findings, Conclusions and Recommendations by Evaluation Question

QUESTION 1. To what extent are the objectives and approach of QMP relevant?

• 18. Our evaluation team examined the rationale and relevance of QMP based on the following criteria / indicators:
  • extent to which objectives and requirements are based on sound science;
  • extent to which priorities are planned based on risk to human health;
  • level of performance measurement and continuous improvement;
  • food safety needs of consumers and industry; and
  • international acceptance of HACCP.

1.1 Extent to which objectives and requirements are based on sound science

• 19. CFIA identifies two main food safety objectives for QMP: (1) the provision of reasonable assurance of safe fish and seafood products and (2) the enhancement of fish and seafood safety and consumer protection, via the full incorporation of science-based HACCP principles. In terms of food safety, sound science can be defined as a rigorous and technical approach to address biological, chemical and physical hazards which affect food production.
  
• 20. QMP is based on the internationally recognized HACCP approach which focuses on preventing product and process defects from occurring as opposed to inspecting defects out of finished products. This approach is detailed in QMP documents, such as the QMP Reference Standard which describes the requirements for documentation and application of the federally registered fish and seafood establishments' QMP plan.
  
• 21. The scientific basis for QMP's objectives and requirements is supported by many internal manuals, program documents and guidelines, as well as internal food science expertise and external advice and consultation with Health Canada and other government departments. Furthermore, QMP objectives and requirements are consistent with the internationally recognized Recommended International Code of Practice - General Principles of Food Hygiene^{Footnote 4} of the Codex Alimentarius Commission (hereinafter referred to as "Code of Practice"), which is a subsidiary body of the United Nations' World Health Organization and Food and Agriculture Organization.

1.2 Extent to which priorities are planned based on risk to human health

• 22. QMP priorities are implemented through CFIA Area work plans which are based on Bulletin 24, a guidance document to supplement the Facilities Inspection Manual. The fundamental risk prioritization of Bulletin 24 is based on the existence of the establishment's HACCP plan. Those establishments that have identified significant hazards are required to develop a HACCP plan, and therefore a higher frequency of verification is required to ensure that these hazards are addressed. Those establishments that do not have significant hazards do not require a HACCP plan and therefore verifications are at a lower frequency.
  
• 23. CFIA Area work plans also consider compliance history, commodity and hazard, emerging issues, seasonality, incident frequency, and resource availability, as well as Area sampling plans linked to the Fish, Seafood and Production Division's (FSPD) national sampling program. Since 2003, FSPD has developed a new and more elaborate sampling program, known as the Product Verification Program (PVP). QMP activities include participation in the PVP. Adjustments to work plans are also made to give emergencies and incidents priority.
  
• 24. Both the frequency and scope of compliance verifications (CVs) demonstrate that QMP priorities are based on risk to human health. The scheduling of CVs are based on establishments' compliance and product profiles. An establishment's compliance profile is based on its overall ability to maintain operational controls and on its compliance with regulatory requirements. Product profiles are based on the level of health and safety risk and production volume as well as particular circumstances as in the case of speciality products to niche markets. The approach to CVs in a particular establishment is flexible so that if necessary, the scope can change when, for example, a critical non-conformity is found that impacts health and safety. Currently there is no formalized document that provides general guidance to the Areas for risk ranking based on commodity/hazard concerns by operation type. A document such as this could support or enhance Bulletin 24.

1.3 Level of performance measurement and continuous improvement

• 25. Since January 2003, QMP has participated in CFIA's ongoing Performance Measurement Framework (PMF) initiative which collects performance indicators to measure program effectiveness. Preliminary data from January to September 2003 has shown that PMF may be an effective tool for measuring program results (outcomes). The PMF information collected over time should be useful in identifying problem areas and trends that can lead to continuous improvements. The following is an annotated list of the performance indicators used in PMF:
  
  
  • CV Delivery Rate: an indicator of the extent of CV coverage;
  • Enforcement Profile: the number of warning letters, suspensions and revocations of certificates of registration;
  • Export Incident Count: the number of exported fish product incidents determined to have resulted from failures in the establishments' QMP plan;
  • Domestic Incident Count: the number of domestic fish product incidents determined to have resulted from failures in the establishments' QMP plan;
  • Product Compliance Rate-Export: the product compliance rate as a result of product inspections stemming from export certification requests;
  • Facility Conformity Rate: the number of findings of conformity with respect to critical and non-critical issues relative to the number of QMP Reference Standard sections assessed; and
  • Industry Compliance Rate: the number of certificates of registration that are suspended or revoked in relation to the number of establishments assessed.
    
• 26. Prior to PMF, CFIA managers responsible for QMP activities collected some performance data used for continuous improvement, including monthly tracking reports. Local data from Area offices was collected for these reports which tracked program implementation, such as the number of SVs and CVs conducted.
  
• 27. In addition, our review of various CFIA committee minutes and other program documentation identified several initiatives which demonstrated that QMP incorporates continuous improvement. For example, internal audits were conducted and the findings were used for program and policy changes. Also, Program Clarification Requests provide a useful mechanism to allow CFIA Area staff to seek national policy guidance which lead to improvements of program policies and more consistent program delivery.

1.4 Food safety needs of consumers and industry

• 28. For the purpose of this evaluation, the food safety needs of consumer and industry are defined as a reasonable assurance that fish and seafood products are safe and suitable for consumption. Many of QMP's planned activities contribute to addressing the food safety needs of consumers and industry. These activities (such as compliance verifications (CVs), incident investigations and enforcement actions) are based on risk and sound science and take into account emerging issues.
  
• 29. QMP staff work with industry on a continuous basis to develop policy and inspection guides and tools and incorporate their feedback for program improvements. Based on stakeholder interviews and our industry survey results, we found broad support for QMP amongst the fish and seafood industry.
  
• 30. In our industry survey^{Footnote 5}, 90% of 359 respondents rated how CFIA's QMP addresses food safety risks in their establishments' products as 4 or 5, where 5 was considered excellent. Using the same scale, 79% of the respondents rated CFIA as 4 or 5 in terms of its ability to respond to QMP related questions.
  
• 31. The design of QMP addresses consumers' food safety needs by the following activities:
  
  
  • communication with the Canadian public on food safety issues;
  • interaction with other responsible government departments and provincial / territorial partners in responding to consumer complaints; and
  • collaboration with provincial/ territorial departments and other partners and stakeholders, in the development of policies.
    
• 32. A 2001 survey of Canadian consumers, conducted by Ipsos-Reid for CFIA, indicated that 76% of Canadians trust CFIA to protect them from food-borne illnesses. While we cannot attribute this percentage directly to QMP, it does give an indication that some of consumers' food safety needs in relation to protection from food-borne illnesses are generally being met.

1.5 International acceptance of HACCP

• 33. The HACCP system of food safety management is internationally recognized as the most effective way for industry to ensure food safety. International recognition of QMP facilitates access to foreign markets and encourages a more consistent, risk based approach to fish and seafood safety as a whole. The United States Food & Drug Administration (USFDA) concluded in a 2000 audit report that overall the objectives of CFIA's QMP were being met through sound design and implementation of the inspection system which is based on HACCP principles. Based on our review, QMP is fully compatible with international HACCP principles.
  
• 34. CONCLUSION FOR QUESTION 1: We found QMP's objectives and overall approach to be relevant, based on meeting the ongoing food safety needs of industry and consumers, and its strong scientific base. QMP's design is based on HACCP principles, and its objectives and requirements parallel the internationally recognized Code of Practice of the Codex Alimentarius Commission. While the current tools for prioritization of activities based on risk to human health seem adequate, we would still recommend that CFIA managers responsible for QMP consider additional tools for QMP risk prioritization for work planning and activities. Finally, QMP has begun to measure program performance through its Performance Measurement Framework (PMF) which should be an effective tool that may lead to program improvements and further the food safety objectives of the program.
  
• 35. RECOMMENDATION #1: An additional tool for work planning and risk prioritization of compliance verifications (CVs), such as a general guideline for risk ranking based on commodity/hazard concerns by operation type, should be considered to enhance risk management strategies, in support of Bulletin 24.
  
• 36. RECOMMENDATION #2: After analysing the PMF indicators chosen for QMP and data collected for a nine month period since its implementation, we would recommend the following additional enhancements:
  
  
  • For the CV Delivery Rate, where CV coverage is less than planned, an analysis of the risk level for those establishments that did not receive the expected number of verifications could be useful in modifying program priorities in the event that there is a need to concentrate on higher risk establishments.
  • In the Enforcement Profile, the tracking of those enforcement activities related to health and safety could be advantageous.
  • Both the Domestic and Export Incident Count could include the volume of product shipped which would put these statistics into perspective. It would also be helpful to extract the number of recalls (due to failure in the establishments' QMP plan) from the Domestic Incident count that resulted in or potentially could result in food-borne illness.
  • The tracking of PMF indicators by establishment size, commodity and operation type might prove useful to identify where activities could be focussed to respond to issues that may be specific to a segment of the industry.
  • The current Product Compliance Rate-Export measures health and safety issues separately. It would also be advantageous to have a similar measure for product shipped to Canadian markets, for a domestic product compliance rate. This could be achieved by using data such as laboratory results obtained from the Fish, Seafood and Production Division's (FSPD) Product Verification Program (PVP).
  • Some other possible performance indicators to consider could include: tracking the time it takes recalling processors to notify their first shipping destinations, tracking unsatisfactory lab results based on the failures in the establishments' QMP plan and linking compliance results with CV delivery rate, when year-to-year data becomes available.

QUESTION 2: To what extent is the fish and seafood sector complying with regulations through the implementation of risk management practices?

• 37. Industry compliance to QMP requirements can be assessed based on the following types of data: enforcement actions, facility conformity rate, industry compliance rate, and recalls. While each of these data sources has limitations, when combined they offer a good indication of industry compliance to QMP requirements. In meeting program requirements which are based on HACCP, industry is implementing risk management practices that promote food safety.

2.1 Enforcement actions

• 38. When an establishment is non-compliant, the CFIA inspector can take a variety of graduated enforcement actions. This commences with warning letters, and may lead to more severe enforcement actions such as suspensions or revocations of certificates of registration. For the fiscal year 2001 - 2002, based on the CFIA Annual Report, there were 86 written warnings, 4 revocations and 2 refusals of federal registration. This needs to be placed in the context of an average of one million tonnes of fish landed in Canada per year and approximately 940 registered QMP establishments.
  
• 39. From January to September 2003, according to PMF data, QMP had a total of 31 warning letters, 7 suspensions, and no revocations of registration, based on a total of 491 compliance verifications (CVs) delivered to 328 establishments during the same period.
  
• 40. According to our analysis of the 2001-2002 CFIA Annual Report data and preliminary PMF data, we found industry is generally compliant considering the low numbers of enforcement actions taken during these time periods in relation to overall inspection activities. Where there has been non-compliance, the issuance of warning letters as a first step appears to be an effective tool in promoting compliance given the low number of suspensions and revocations compared to written warnings. In other words, the data shows that a written warning letter is often enough to encourage industry compliance. With continuation of PMF, program management should be able to establish trends for industry compliance.

2.2 Conformity rates as a measure of compliance

• 41. Based on the preliminary PMF data from January to September 2003, QMP had a 98.4 % facility critical conformity rate. The facility critical conformity rate is the number of findings of conformity with respect to critical issues relative to the number of QMP Reference Standard sections assessed. The 98.4% is out of 491 CVs delivered to 328 QMP establishments and 2,552 QMP Reference Standard sections assessed.
  
• 42. Out of the 2,552 sections assessed, there were 41 critical non-conformities(1.6%). A critical non-conformity is a failure of the QMP system that could result, or has resulted in the production of unsafe or fraudulent product. Where a critical non-conformity is found, processing is stopped and corrective action is immediate.
  
• 43. During the same period, the PMF industry compliance rate was on average 98.6%, out of 328 establishments assessed. The PMF industry compliance rate is the percentage of establishments subjected to a CV resulting in no suspensions or revocations of registration.

2.3 Recalls

• 44. When an unsafe product is identified and a recall is warranted, an assessment is required to determine the root cause of the problem and to ascertain if the establishment's QMP has failed.
  
• 45. Between March 3, 2000 and February 27, 2003, there were 44 recalls of domestically processed fish and seafood products, averaging about 15 recalls per year. Among the 44 recalls, there were 4 Class I (all microbiological), 24 Class II (13 microbiological and 11 marine biotoxin) and 16 Class III (13 microbiological, 1 marine biotoxin, 1 quality and 1 labelling)^{Footnote 6}. The marine bio-toxin recalls do not necessarily reflect a fault at the processing establishment but are due to contamination at the harvest site which falls under the Canadian Shellfish Sanitation Program (CSSP) ^{Footnote 7}. Out of the 44 recalls, during this three year period, 27 were due to contaminated raw molluscan shellfish (some due to marine biotoxin and some microbiological).
  
• 46. For the specific fiscal year of 2002-2003, from a total of 381 food recalls reported from all CFIA programs, 16 involved QMP establishments and only two of these were due to failure of the establishment's QMP plan. These two recalls should be considered in the context of an average of one million tonnes of fish landed in Canada per year and approximately 940 registered QMP establishments. While this data is too preliminary to establish any long term trends, the above recall data provides a snapshot for 2002 - 2003 and can show the level of compliance of industry with QMP requirements.
  
• 47. CONCLUSION FOR QUESTION 2: By meeting QMP requirements and complying with regulations, industry is implementing risk management practices. We have found industry is generally in compliance with QMP requirements based on a number of data sources reviewed. Preliminary PMF data shows a low number of enforcement actions (no revocations, and a low number of suspensions) in relation to the 328 QMP establishments assessed. For the same period, PMF data also shows a facility critical conformity rate of 98.4% and an average industry compliance rate of 98.6%.

QUESTION 3: To what extent are fish and seafood product emergencies and incidents effectively managed?

• 48. To answer this question our evaluation team looked at QMP's capacity to manage emergencies and incidents, and their facilitation of recalls, by reviewing manuals and guidelines, and examining data from the Issue Management System (IMS) and other corresponding program data.
  
• 49. The roles and responsibilities for fish and seafood recalls are shared between the Office of Food Safety and Recall (OFSR) and CFIA Area staff. OFSR is responsible and accountable for the coordination and consistency of decision making on food safety issues and recalls. Area staff are responsible for the investigation of food safety issues and recalls and for the establishment's implementation of recalls and follow up actions.

3.1 Capacity to manage emergencies and incidents

• 50. For this evaluation, capacity is defined as having the necessary legislative authority, and the availability and adequacy of tools (guidelines, manuals) to manage emergencies and incidents.
  
• 51. The Fish Inspection Act and Regulations provide the legislative authority for the investigation of emergencies and incidents. The guiding documents for these activities are:
  
  
  • CFIA's Food Emergency Response Manual which deals with this subject in-depth and provides guidelines and procedures to follow in case of a food emergency;
  • Chapter 11 of the Fish Products Inspection Manual (July 19, 2002), which outlines the policy and procedures governing the investigation of consumer and trade complaints; and
  • the Facilities Inspection Manual which outlines procedures to follow and action to be taken when an incident occurs.
    
• 52. Analysis of program documents indicated that the above guidelines and manuals are adequate and support QMP's capacity to manage emergencies and incidents. In addition to the guidelines and manuals, QMP has monitoring tools which help in the management of emergencies and incidents. For example, the Fish Seafood Products Complaint Summaries from 1999-2000 to 2001-2003 demonstrate that QMP kept track of complaints and provided conclusions after inspectors' investigations. Current program performance information (PMF data) also includes tracking such incidents. Currently, there has not been any trend analysis of these complaint summaries or available PMF data.
  
• 53. Our review of program data and our interviews with CFIA staff involved with QMP provided evidence of collaboration with partners such as Health Canada and provincial ministries for risk management of potential emergencies and incidents. An example was a request for a Health Canada Health Risk Assessment for cadmium levels in oysters, which resulted in the issuing of a fact sheet on the CFIA website, notifying consumers of a potential risk to health.
  
• 54. Generally, we found that QMP has a good capacity to effectively manage emergencies and incidents due to relevant legislation, adequate guidelines and manuals, and collaborative working arrangements with partners and stakeholders.

3.2 Facilitation of recalls

• 55. In instances where there is a potential risk to human health from a marketed product, recalls are necessary to ensure that the unsafe product is quickly removed from the marketplace.
  
• 56. QMP requires establishments to have a documented product identification and distribution system that allows for the rapid identification of the first shipping destination of a product implicated in a recall. According to CFIA's Food Safety and Recall Workshop Manual, (2001), the recalling establishment is required to send its distribution lists to the CFIA within 24 hours of the decision to recall.
  
• 57. From a list of 44 recalls of domestically processed fish and seafood for the period of March 3, 2000 to February 27, 2003, we selected 21 recalls based on potential risk to human health for our review. We found there were prompt responses to unsatisfactory laboratory results and/or assessments from the CFIA technical specialists. With respect to the recalling establishments meeting their 24 hour requirement to submit their lists of first shipping destinations, 16 met this requirement. In three of the other cases, there wasn't sufficient data to conclude if the establishments met the requirement, and in the other two cases, the establishments had delays.
  
• 58. Most of these 21 recalls reviewed were triggered by product sampling and some were triggered by an SV, CVs, consumer and trade complaints, or by information received from CFIA's Canadian Shellfish Sanitation Program (CSSP). From our analysis, we found that 10 of these 21 recalls analysed were due to failures in the establishments' QMP plan, where the fault occurred at the processor.
  
• 59. During our review of recall data, we found that information was inconsistently recorded and in some cases, data was missing. We attempted to augment the incomplete data by reviewing additional data from the OFSR. However, in some cases we still found data gaps and inconsistencies such as: the exact dates of activities and complaints were unclear; production codes were not recorded consistently; and some follow-up actions were not clearly reported, nor was the time frame for recalling establishments to notify first shipping destinations of the recall.
  
• 60. From the 21 recalls selected for our review, we conducted an in-depth case study of three recalls to ascertain if the recalls were effectively managed and to see whether appropriate follow-up actions were taken. These three cases were selected based on the severity of risk to human health involved. We reviewed copies of laboratory results, CV reports, and any other documentation pertinent to the recall such as correspondence and corrective action plans developed by the establishment.
  
• 61. We found, in all three cases, that investigations were initiated promptly following notification of laboratory results and technical assessments, and critical non-conformity procedures were followed (e.g. development of corrective action plans, detention of affected product, and cessation of processing). However in one case, the distribution lists of the first shipping destination were not provided to the OFSR within 24 hours. As well, important information, (such as data on specific follow-up steps, impacts of all products affected, and certain processing details) was sometimes missing. Therefore, it was not possible to ascertain if all the appropriate steps in the investigation and follow-up actions were taken in the three cases reviewed.
  
• 62. Our interviews with CFIA staff also pointed out that recall and issue management information between databases is not generally linked, such as between the laboratory sampling system and the Issue Management System (IMS), or between IMS and the results of follow-up CVs. Also, the lack of quality assurance of IMS data remains a barrier to complete a full analysis of investigation and follow-up actions.
  
• 63. CONCLUSION QUESTION 3: Based on our examination of pertinent program documentation, we found that the legislation is relevant and guidelines and manuals are adequate and support QMP's capacity to effectively manage emergencies and incidents. In addition, there are monitoring tools in place (such as PMF), which aid in the capacity to effectively manage emergencies and incidents. However, trend analysis of this performance data is required.
  
• 64. In our examination of 21 recalls, which includes the three in-depth case studies, we found there were prompt responses to unsatisfactory laboratory results and/or assessments from the CFIA technical specialists. We also found that most QMP recalls were handled effectively with respect to recalling firms meeting the 24 hour requirement to submit their lists of first shipping destinations. However several information gaps did exist. Because of these information gaps, we were not able to conclude if all appropriate steps in the investigation and follow-up were taken, but not recorded, or not taken at all.
  
• 65. Due to these data gaps, we were not able to conclude on the extent to which emergencies and incidents were effectively managed. The information gaps indicate a need for improved data quality which could improve the overall management of emergencies and incidents.
  
• 66. RECOMMENDATION #3: CFIA should consider ongoing quality assurance of Issue Management System (IMS) to ensure pertinent information is reported and appropriate corrective actions are recorded during an incident. CFIA should also improve linkages between relevant informatics systems, and improve data retrieval capabilities to assist decision making in the management of emergencies and incidents.
  
• 67. RECOMMENDATION #4: CFIA managers responsible for QMP should conduct trend analysis of complaints and incident data, which would enhance the management of emergencies and incidents.

QUESTION 4: To what extent do stakeholders understand and are committed to regulations and policies?

• 68. As in any HACCP-based regulatory program, the role of industry's understanding and participation is essential for the managing of day-to-day food safety risks at the processing level. To answer this evaluation question, four indicators / criteria were chosen:
  
  
  • the incorporation of stakeholder engagement into QMP design and delivery;
  • the extent of stakeholder participation;
  • the extent of industry's understanding and commitment to QMP; and
  • the extent to which industry complies with QMP.

4.1 The incorporation of stakeholder engagement into QMP design and delivery

• 69. QMP's primary stakeholders are the fish and seafood registered establishments and their industry associations. The 1999 redesign of QMP was preceded by active stakeholder consultations. QMP representatives participated in numerous committees, workshops, and fora involving the fish and seafood industry. Many of these consultations and committees continued through the implementation years and have given industry an opportunity to provide feedback. This feedback has affected program design and delivery. For example, at the national level, the Seafood Inspection Policy Advisory Committee (SIPAC) is a joint industry and government group which provides policy advice on seafood inspection issues and is a forum for discussion and feedback. In addition, there is ongoing liaison and regular communication between QMP Area staff and local industry and associations. One example of incorporating stakeholder feedback is the amendment to the Fish Inspection Regulations and Meat Inspection Regulations based on concerns raised by industry over the duplication of inspection requirements in dual federally registered establishments (e.g. processors of bacon-wrapped scallops).
  
• 70. The results of stakeholder engagement are also reflected in our industry survey, with 90% of the 359 respondents stating that a feedback mechanism to CFIA was somewhat important to very important. For those establishments (193 out of 359 respondents) that provided feedback on program policy, 80% indicated that CFIA was responsive to their input.
  
• 71. CFIA's web site is a useful tool for engaging industry by providing information such as inspection manuals, reference material, links to applicable acts and regulations, communiques, questions and answers. Our industry survey indicated that 70% of respondents use the CFIA web site to keep up to date. Industry stakeholders can also receive e-mail updates for program information. Until 2001, QMP newsletters were published on the CFIA web site which was a useful tool to keep industry informed during the initial phase of the re-engineering of QMP.

4.2 The extent of stakeholder participation

• 72. The fish and seafood processing industry appears to be well informed and connected with QMP. Over half of the industry representatives participated in consultation workshops leading up to the re-engineering of the program. Current industry participation related to QMP includes industry sponsored activities such as training sessions. Other types of interaction can also include attendance at workshops, information sessions, and participation in national and regional committees sponsored by CFIA.
  
• 73. Nationally, the Fisheries Council of Canada (FCC) is an umbrella association that includes many smaller industry associations, which also provides feedback to CFIA managers responsible for QMP. For example, during the re-engineering of the program, the FCC helped form a committee of industry representatives and CFIA. The purpose of this committee was to ensure that the re-engineering of QMP was in keeping with the technical capabilities of industry and to gain support for these changes.
  
• 74. Regional industry associations also provide a feedback mechanism to CFIA Areas while assisting their members in meeting QMP requirements. For example, the PEI Seafood Processors Association (PEISPA) provides ongoing support to participating establishments with such activities as in-plant training, seminars for quality managers and annual QMP verifications. In addition, they developed a newsletter and an extensive website to provide information to the industry.
  
• 75. Another key program stakeholder is the National Seafood Sector Council (NSSC)^{Footnote 8} which develops training courses on such topics as food safety and the regulatory requirements of QMP. CFIA supports this activity by reviewing training material. The training is delivered directly to the industry through partner institutions, such as the Marine Institute of Memorial University and the British Columbia Institute of Technology.

4.3 The extent of industry's understanding and commitment to the Quality Management Program (QMP)

• 76. Stakeholder interviews revealed that the implementation of QMP has generally expanded industry knowledge of food safety risks. Food safety knowledge has increased among workers, and industry feedback indicates that the program effectively addresses food safety risks. We also reviewed aspects such as management commitment, challenges to industry, training, and industry compliance to ascertain the overall level of industry's understanding and commitment to QMP.
  
• 77. Based on our interviews, we found that management commitment is essential to the success of an establishment's QMP plan. Indications are that management commitment is high, based on stakeholder feedback and the overall industry compliance rate. Furthermore, management commitment was not identified as a major challenge in our industry survey. Our survey also found 83% of respondents indicated that their production employees' understanding of food safety has increased since implementation of their QMP plans. In addition, 82% found that it was somewhat important to very important to have their production employees participate in QMP-related training.
  
• 78. Meeting QMP requirements has posed some challenges for some segments of industry. Exhibit 1 from our industry survey illustrates how establishments rate the level of ease / difficulty in complying with QMP requirements:
  
  Exhibit 1

  Self Reported Level of Ease / Difficulty in Complying With QMP Requirements

  Size of establishmentFootnote *	Small (159 responses) abFootnote **	Medium (101 responses) a	Large (69 responses) b
  Footnote * For the purposes of our analysis, small establishments were defined as less than 25 employees (159 responses), medium ranged from 25 to 100 employees (101 responses), and large, greater than 100 employees (69 responses). Return to footnote * referrer Footnote ** Categories with the same letter are not significantly different (p>0.05) in their distribution of responses for pairwise comp arisons using Fisher's exact test. Note, the establishments who provided no response to this question were not included in the test. Return to footnote ** referrer
  Difficult	25.2%	31.7%	15.9%
  Easy	37.1%	27.7%	47.8%
  Neither difficult nor easy	36.5%	39.6%	34.8%
  No response	1.3%	1.0%	1.4%

• 79. Exhibit 1 indicates that small and medium sized establishments generally find it more difficult to comply with QMP requirements (25.2% and 31.7% respectively). Among survey respondents who answered the question about level of ease, a greater proportion of medium sized establishments had difficulty complying than large establishments. Results also demonstrated that a lesser proportion of medium sized establishments had an easier time complying than the large establishments. At this time, we cannot determine that this represents an area of significant concern, as we should take into account the overall industry compliance rate of 98.6%, (out of 328 establishments assessed for PMF for the period of January to September 2003). However, future analysis of PMF data may be able to identify areas where industry support can be increased, if needed.
  
• 80. In our survey, industry identified the major challenges in complying with QMP requirements as: time, employee commitment, cost, seasonality, high turnover of employees, and understanding of QMP requirements. Where clarifications were needed, industry indicated the following as their top four sources for technical information and advice: CFIA staff, CFIA website, the United States Food and Drug Administration (USFDA), and industry associations.

4.4 The extent to which industry complies with QMP

• 81. This indicator is explored in our second evaluation question "To what extent is the fish and seafood sector complying with regulations through the implementation of risk management practices?" While there are some challenges for the fish industry, our analysis of our second evaluation question indicates that fish and seafood registered establishments are generally compliant and able to meet QMP requirements.
  
• 82. CONCLUSION FOR QUESTION 4: Based on our industry survey, key informant interviews, and a review of program data, we have found that industry stakeholders generally have a good understanding and strong commitment to regulations and policies. Industry participates in numerous workshops, committees and fora and has an opportunity to provide feedback that has affected QMP design and delivery. Our survey confirmed that industry values this feedback mechanism. Current participation in industry sponsored training sessions could also lead to an increase in understanding and commitment in QMP. Our interviews with stakeholders indicated that food safety knowledge has increased among workers since the implementation of QMP. For some segments of the industry, there are challenges in meeting program requirements; this may or may not be related to size of establishments. However, an overall industry compliance rate of 98.6% (out of 328 establishments assessed for PMF for the period of January to September 2003) indicates that these segments of the industry are complying with the program requirements in spite of the identified challenges. This compliance rate was not broken down by establishment size.
  
• 83. RECOMMENDATION #5: CFIA managers responsible for QMP should examine program data to identify whether there are challenges relating to small and medium sized establishments in terms of understanding and meeting QMP requirements. This might prove useful to identify where activities could be focussed to respond to issues that may be specific to these segments of the industry.

QUESTION 5: To what extent are fish and seafood safe and suitable for consumption?

• 84. In order to measure the extent to which fish and seafood in Canada (from federally registered establishments) are safe and suitable for consumption, we examined the following criteria:
  
  
  • extent to which QMP activities enhance the safety of fish and seafood from federally registered establishments; and
  • extent to which seafood products are implicated in recalls, incidents and food-borne illnesses.

5.1 Extent to which QMP activities enhance fish and seafood safety

• 85. Many QMP activities, such as SVs, CVs, enforcement actions, and management of emergencies and incidents help to enhance the safety of fish and seafood from federally registered establishments. Through systems verifications (SVs), the program ensures that establishments have designed a QMP plan that identifies and addresses food safety hazards. Compliance verification (CV) activities ensure that the establishment's QMP plan has been implemented as designed. These activities are planned and conducted on a frequency based on risk to human health. Enforcement actions such as product detention and disposal, warning letters, and suspension or revocation of registration, address non-compliance and enhance fish and seafood safety. In addition, through the management of food safety emergencies and incidents, QMP can help to prevent unsafe product from federally registered establishments from reaching the Canadian market.
  
• 86. CFIA managers responsible for QMP collaborate with partners and stakeholders to enhance the safety of fish and seafood. For example, CFIA staff participate with Health Canada in working groups to discuss health and safety issues and to facilitate the development of microbiological / chemical safety standards or guidelines for fish and seafood products. As well, program activities are monitored by tracking performance indicators in their PMF.
  
• 87. In our industry survey, 90% of respondents rated QMP as good to excellent on how well it addressed food safety risk in their products. Furthermore, respondents indicated that their production employees' understanding of food safety has increased since they implemented their QMP plans, which contributes to the overall enhancement of fish and seafood safety. Our stakeholder interviews also confirmed an increase in food safety knowledge amongst the fish and seafood industry.

5.2 Extent to which seafood products are implicated in recalls, incidents and food-borne illnesses

• 88. The occurrence of recalls, incidents and food-borne illnesses are another indication of the effectiveness of QMP in enhancing safety in fish and seafood from federally registered establishments. For the period of December 1999 to March 2002, there were approximately 518 consumer and trade complaints related to both imported and domestic fish and seafood products. Sixteen of these were domestic and related to alleged illnesses. Among these 16 consumer complaints, only one reported food-borne illness was confirmed to be the fault of a QMP establishment and resulted in a recall. This is within the context of an average of one million tonnes of fish landed in Canada per year and approximately 940 QMP establishments.
  
• 89. For the period between March 3, 2000 and February 27, 2003, there were 44 recalls of domestically processed fish and seafood. Among the 44 recalls, there were 4 Class I , 23 Class II and 17 Class III. Ten out of the 21 recalls that we analysed were due to failures in the establishments' QMP plan, where the fault occurred at the processor. (Refer to Evaluation Question 2, section 2.3 for further detail on these recalls).
  
• 90. Results of PMF data from January to September 2003 indicated that there were 28 QMP Domestic Incidents related to QMP establishments and nine of these were determined by CFIA to have resulted from a QMP failure in the establishment. From the Export Incident Count (which can be a reflection of the effectiveness of the establishment's QMP controls) only one incident was due to a failure of the establishment's QMP plan.
  
• 91. CONCLUSION FOR QUESTION 5: Based on our review of program documents, our industry survey results, and key informant interviews, we found that QMP contributes to the safety of fish and seafood products from federally registered establishments being safe and suitable for consumption. QMP activities such as SVs, CVs, enforcement actions, and program performance measurement all contribute to the overall enhancement of fish and seafood safety. Based on our survey results and stakeholder interviews, we found that QMP contributes to greater food safety knowledge and understanding amongst the fish and seafood industry.
  
• 92. Another indication of QMP contribution to the safety of fish and seafood in Canada (from federally registered establishments) can be shown by the number of incidents and recalls related to food-borne illnesses. From 1999 to 2002, there was only one reported food-borne illness confirmed due to a failure in an establishment's QMP plan and resulting in a recall. This is in relation to an average of one million tonnes of fish landed in Canada per year and approximately 940 registered QMP establishments. While the confirmed cases of food-borne illness attributable to QMP establishments appears low, it should be noted that there can be a high degree of under-reporting of food-borne illnesses, and other factors such as a lack of epidemiological data that influence the numbers of reportable cases.

QUESTION 6: To what extent are the public and market confident in the safety of Canadian fish and seafood?

• 93. For this question we examined the following three criteria: level of consumer confidence, international confidence, and industry confidence in QMP.

6.1 Level of consumer confidence

• 94. According to Statistics Canada's Food Consumption in Canada - 2002, "Fish consumption increased slightly in 2002 to stand at just over 7 kilograms per person. The demand for other sources of protein, marketing and promotional campaigns, and the availability of convenient and easy to prepare products have been factors contributing to this increase." This increase in consumption represents a general upward trend from 1991 when consumption was 6.27 kg per person.
  
• 95. Results from our interviews with several industry associations indicated that they felt that QMP has helped to enhance buyer / consumer confidence. Our industry survey indicated that 40 % of the respondents felt that buyer / consumer confidence related to food safety and quality of their products has increased, while 45% indicated that it stayed the same since the implementation of QMP. Approximately 14% could not say whether QMP enhanced buyer / consumer confidence. Although this data reflects the opinion of industry respondents, these results can provide an indirect indication of consumer confidence in fish and seafood safety.
  
• 96. In addition, according to a 2001 Ipsos-Reid survey, conducted on behalf of CFIA, 76% of Canadians trust CFIA to protect them from food-borne illnesses. While we cannot attribute the above percentage directly to QMP, it does give some indication of consumer confidence in CFIA's ability to provide protection from food-borne illnesses. More data about consumer confidence relating to domestic fish and seafood safety would be useful.

6.2 Level of international confidence

• 97. Canada is the sixth largest exporter of fish and seafood products, shipping to over 80 countries^{Footnote 9}. In 2001, Canada exported in excess of 547,000 tonnes of fish valued at approximately $ 4.2 billion. In 2002, Canada exported 502,942 tonnes of fish and seafood. According to PMF data, from January to September 2003, there was only one lot rejected for health and safety reasons out of 309 lots inspected for export.
  
• 98. The United States remains Canada's largest export customer with fish and seafood exports reaching $3.1 billion in 2001. According to a USFDA audit report in 2000, there were 19 detentions (0.02%) of fish or seafood products for any reason out of nearly 628 million pounds of fish entering the US from Canada in 1999.
  
• 99. In addition, QMP has made several contributions to international conferences and fora on fish and seafood safety, including the Codex Alimentarus. For example, QMP contributed to the 5^th World Fish Inspection and Quality Control Conference in October 2003 through CFIA's presentation on performance measurement of HACCP systems. As well, CFIA has provided QMP expertise and contributed to the development of the Codex HACCP standard which is closely aligned to Canadian standards. Canada has also assisted developing countries by providing technical expertise on fish processing practices, a recent example is CFIA's participation in the Aquatic Food Project Initiative^{Footnote 10}.
  
• 100. Interviews with fish and seafood industry associations indicated that many felt QMP has a high international reputation. This is further supported by our industry survey results which indicated that 193 or 54% (out of 359 respondents) felt that QMP has helped their establishments to maintain markets (domestic and foreign) for their products. As well, 42% of respondents felt that QMP has helped their establishments to enter new markets for their products and 38% felt QMP had no effect.

6.3 Level of industry confidence

• 101. Based on our interviews, industry stakeholders indicated that QMP provided an advantage in the marketplace, and that there was an overall increase in the level of industry confidence in the safety of Canadian fish and seafood products. Industry stakeholders have also indicated that the implementation of industry's QMP plans has resulted in a more educated workforce.
  
• 102. In our industry survey, respondents felt that the most positive aspects of QMP were food safety and quality control, better in-house monitoring, the compliance verification process, and that it was easy to follow. When asked how well CFIA's QMP addressed food safety risks in their products, 90% of respondents rated it 4 or 5, on a scale of 1 to 5, where 5 was considered excellent.
  
• 103. CONCLUSION FOR QUESTION 6: There are many external influences, such as food illness outbreaks and media reports that affect public and market confidence in the safety of Canadian fish and seafood. However, the level of consumer, international, and industry confidence can provide some indication of the extent of public and market confidence in the safety of Canadian fish and seafood. For example, Canada remains one of the world's largest exporters of fish and seafood products; this can show an indication of international confidence. Trends show a modest increase in fish and seafood consumption for the period of 1991 to 2002, which can be another indicator of consumer confidence.
  
• 104. Consumer confidence can not be directly attributed to QMP based on the information available for this evaluation. An Ipsos-Reid survey in 2001 showed a 76% consumer confidence level in CFIA's overall ability to protect them from food-borne illnesses; however this information can not be directly linked to QMP. Our industry survey indicated that 40% of the respondents felt that the buyer / consumer confidence related to food safety and quality of their products has increased, while 45% indicated that it stayed the same since the implementation of QMP. More data would be useful to determine the extent of consumer confidence in Canadian fish and seafood products.
  
• 105. QMP is internationally recognized and provides information and expertise to many international conferences and fora. Stakeholders indicated in our interviews that QMP provided a market advantage, and there was an overall increase in the level of industry confidence in the safety of Canadian fish and seafood products. In addition, respondents to our industry survey strongly indicated that QMP addressed the food safety risks in their products. While we can not attribute all of the above information directly to QMP, it does give some indication of the extent of public and market confidence in the safety of Canadian fish and seafood.
  
• 106. RECOMMENDATION #6: CFIA managers responsible for QMP should collect more data on consumer confidence related to fish and seafood safety, which would be useful for program performance measurement and continuous improvement.

QUESTION 7: To what extent does the program contribute towards the health and safety of Canadians through the availability of a safe fish and seafood supply?

• 107. There are many factors, such as safe processing practices, effective inspection programs, and proper consumer handling that contribute to the availability of a safe fish and seafood supply. Thus attribution becomes difficult to assign to any one influence. To look at this question we examined the extent to which QMP contributes to the health and safety of Canadians, the extent to which QMP recalls and incident related activities contribute to the avoidance of food-borne illnesses, and QMP's influence on the food safety continuum.

7.1 Extent to which QMP contributes to the health and safety of Canadians

• 108. Based on our previous analysis, we found QMP's activities, such as compliance verifications, enforcement actions, investigations and recalls are contributing to the health and safety of Canadians. Preliminary PMF data for these activities show a facility critical conformity rate of 98.4% and an average industry compliance rate of 98.6% (out of 328 establishments assessed). Even though this is preliminary data, it does provide an indication of industry's compliance to QMP requirements.
  
• 109. In annual reports and other documents, CFIA has indicated that food safety is its top priority and that it is committed to enhancing the safety of food products and Canada's food system. Our evaluation has shown that QMP has contributed to CFIA's commitment, through the implementation of a science and risk based program for fish and seafood products.
  
• 110. Fish and seafood products are an excellent source of high quality protein and are low in saturated fat, which make them a healthy food choice. According to Statistics Canada, consumption patterns are on an upward trend, perhaps due to the demand for other sources of protein, marketing and promotional campaigns, and the availability of convenient and easy to prepare products. QMP has contributed to the health of Canadians by helping to ensure that safe fish and seafood products are available to those who wish to make this food choice.

7.2 Extent to which QMP recalls and incident related activities contribute to the avoidance of food-borne illness related to fish and seafood products

• 111. One possible indicator of safe food is data on the occurrence of food-borne illnesses; however, it is difficult to get complete and consistent data to form a conclusion, due to such factors as under-reporting of illnesses and lack of attribution to QMP. Based on an analysis of the available consumer complaint data from December 1999 to March 2002, we could identify only one incident of reported food-borne illness confirmed due to failure in an establishment's QMP plan.
  
• 112. QMP's recall activities also contribute to the avoidance of food-borne illness. Based on our review of 21 recalls, including three in-depth case studies, we found there were quick responses to unsatisfactory laboratory results and/or assessments from the CFIA technical specialists. We found that most of the establishments met the 24 hour requirement to submit their lists of first shipping destinations. However due to information gaps, we were not able to conclude the extent to which emergencies and incidents were managed effectively.

7.3 Extent to which QMP has an influence on the food continuum

• 113. The fish and seafood products continuum includes many stages where possible food safety hazards can occur. This continuum can include: aquaculture, harvesting, post-harvest distribution, incoming raw materials, processing, distribution, storage, retail, and final preparation by the consumer and food service industry. While this is a broad area, we examined QMP's influence on the food continuum in a few specific areas and initiatives.
  
• 114. One area where QMP has influence on the food continuum is the CFIA's Canadian Shellfish Sanitation Program (CSSP) which ensures the safety of molluscan shellfish from the harvesting to processing stages along the food continuum. At the processing stage, QMP requires that establishments ensure that shellfish received as raw material has been harvested from an open area. QMP assists the CSSP by identifying and communicating any deficient incoming raw material, which could affect the harvesting stage.
  
• 115. CFIA managers responsible for QMP also collaborate with industries that are along the food continuum. One recent example in 2003 was a seminar to promote the HACCP approach to food safety presented to the aquaculture industry and finfish sector.
  
• 116. Finally, CFIA staff involved with the design and delivery of QMP interact with provincial, territorial and municipal governments regarding food safety responsibilities along the food continuum, such as at retail and food service levels and at local non-federally registered fish and seafood establishments. They also work collaboratively with provincial partners for the management of emergencies and incidents. This directly affects the consumer and retail level of the food continuum.
  
• 117. CONCLUSION FOR QUESTION 7: QMP makes a contribution towards the health and safety of Canadians through implementation of a science and risk based program for fish and seafood products from federally registered establishments. QMP activities such as compliance verifications, enforcement actions, and management of emergencies and incidents, as well as industry compliance to QMP requirements, all contribute to a safe fish and seafood supply. QMP interacts with partners and stakeholders along the food continuum and collaborates with other food safety programs. Information and feedback from its partners and stakeholders are incorporated into the design and delivery of QMP. These linkages help to contribute to the health and safety of Canadians through a safe and suitable fish and seafood supply.

Appendix 1 Consumption and Hazards Associated With Fish and Seafood Products

1. There exists a wide range of health and safety hazards associated with the production of fish and seafood products in Canada that could result in food-borne illness. A "hazard" as defined by the Codex Alimentarius Commission, is a biological, chemical or physical agent in, or condition of, food with the potential to cause an adverse health effect.
   
2. The most significant and frequent hazards are microbial and include all bacteria of public health significance such as: Clostridium botulinum, Clostridium perfringens, Listeria monocytogenes, Staphylococcus aureus, and Vibrio parahaemolyticus. Viruses, such as Hepatitis A, are also of concern, particularly in bivalve shellfish that may have been harvested from contaminated waters. Parasites capable of causing injury to consumers can also occur naturally in fish and seafood, with those of most concern being the nematode or roundworm (Anasakis simplex) and the fish tapeworm (Diphyllobothrium latum).
   
3. Chemical hazards can include natural toxins produced in the fish themselves or assimilated from their environment. Other chemical hazards include toxic elements such as mercury and pesticides, and the incorrect use of additives and aquacultural drugs. Physical hazards can include foreign objects such as glass or metal fragments that may occur during processing, handling and storage.
   
4. Fish and seafood products constitute an important dietary source of protein. According to Statistics Canada, yearly consumption has been moderately increasing since 1991 to just over 7 kg per person in 2002. However, these statistics do not differentiate between domestic and imported products.
   
5. It is difficult to verify the link between fish and seafood consumption and food-borne illness due to limits in data collection, analysis and communication. Although CFIA and provincial health officials carry out investigations, the source of contamination and underlying causes may not always be identified.

Appendix 2 Evaluation Questions and Methodology

Evaluation Questions

1. The following seven questions were examined for this evaluation. These questions are based on the outcomes of QMP found in the QMP Logic Model in Appendix 3.
   • To what extent are the objectives and approach of QMP relevant?
   • To what extent is the fish and seafood sector complying with regulations through the implementation of risk management practices?
   • To what extent are fish and seafood product emergencies and incidents effectively managed?
   • To what extent do stakeholders understand and are committed to regulations and policies?
   • To what extent are fish and seafood safe and suitable for consumption?
   • To what extent are the public and market confident in the safety of Canadian fish and seafood?
   • To what extent does the program contribute towards the health and safety of Canadians through the availability of a safe fish and seafood supply?

Evaluation Methodologies

• 2. The evaluation was conducted using the following methodologies:
  
  Program Document Review
  
  This included a review of relevant background documents and information gathered on program results and performance measurement framework data (PMF), which demonstrated program effectiveness and impacts. Other relevant documents from key stakeholders, partners and CFIA were also reviewed.
  
  Literature Review
  
  A literature review of both published and unpublished academic and scientific papers was conducted and information analysed to answer particular questions related to the evaluation. This was used in addressing topics related to the design and delivery of the program and the program's effect on human health.
  
  Key Informant Interviews
  
  The evaluation team interviewed appropriate CFIA managers and staff responsible for QMP at headquarters and the Areas, as well as Health Canada staff, industry and academia for this evaluation. The interviews with key informants provided critical information and perspective on the program.
  
  Industry Survey
  
  An industry survey was designed in collaboration with CFIA staff responsible for QMP and with the assistance of Statistics Canada; the Communications, Marketing and Consultation Directorate of Health Canada; and Decima Research Inc. The survey questions were tested with the assistance of several registered fish and seafood establishments and sent out between October 3 to November 7, 2003. Out of approximately 940 establishments, the response rate for the survey was roughly 37% or 359 establishments, which is considered above average based on similar surveys. Most of the survey questions were answered in their entirety by all those who responded to the survey. We have indicated in the report where a smaller number responded to a specific question. Refer to Appendix 6 for list of survey questions.
  
  Recall Case Studies and Review
  
  This involved an in-depth case study of three recalls to ascertain if the recalls were effectively managed and to see whether appropriate follow-up actions were taken. These recalls were selected based on the severity of risk to human health involved. We reviewed copies of laboratory results, compliance verification reports, and any other documentation pertinent to the recall such as correspondence and corrective action plans developed by the establishment.

Evaluation Team
Senior Project Manager:
Darren Goodyear

A/Senior Project Manager:
Shirley Chalouh

Team members:
Freddy Wu
Jason Pelletier
Michael Andre

Appendix 3 Quality Management Program Logic Model

Appendix 4 List of Acronyms

BFSA - Bureau of Food Safety Assessment (Health Canada)

CAP - Corrective Action Plan

CFIA - Canadian Food Inspection Agency

CSSP - Canadian Shellfish Sanitation Program (Canadian Food Inspection Agency)

CV - Compliance Verification

FAO - Food and Agriculture Organization (of the United Nations)

FCC - Fisheries Council of Canada

FSPD - Fish, Seafood and Production Division (Canadian Food Inspection Agency)

HACCP - Hazard Analysis Critical Control Point

HC - Health Canada

IMS - Issue Management System

NSSC - National Seafood Sector Council

OFSR - Office of Food Safety and Recall (Canadian Food Inspection Agency)

PEISPA - Prince Edward Island Seafood Processors Association

PMF - Performance Measurement Framework

PVP - Product Verification Program

QMP - Quality Management Program

SIPAC - Seafood Inspection Policy Advisory Committee

SV - Systems Verification

USFDA - United States Food & Drug Administration

Appendix 5 Glossary of Terms

CFIA Areas - with its headquarters in the National Capital Region, the CFIA is organized into four operations Areas (Atlantic, Quebec, Ontario, and Western) that are subdivided into 18 regional offices, 185 field offices (including border points of entry), and 408 offices in non-government establishments, such as processing facilities.

Codex Alimentarius Commission - a subsidiary body of the United Nations' World Health Organization and Food & Agriculture Organization.

Compliance Verification (CV) - activities carried out under QMP to verify that a federally registered fish processing establishment has implemented their QMP plan as designed and that it meets the requirements set out in the Fish Inspection Regulations and QMP Reference Standard. This includes a combination of audit and inspection activities.

Consumer Complaint - see definition for incident.

Critical Non-conformity - a failure of a processing establishment's QMP system that may result, or has already resulted, in the production of an unsafe or fraudulent product.

Effectiveness - the extent to which a program achieves its objectives. In the case of BFSA's food safety assessments, effectiveness also includes the extent to which the CFIA's activities are contributing to the safety and nutritional quality of food sold in Canada.

Enforcement Actions - QMP provides a compliance regime which includes a range of enforcement activities that are employed when necessary to ensure compliance with federal regulations and standards. These include warning letters, product detentions, and suspension or revocation of certificates of federal registration. These activities may lead to product recalls or prosecutions, which are coordinated respectively by CFIA's Office of Food Safety and Recall (OFSR) and Enforcement and Investigations Services.

Food Emergency - is broadly defined as any situation involving or potentially involving food which may pose a high health and safety concern to humans. Emergencies usually involve significant resources and require the coordination of a timely and/or extraordinary operational response.

Hazard - according to the Codex Alimentarius Commission of the United Nations' Food and Agriculture Organization (FAO), a hazard is a biological, chemical or physical agent in, or condition of, food with the potential to cause an adverse health effect.

Hazard Analysis Critical Control Point (HACCP) principles - identify specific hazards and specify measures to control them rather than relying mainly on end product testing to provide a reasonable assurance of food safety.

Incident - an issue that requires documentation and/or investigation into food safety, quality, labelling or fraud concerns within the CFIA's Program's priorities and mandate. An incident can be generated by a variety of sources including consumer complaints.

Issue Management System (IMS) - CFIA's informatics system for tracking information regarding incidents which could include consumer complaints, emergencies and recalls.

Industry - for this evaluation, "industry" denotes federally registered Canadian processors and manufacturers of fish and seafood products.

Non-conformity - a deviation from a processing establishment's QMP system that results in the establishment not following its QPM plan or not complying with the Fish Inspection Regulations.

Performance Measurement Framework (PMF) - a CFIA initiative to collect data for performance measurement indicators to measure program effectiveness and outcomes. QMP was selected as one of three pilots and was implemented from January 2003 to March 2003. The PMF project for QMP has been extended and continues to be implemented.

Prerequisite Plan - part of the establishment's QMP plan which includes programs that control the plant environment, and recall procedures to ensure compliance with the Fish Inspection Regulations.

Program Clarification Requests - a mechanism to allow CFIA Operations and Program staff to seek national policy guidance.

QMP Plan - each individual federally registered fish processing establishment must develop and maintain an individual QMP plan in accordance with the QMP Reference Standard, which includes procedures, inspections, and record keeping to verify and document the safety and quality of fish and seafood processed in Canada. CFIA's QMP places a greater responsibility on the industry for monitoring and controlling their processes in compliance with the regulations, which has enabled CFIA to rely more on audit techniques in carrying out their inspection activities.

QMP Reference Standard - the standard is based on the Fish Inspection Regulations and details the requirements for the documentation and application of a fish processing establishment's QMP plan. There are seven sections to the Reference Standard which include management roles and responsibilities, background product and process information, Prerequisite Plan, Regulatory Action Point (RAP) Plan, HACCP Plan, verification and maintenance of the industry's QMP plan, and record keeping.

Recall - action where an establishment removes from further sale or use, or corrects, a marketed product that contravenes legislation administered and/or enforced by CFIA.

Recall Classification - the numerical designation (i.e. Class I, Class II or Class III) assigned by the OFSR to a particular product recall to indicate the relative degree of health risk presented by the product being recalled.

Class I Recall - is a situation in which there is a reasonable probability that the use of, or exposure to, a violative product will cause serious adverse health consequences or death.

Class II Recall - is a situation in which the use of, or exposure to, a violative product may cause temporary adverse health consequences or where the probability of serious adverse health consequences is remote.

Class III Recall - is a situation in which the use of, or exposure to, a violative product is not likely to cause any adverse health consequences.

Re-engineered QMP - term used by CFIA to describe the new QMP that was re-designed in 1999 to include such features as all of the 7 HACCP principles.

Regulatory Action Point (RAP) Plan- a processing step where control measures can be applied to ensure that the product meets regulatory requirements, focussing on minimum standards for acceptable fish products, input materials and product labelling.

Revocation - a certificate of registration, licence or permit issued pursuant to the Fish Inspection Regulations is cancelled and withdrawn for violations of the Fish Inspection Regulations and that all privileges with respect to certificate of registration, licence or permit are removed. Without this federal registration, the establishment is not allowed to ship its product inter-provincially or internationally.

Stakeholders - QMP's primary stakeholders are the fish and seafood registered establishments and their industry associations.

Suspension - a certificate of registration, licence or permit issued pursuant to the Fish Inspection Regulations is temporarily withdrawn for the specific period of time noted in the notice of suspension and that all privileges with respect to the certificate of registration, licence or permit are temporarily removed. Without this federal registration, the establishment is not allowed to ship its product inter-provincially or internationally.

Systems Verification (SV) - an evaluation of a federally registered fish processing establishment's documented QMP plan against the QMP Reference Standard to verify that it contains all the necessary components and has the necessary controls to ensure compliance with the Fish Inspection Regulations.

Appendix 6 Survey Questions

QMP and Food Safety Risks

1. Using a scale of one to five (where one is poor and five is excellent), how well does CFIA's QMP address food safety risks in your establishment's products?
   

   1	2	3	4	5	Don't know
   (Poor				Excellent)

2. How would you rate CFIA's ability to respond to your establishment's questions related to QMP?
   

   1	2	3	4	5	Don't know
   (Poor				Excellent)

3. Since the implementation of re-engineered QMP, has your establishment's ability to track its product to the 1st shipping destination...?
   
   
   1. Been hindered significantly
   2. Been hindered moderately
   3. Had no impact
   4. Been helped moderately
   5. Been helped significantly
   6. Cannot stay

General Experience with QMP

• 4. In your opinion, what are the most positive aspects of CFIA's QMP?
  
• 5. If you could suggest specific improvements in CFIA's QMP, what would they be?
  
• 6. How would you rate your establishment's level of ease or difficulty in complying with the QMP requirements to date?
  
  
  1. Very difficult
  2. Somewhat difficult
  3. Neither difficult nor easy
  4. Somewhat easy
  5. Very easy
  6. Cannot say
     
• 7. What are the major challenges your establishment experienced in complying with the QMP requirements? (Please check all that apply)
  
  
  1. Management commitment
  2. Employee commitment/attitude
  3. High Turn-over of employees
  4. Seasonality
  5. Access to training
  6. Access to technical and scientific information
  7. Cost
  8. Time
  9. Understanding QMP requirements
  10. Language/literacy
  11. Other (specify)
  12. No challenges
  13. Cannot say

Impact of QMP on Consumers and Buyers

• 8. Since the implementation of QMP, has buyer/consumer confidence related to food safety and quality of your establishment's products...?
  
  
  1. Decreased a lot
  2. Decreased somewhat
  3. Stayed the same
  4. Increased somewhat
  5. Increased a lot
  6. Cannot say
     
• 9. How does your establishment gauge consumer/buyer confidence in its product(s)? (Please check all that apply)
  
  
  1. Consumer satisfaction survey
  2. Consumer feedback
  3. Buyer feedback
  4. Third party audits
  5. Sales figures
  6. Do not gauge consumer confidence
  7. Do not gauge buyer confidence
  8. Other (please specify)
  9. Cannot say

Impact of QMP on Markets

• 10. Has QMP hindered or helped your establishment to maintain markets (domestic and foreign) for its products?
  
  
  1. Hindered significantly
  2. Hindered moderately
  3. Had no impact
  4. Helped moderately
  5. Helped significantly
  6. Cannot say
     
• 11. Has QMP hindered or helped your establishment to enter new markets for its products?
  
  
  1. Hindered significantly
  2. Hindered moderately
  3. Had no impact
  4. Helped moderately
  5. Helped significantly
  6. Cannot say
     
• 12. (If helped in Q.11 above) In which region(s) has QMP helped your establishment to enter new markets? Please check all that apply
  
  
  1. Canadian
  2. United States
  3. European Union
  4. Asia
  5. Other (please specify)
  6. Cannot say

Industry and QMP

• 13. Has your establishment provided feedback into CFIA's QMP Program policy? Check one only
  
  
  1. No feedback at all
  2. Very little feedback
  3. Some feedback
  4. Significant feedback
  5. Feedback is provided through industry association
  6. Cannot say
     
• 14. How well does CFIA respond to your establishment's feedback into QMP? Check one only
  
  
  1. Not at all responsive
  2. Not very responsive
  3. Somewhat responsive
  4. Very responsive
  5. Cannot say
     
• 15. How important is it for industry to have a feedback mechanism so that they can comment on QMP requirements and verification procedures? Check one only
  
  
  1. Not at all important
  2. Not very important
  3. Somewhat important
  4. Very important
  5. Cannot say

Training

• 16. In the past 12 months, on average, how much time (hours/days) did each production employee (not including managers and supervisors) spend on training related to your establishment's QMP Plan? Please specify average number of hours.
  
  
  1. Average training per employee in Hours or Days
  2. None. (No training took place in that time period)
  3. Cannot say
     
• 17. On average, how much time does each new production employee (not including managers and supervisors) spend on learning about the requirements of your establishment's QMP plan? Check one only
  
  
  1. Zero hours
  2. Less than half a day
  3. Half a day
  4. One day
  5. 1-3 days
  6. More than 3 days
  7. Cannot say
     
• 18. Since your establishment implemented its QMP Plan, would you say that its production employees' (not including managers and supervisors) understanding of food safety hazards and controls:
  
  
  1. Has decreased significantly
  2. Has decreased somewhat
  3. Has remained about the same
  4. Has increased somewhat
  5. Has increased significantly
  6. Cannot say
     
• 19. What training challenges, if any, has your establishment experienced? Please check all that apply.
  
  
  1. No challenges
  2. Accessing training material
  3. Distance from training institutions
  4. Cost associated with training
  5. High turn-over of employees
  6. Seasonality
  7. Lack of internet access
  8. Other Challenge(s) (Specify)
  9. Cannot say
     
• 20. How important is it for your establishment to have its production employees (not including managers and supervisors) participate in training related to its QMP Plan?
  
  
  1. Not at all important
  2. Not very important
  3. Somewhat important
  4. Very important
  5. Cannot say

Information Availability

• 21. Where does your establishment find technical information and advice to satisfy its requirements under QMP? Please check the three most important sources
  
  
  1. CFIA web site
  2. Other web sites (Specify)
  3. CFIA staff
  4. US Food and Drug Administration
  5. Foreign country
  6. Universities/Colleges
  7. Industry associations
  8. Private consultant
  9. Libraries
  10. Other source(s) (Specify)
  11. None
  12. Cannot say
      
• 22. Does your establishment use the CFIA QMP Website to keep up to date?
  
  
  1. Yes
  2. No
  3. Not currently connected to the internet
  4. Not aware of website
  5. Cannot say

Establishment Profile

• 23. In which province or territory is your establishment located?
  
  
  1. 
     
• 24. What is the approximate production volume per year of your establishment, including all products? Please indicate appropriate unit of measure and average volume
  
  
  1. Average annual production volume
  2. Cannot say
     
• 25. Please indicate which of the following are your establishment's top three selling products in Canada over the past 24 months. For these three products, please also indicate the average number of Critical Control Points (CCP) that have been identified.
  
  
  1. Canned fish
  2. Smoked fish
  3. Marinated / pickled
  4. Salted
  5. Cooked
  6. Fresh frozen
  7. Moluscan shellfish
  8. Breaded and battered products
  9. Prepared meals
  10. Other Product (Specify)
  11. Cannot say
      
• 26. In the last 24 months, in terms of quantity (weight), what are your establishment's top three selling products for export? Please indicate the approximate quantity and unit of measure for these three products
  
  
  1. Canned fish
  2. Smoked fish
  3. Marinated / pickled
  4. Salted
  5. Cooked
  6. Fresh frozen
  7. Moluscan shellfish
  8. Breaded and battered products
  9. Prepared meals
  10. Other Product (Specify)
  11. Cannot say
      
• 27. How many production employees does your establishment have during full production?
  
  
  1. Number of employees
  2. Cannot say
     
• 28. In which food industry association(s) is your establishment currently a member?
  
  
  1. Please specify:
  2. None - Not currently a member of an industry association
  3. Cannot say
     
• 29. What is your position or job title in the establishment?
  
  
  1. Current Position/Title:
     
• 30. How long have you held this position?
  
  
  1. Number of years:
     
• 31. What was your own level of participation in the development of your establishment's current QMP Plan?
  
  
  1. No participation
  2. Some Participation
  3. Moderate Participation
  4. A lot of Participation
  5. Full participation
  6. Cannot say
     
• 32. What is your own level of participation in the day-to-day operation of your establishment's current QMP Plan?
  
  
  1. No participation
  2. Some Participation
  3. Moderate Participation
  4. A lot of Participation
  5. Full participation
  6. Cannot say