ARCHIVED - Evaluation Framework of the Canadian Food Inspection Agency's Food Safety Activities Related to Fresh Fruit and Vegetables

Abstract

This evaluation framework is a plan for evaluating CFIA's food safety activities related to fresh fruit and vegetables. The two key elements of this framework are the logic model of CFIA's food safety activities related to fresh fruit and vegetables and the evaluation matrix. The logic model is a diagram which depicts the main inputs, activities, outputs and outcomes related to the safety of fresh fruit and vegetables available in Canada. The evaluation matrix tabulates evaluation questions, as well as their indicators and proposed data analysis and methodologies using multiple lines of evidence to answer these evaluation questions. The evaluation questions are organized under four broad categories: rationale, design, delivery, and outcomes. The implementation of this plan in a future evaluation will determine the CFIA's effectiveness in achieving its intended goals and objectives related to the safety of fresh fruit and vegetables.

Bureau of Food Safety Assessment, Health Canada
July 2007

ISBN: 0-662-46243-9
Cat. H164-46/2007E
HC Pub. No. 4299

Table Of Contents

• Copyright
• Executive Summary
• 1.0 Introduction
  • 1.1 Evaluation Scope
  • 1.2 Evaluation Approach
  • 1.3 Fresh Fruit and Vegetables Industry and Consumption in Canada
  • 1.4 Food Safety Concerns Related to FFV
  • 1.5 Food borne Illness Data
• 2.0 Program Profile
  • 2.1 CFIA Governance Structure
  • 2.2 FFV Program Description
• 3.0 Logic Model
  • 3.1 Narrative for the Logic Model
    • 3.1.1 Inputs
    • 3.1.2 Activities and Outputs
    • 3.1.3 Reach
    • 3.1.4 Outcomes
    • 3.1.5 Immediate Outcomes
    • 3.1.6 Intermediate Outcomes
    • 3.1.7 Ultimate Outcome
  • 3.2 Logic Model Diagram
• 4.0 Evaluation Matrix
  • 4.1 Evaluation Questions
  • 4.2 Evaluation Matrix Table
  • 4.3 Evaluation Methodologies
• 5.0 Evaluation Challenges
  • 5.1 Food Continuum and Legislative Mandate of the Program
  • 5.2 Food Safety Risks
  • 5.3 FFVP Food Safety Data
  • 5.4 Capacity to Conclude on Industry Adherence to Food Safety Standards
• 6.0 Evaluation Strategy
• Appendix A - Terms of Reference (without appendices)
• Appendix B - CFIA-FFVP Jurisdiction Flowchart
• Appendix C - Fresh Fruit and Vegetables Assessment Committee Members
• Appendix D - Health Canada Evaluation Team Members
• Appendix E - Glossary of Terms
• Appendix F - List of Acronyms

Copyright

"Our mission is to help the people of Canada maintain and improve their health"
Health Canada

Contents may not be reproduced for commercial purposes, but any other reproduction, with acknowledgements, is encouraged.

Également offert en français sous le titre :

"Cadre d'évaluation des activités de l'Agence canadienne d'inspection des aliments en matière de salubrité des fruits et légumes frais".

For further information or to obtain additional copies, also in alternate formats, please contact:

Ashwani Wadhera, Director
Bureau of Food Safety Assessment
Sir Frederick Banting Building
4 East - 2204D
Ottawa, Ontario K1A 0L2

Telephone: (613) 954-2996
FAX: (613) 954-0149
E-mail: BFSA_BESA@hc-sc.gc.ca

© Her Majesty the Queen in Right of Canada, represented by the Minister of Health, 2007

ISBN: 0-662-46243-9
Cat. H164-46/2007E
HC Pub. No. 4299

Executive Summary

The Health Canada's (HC's) Bureau of Food Safety Assessment (BFSA) uses an evaluation approach in accordance with Treasury Board of Canada Secretariat's Evaluation Policy to evaluate the effectiveness of the Canadian Food Inspection Agency's (CFIA) activities related to food safety. BFSA also provides feedback to HC's Food Directorate with respect to developing policies, standards, and guidelines.

This evaluation framework is a plan for evaluating CFIA's food safety activities related to fresh fruit and vegetables (FFV). The implementation of this plan in a future evaluation will determine the CFIA's effectiveness in achieving its intended goals and objectives related to the safety of fresh fruit and vegetables. The time of implementation of the evaluation will be determined by the joint HC-CFIA Food Safety Advisory Committee at its next planning session.

Under the Canada Agricultural Products Act, the Fresh Fruit and Vegetable Regulations objective is to provide fresh fruit and vegetables that are safe, wholesome, properly graded, packaged and labelled. The federal regulatory control of FFV falls under CFIA"s Fresh Fruit and Vegetables Program (FFVP) in the Agri-Food Division reporting to the Food Safety Directorate. The FFVP through product inspections, aims to contribute to the safety of fresh fruit and vegetables produced domestically, imported into Canada and/or traded inter-provincially.

The FFVP is supported by internal partners such as the Food Safety Investigations Program (FSIP) in the Food Safety Division (formerly the Bureau of Food Safety and Consumer Protection), Food Safety Directorate which enforces the food safety provisions of the Food and Drugs Act and Regulations through activities such as establishment assessments and inspections, food safety investigations, and risk-management practices. The FSIP focuses on imported foods and those domestic foods that are not manufactured or produced under other federal inspection programs such as the FFVP. The FFVP is also supported by the Food Microbiology and Chemistry Evaluation Division (FMCED), which works closely with FFVP on risk assessment issues and trend analysis, the design and development of sampling plans and selection of analytical methodologies.

The two key elements of this framework are the logic model of CFIA's food safety activities related to fresh fruit and vegetables and the evaluation matrix. The logic model is a diagram which depicts the main inputs, activities, outputs and outcomes (in both ongoing and aspired terms) related to the safety of fresh fruit and vegetables available in Canada. The evaluation matrix tabulates evaluation questions, their indicators as well as proposed data analysis and methodologies using multiple lines of evidence to answer these evaluation questions. The evaluation questions are organized under four broad categories: rationale, design, delivery, and outcomes. As the FFVP intends to go through various changes in the future (stronger food safety focus), this evaluation places emphasis on program design and its delivery along with the potential impacts for the future re-design. This framework can also assist the CFIA in the development of performance measures for FFVP and to further promote the Program's science-based and risk-management practices.

This evaluation framework consists of a proposed evaluation strategy agreed to by the joint Health Canada and Canadian Food Inspection Agency Food Safety Assessment and Advisory Committees.

1.0 Introduction

1. There is an increasing demand by the Canadian public for a continuous supply of fresh fruit and vegetables (FFV) in order to fulfill Canadians' dietary needs and nutritional goals. Traditionally, the public perceived that the food safety risks (with the exception of pesticides, irradiation and fruits and vegetables derived from genetically modified plants) associated with FFV were low whereas reality indicates a major concern is with microbial contamination of produce as indicated in section 1.4 below ^{1 2 3}. As well, developments in globalized trade with a wide variety of imported fresh produce, new process technologies, and new products (fresh-cut and prepackaged salads) have brought new identified hazards and risks.

2. Consequently, CFIA's FFV Program (FFVP) (as depicted in the FFVP logic model (section 3.2)) intends to re-design its program and activities related to domestic and imported fresh fruit and vegetables. The re-design objective is to improve overall coverage, encompassing all FFV related food safety hazards while introducing new activities and controls to reduce, mitigate, and eliminate food safety risks associated with FFV across the food continuum.

1 Views of the Public & Stakeholders on Health Products and Food: A report to Assist Policy Development in HPFB (Paul MacNeil Consulting, 2001).
2 Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2005
3 "Foodborne Outbreaks in Canada Linked to Produce" A.M. Sewell & J.M. Farber, Journal of Food Protection, 2001

1.1 Evaluation Scope

3. Section 11(4) of the Canadian Food Inspection Agency Act provides the Minister of Health with the responsibility of assessing the effectiveness of CFIA's food safety activities in accordance with the Treasury Board of Canada Secretariat's Evaluation Policy. Health Canada's Bureau of Food Safety Assessment (BFSA) conducts these assessments with the objectives of providing advice and guidance to CFIA on its food safety activities, and providing feedback to HC to assist in carrying out its roles of developing food safety and nutrition policies and standards.

4. In September 2003, the joint HC-CFIA Food Safety Assessment Advisory Committee (referred to as the Advisory Committee in this document) comprising senior representatives from HC and CFIA identified domestic fresh fruit and vegetables as an evaluation priority for the fiscal year 2004-2005 (see Appendix A).

In March 2005, due to the proposed re-design of the FFVP and after negotiations with the Advisory Committee, it was decided to develop an evaluation framework and logic model for the FFVP, that included the proposed FFVP re-design and import activities. The full FFVP evaluation would be conducted at a later date, to be agreed upon by the Advisory Committee. In addition, any other CFIA activities related to FFV, such as the sprouts project under the CFIA's Food Safety Investigation Program (FSIP), may be considered in the future evaluation.

1.2 Evaluation Approach

5. In collaboration with the FFVP, BFSA developed an evaluation framework which is essentially a guidance document for assessing the effectiveness of FFVP's food safety activities in a future evaluation. The evaluation framework includes a logic model of both the current and proposed FFVP food safety activities. The logic model is a visual diagram that outlines activities and outputs of the program as well as intended outcomes. This logic model was used to formulate evaluation questions concerning program rationale/ relevance, design, delivery, and outcomes. As part of the evaluation framework, these evaluation questions are presented in a matrix that identifies the information required to address them (indicators), data sources, as well as methodologies for collecting and analysing the information. These evaluation questions are the basis on which the effectiveness of CFIA's food safety activities related to FFV will be assessed in the future evaluation. The evaluation framework also outlines some evaluation challenges that need to be examined such as data availability, multiple jurisdictions, food continuum, and provides an evaluation strategy for the future assessment of the FFVP.

6. In order to understand the FFVP and identify key food safety activities for the development of a logic model, the assessment team interviewed Agency staff at National Headquarters, and in the Ontario and Quebec Area offices which account for 36% and 15% of farm cash receipts for the agricultural sectors in Canada respectively ⁴. The team also met with industry stakeholders ⁵ to obtain a better understanding of the FFV community and its concerns within Canada. Interviews were also conducted with key partners within CFIA involved in FFV activities such as the FSIP which were involved in a special project between the FFVP, Provinces and industry stakeholders to mitigate recent food borne illnesses linked to sprouts.

4 A Profile of the Horticultural Sector and Associated Economic Contribution JRG Consulting Group, October 2005
5 Stakeholders include: consumers, wholesalers, retailers, producers, manufacturers, distribution centres and importers and their respective associations.

1.3 Fresh Fruit and Vegetables Industry and Consumption in Canada

7. The fresh fruit and vegetable industry in Canada comprises approximately 25, 000 domestic farms, some 80-100 minimal processors ⁶ , 800-1000 packers and re-packers, and 1000 wholesalers, food services and importers involved in interprovincial and international trade (see Appendix B). ⁷

8. The CFIA's proposed re-design of the fresh fruit and vegetable's food safety program directly and indirectly targets packers, minimal processors, wholesalers, importers and food services in a manner that influences the primary production sector to implement food safety provisions. This re-design would result in new CFIA activities for the middle segment of the industry and subsequently promote their implementation of controls such as:

• Good Agricultural Practices (GAPs);
• On Farm Food Safety Programs (OFFS);
• Memoranda of Understanding (MOUs); and
• and Good Importing Practices (GIPs).

9. The "Canada's Food Guide to Healthy Eating⁸" indicates that fresh fruit and vegetables can contribute significantly to the health and well-being of Canadians and recommends that these foods constitute a major component of their diets. In addition, there is a growing array of fresh-cut products, prepackaged salads, processed products, and imported produce available in the marketplace because of the increased globalization of the food supply. Imports account for 88% of the total volume of fresh fruit and 48% of the fresh vegetables, and originate from about 150 countries with the majority coming from United States, Mexico, the Caribbean, and other Central and South American countries. ⁹

10. According to Statistics Canada, fresh fruit and vegetables are the highest consumed commodities in the Canadian diet in terms of kg per person. Consumption patterns have increased from 34.64 to 37.59 kg of fresh fruit and 62.58 to 74.78 kg of fresh vegetables per person, per year (1976-2004) ¹⁰. Though traditional fruit and vegetables are still the main choice, tropical and foreign produce is becoming increasingly popular. Moreover, new vegetables are continuously being incorporated into the Canadian diet.

6 Minimally processed vegetables consist of raw vegetables that have been peeled, sliced, chopped or shredded prior to being packaged for sale in Canada.
7 Update to the draft Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2006.
8 Canada's Food Guide to Healthy Eating
http://www.hc-sc.gc.ca/fn-an/food-guide-aliment/index-eng.php, Health Canada, 2007-09-05
9 Draft Proposal for the Re-design of the FFV Food Safety Program, CFIA. 2005.
10 Food Statistics, 2004 Table 5, http://www.statcan.ca/english/freepub/21-020-XIE/21-020-XIE2004002.pdf, Statistics Canada, 2007-09-05

1.4 Food Safety Concerns Related to Domestic and Imported Fresh Fruit and Vegetables

11. Raw fresh fruit and vegetable commodities may contain safety hazards and if not controlled, could result in foodborne illness. A "hazard" as defined by the Codex Alimentarius Commission is a biological, chemical or a physical agent in or condition of food with the potential to cause an adverse health effect.

12. Over the past decade, agricultural practices have changed and new commodities from countries with diverse food safety infrastructures and controls have arrived, challenging the FFVP's current food safety activities used to assess potential health hazards. Many microbiological pathogens are ubiquitous in the environment and their presence on fresh fruit and vegetables are unavoidable. The control of organisms must occur throughout the food continuum including in the processing/packaging (fresh cut) environment, by preventing conditions favourable to their growth. There is increased evidence that microbiological hazards including bacteria, viruses and protozoans such as Escherichia coli O157:H7, Listeria monocytogenes, Shigella, and Cyclosporeae cayetanensis and Hepatitis A, have been implicated in FFV-related foodborne outbreaks in the past two decades. ¹¹ In addition, concern exists that certain fruit and vegetables are more prone to contamination based on production practices, physical characteristics, and extensive human handling. ¹²

13. Chemical contaminants associated with fresh fruit and vegetables including pesticides, environmental contaminants, metals, and natural toxins may have undesirable short-term or long-term effects on human health, such as contributing to chronic diseases (e.g., pesticide residues), and causing acute illnesses, including mortality (e.g., some natural toxins).

14. A CFIA-commissioned report ¹³ ranked the FFVP as a high priority for a re-design review to ensure adequate coverage of potential chemical hazards, such as pesticides, heavy metals, environmental contaminants, safe protective waxes and coatings, food additives (sulfites) and natural toxins.

Other report highlights comment on the FFVP's capacity to manage the concerns related to genetically modified fruit and vegetables, irradiation, nutritional labelling and oral allergy syndrome.

15. According to the CFIA's Annual Performance Report of 2004-2005, the compliance rates for FFV chemical residue testing conducted by the Food Microbiology and Chemical Evaluation Division (FMCED) were 97.6%, 98.9% and 99.8% in 2002-03, 2003-04, and 2004-05 respectively. The compliance rates for FFV microbiological testing conducted by FMCED were 93.7%, 99.8% and 99.2% in 2002-03, 2003-04, and 2004-05 respectively. Work plans typically call for approximately 10,500 chemical residue samples (2,500 domestic and 8,000 import produce) and 700 microbial samples (400 domestic and 300 import produce) per year. ¹⁴

11 "Foodborne Outbreaks in Canada Linked to Produce" A.M. Sewell & J.M. Farber, Journal of Food Protection, 2001
12 Results of the Ontario Fruit and Vegetable Microbial Baseline Study, Ontario Ministry of Agriculture, Food and Rural Affairs, 2005
13 CFIA Resource Review Program Re-design Needs and Priority, KPMG, 2002
14 FFV Food Safety Program Presentation, CFIA, 2006

1.5 Food-borne Illness Data

16. Between 1991 and 2000, Canada had 15 microbial foodborne outbreaks associated with fresh fruit and vegetables, resulting in 1,364 illnesses and one death. Thirteen of these outbreaks occurred between 1995 and 2000 (no data available for 2001-2004). In 2005-2006 there were four fresh fruit and vegetable related microbial outbreaks resulting in 900 illnesses respectively in Canada. Fresh produce implicated in foodborne illnesses include berries, cantaloupes, field tomatoes, green onions, fresh herbs, sprouts, and fresh-cut fruit and vegetables.

17. It is difficult to verify the link between FFV and foodborne illness due to limits in data collection and analysis both within CFIA and their stakeholders. Foodborne disease is greatly under reported for a variety of reasons such as: the ill individual does not seek medical help; the correct specimens may not be collected or discarded, and the appropriate tests performed; and the results may not be reported in a manner resulting in the cases being reported at the provincial or national level. As well, although CFIA and provincial health officials carry out investigations, the source of contamination and underlying causes may not always be identified. Increased overall consumption of FFV, along with a corresponding rise in the ready-to-eat raw fresh-cut produce, an increasingly diversified supply, and an ageing population may also contribute to an increase in foodborne illnesses related to FFV contamination.

2.0 Program Profile

2.1 CFIA Governance Structure

18. The CFIA is headed by a President, who reports to the Minister of Agriculture and Agri-Food. Executive Committee members from CFIA's Branches and Directorates are accountable for developing and delivering the Agency's policies, programming, and administrative functions. CFIA's Programs Branch consists of 14 Programs in animal health, plant protection and food safety, each responsible for program design and management. The FFVP, as one of the 14 Programs, falls within the Agri-Food Division of the Food Safety Directorate. The FFVP collaborates with other CFIA partners such as the Office of Food Safety and Recall (OFSR), the Food Microbiology and Chemical Evaluation Division (FMCED), and the Food Safety Division's, Food Safety Investigation Program (FSIP) on food safety activities such as those related to sprouts. The Operations Branch at the national and area levels is responsible for the delivery of the 14 Programs, primarily focussing on monitoring, compliance and enforcement activities. The CFIA's Science Branch conducts research, develops methods and provides scientific advice and analysis in support of all 14 Agency Programs. Scientists in CFIA's laboratories also analyse samples collected each year by CFIA inspectors.

2.2 FFV Program Description

19. The CFIA's FFVP includes within its scope: fresh fruit, fresh vegetables, fresh herbs, sprouts, fresh mushrooms, as well as fresh-cut fruit and vegetables ready-to-eat or not. Ready-to-eat (RTE) fresh fruit and vegetables are defined as those that have been washed and peeled, sliced, chopped or shredded prior to being packaged for sale and are intended to be consumed raw and not for further processing or cooking, e.g., shredded bagged lettuce, coleslaw, baby carrots, fresh-cut melons or fruit salad. Genetically modified organisms (GMO) are also included in the scope of the FFVP.

20. The Program excludes processed fresh fruit and vegetables in respect of a food product that is, canned, cooked, frozen, concentrated, pickled or otherwise prepared to assure preservation of the food product in transport, distribution and storage, but does not include the final cooking or preparation of a food product for use as a meal or part of a meal such as may be done by restaurants, hospitals, food centres, catering establishments, central kitchens or similar establishments where food products are prepared for consumption rather than for extended preservation.¹⁵

21. The FFVP activities primarily focus on national food safety monitoring where samples are taken and analysed for chemical and microbiological hazards associated with domestic and imported FFV commodities. Other FFVP food safety activities are reactive in nature and include investigation and follow-up on product violations, foodborne illness outbreaks, and consumer complaints. These activities are managed jointly by Programs Branch and Operations Branch under the FFVP.

22. The CFIA has the authority to establish regular inspection programs for the industry involved in interprovincial and international trade of FFV. Currently, no establishment inspection program exists with dedicated full time equivalents (FTEs).

23. Other FFVP activities include: collaborative initiatives with national stakeholders such as the On Farm Food Safety Programs (OFFSPs); development of qualitative risk assessments and risk management policies for specific import commodities (e.g., Guatemalan raspberries) and domestic sprouts (e.g., Code of Practice for the Hygienic Production of Sprouted Seeds); and development of consumer food safety fact sheets.

24. The FSIP enforces the safety and nutritional quality provisions of the Food and Drugs Act and Regulations (FDAR) via establishment inspections, food safety investigations (including consumer and trade complaint investigations), and risk management activities. FFVP and FSIP collaborate and investigate specific activities such as management of food safety hazards. If required, OFSR conducts FFV recalls. Science committees were established to ensure a risk-based approach to setting program priorities to risks identified on a horizontal basis across the Agency. The priorities and recommendations developed by the science committees are used to develop specific strategies and projects to manage food safety risks. According to the risk analysis report developed by the Joint CFIA-HC Science Committee(CFIA/HC Food Safety Science Committee Report, Microbiology, November 23-25,2005), fresh produce and sprouts are medium priorities based on their potential contamination with Shigella and Salmonella, respectively.

15 Definition taken from the Processed Products Regulations pursuant to the Canada Agricultural Products Act.

3.0 Logic Model

3.1 Narrative for the Logic Model

25. As noted previously, a logic model is a diagram that depicts the relationships amongst the main elements of a program, thus portraying the intended causal chain of events linking activities, outputs, and related outcomes. As the FFVP may be re-designed, the logic model depicts both current and proposed program elements, the latter identified in shaded boxes. Both current and proposed elements were used to formulate evaluation questions which can be found in section 4.0 Evaluation Matrix.

26. This narrative explains the activities, outputs and the intended outcomes for the FFVP and should be read in conjunction with the diagram of the FFVP logic model found in section 3.2. Both the logic model diagram and the narrative were developed in collaboration with CFIA's Audit, Evaluation and Risk Oversight (AERO) as well as FFV Program and Operations staff. For the purpose of this evaluation framework, the FFV logic model focuses on food safety elements for both domestic and imported fresh fruit and vegetables.

3.1.1 Inputs

Legislation

27. Fresh fruit and vegetables are covered by the following legislation:

• Canadian Food Inspection Agency Act (CFIA Act);
• Canada Agricultural Product Act and FFV Regulations (CAP Act & FFV Regulations);
• Consumer Packaging and Labelling Act and Regulations (CPL Act & Regulations); and
• Food and Drugs Act and Regulations (FDAR).

28. These Acts and Regulations give CFIA inspectors the authorities to inspect fresh fruit and vegetables and collect samples of domestic and imported produce. However, the Acts do not regulate primary production (growing and harvesting) of agricultural products as these generally fall under provincial and territorial jurisdictions. The FFV Regulations contain food safety provisions for post harvest activities and require that produce meets all pertinent provisions of the FDAR. As well, the Consumer Packaging and Labelling Act contain food safety provisions for the labelling of fresh fruit and vegetables. The Food and Drugs Act and Regulations applies to the importation and manufacture of all food sold in Canada.

Food Safety Policies Related to Fresh Fruit and Vegetables

29. Health Canada (HC), through its Food Directorate, has the statutory responsibility to establish policies and standards relating to the safety and nutritional quality of food sold in Canada. HC's Pest Management Regulatory Agency (PMRA) establishes Maximum Residue Limits (MRLs) for pesticides under the FDA. Health Canada also provides CFIA with health risk assessments (HRAs), recommendations on specific food safety issues, and letters of no objection for the usage of non-food products in the food industry.

CFIA is responsible for enforcing the food safety policies and standards that Health Canada sets, by establishing guidelines and codes of practice in collaboration with Health Canada, to reduce the risks associated with fresh fruit and vegetables (see section 3.1.2 for further information).

Partnerships with Provinces or with Foreign Governments

30. Partnership arrangements with provincial governments refers to activities to be conducted with respect to domestic fresh fruit and vegetables, for example sharing of sampling activities between CFIA and le ministère de l'Agriculture, des Pêcheries et de l'Alimentation du Québec (MAPAQ). Another example of an established partnership arrangement between Health Canada, CFIA and the provinces includes the development of the Foodborne Illness Outbreak Response Protocol (FIORP).

31. With respect to the FFV food safety activities, relationships with foreign governments, for example the United States Food and Drug Administration (USFDA) and the Servicio Nacional de Sanidad, Inocuidad y Calidad Agroalimentaria (SENASICA) are maintained and enhanced through ad hoc collaboration on emerging issues. Other collaborative initiatives include a certification program, Import Requirements for Mexican Cantaloupes, and the Import Policy for Guatemalan Fresh Raspberries and Blackberries.

3.1.2 Activities and Outputs

Develop/Review Inspection Materials

32. Inspection materials include but are not limited to food safety standards, inspection manuals and procedures, inspection and import directives, inspection work plans, and food safety training modules. Manuals and procedures are developed and updated by Programs Branch in consultation with Operations Branch (Areas) as well as key stakeholders and partners. For example, the Traceback Investigation Guidelines for Fresh Fruit and Vegetables was developed by Headquarters (HQ) Program in October 2001 and sent for review to CFIA Areas and Industry Associations in 2003. The FFVP in consultation with the Food Safety Division and Health Canada developed a Code of Practice for the Hygienic Production of Sprouted Seeds to assist sprout manufacturers in minimizing the risk associated with the consumption of sprouted seeds and beans. The FFVP also developed a Code of Practice for Minimally Processed Ready-to-eat Vegetables and a Manual on Fresh Fruit and Vegetables Sampling for Laboratory Testing.

33. The FFVP annual work plans (number of samples to collect and type of analysis required) are developed in HQ by the Food Microbiology and Chemical Evaluation Division (FMCED) in collaboration with the FFVP and the Lab Services Division (Science Branch) and then refined and organized by Areas and Regions. These work plans mostly consist of monitoring sampling plans but in the future FFVP re-design would include establishment (domestic and import) inspection plans. The FFVP work plans are developed for both domestic and imported products and are based on risk profiles and data such as compliance rates, non-compliant establishments, and availability of commodities in the marketplace provided by the Areas.

34. The FFVP makes recommendations on their training needs to the Human Resources National Training Unit (NTU) which provides funding for the delivery of the training across all Areas. For example, the Food Safety Division's, Food Safety Investigations Program (FSIP) worked in collaboration with the FFVP to develop and deliver a National Training Initiative (NTI) on Introductions to the Inspection of Sprouts Establishments. In addition, informal training is provided in the Areas/Regions via job shadowing and mentoring by senior inspectors.

35. Outputs for the activities above include standards, manuals, procedures, directives, work plans, and training modules.

Conduct Data (including Trend) Analysis

36. Data analyses are conducted using results from microbiological, chemical, and irradiation analyses, investigation reports, issue tracking sheets, international alerts, and food-borne illness reports for the purpose of revising work plans and also standards, manuals, procedures, and directives. These trend analyses are completed on an annual basis at the HQ level with input from the Areas/Regions. As a result of doing trend analyses, the FFVP developed import directives such as the Import Policy for Guatemalan Fresh Raspberries and Blackberries and the Import Requirements for Mexican Cantaloupes. In the future, results from establishment inspection reports would also be included to develop trend analysis.

37. The FSIP contributes to the roll-up analyses by providing results on sprouts sampling and analyses, and on establishment inspection activities.

38. The outputs of conducting trend analyses are revised work plans (including sampling plans) and also revised standards, manuals, procedures, and directives.

Conduct Product Sampling

39. In the four Areas (18 regions) across Canada, the FFVP operations inspectors collect samples of fresh fruit and vegetables in accordance with national sampling procedures and guidelines, in order to fulfill the monitoring sampling plans for microbiological analysis, chemical analysis, and irradiation analysis. Samples are taken at warehouses, wholesalers, and import establishments. The microbiological and irradiation analyses are conducted in CFIA laboratories while chemical analyses are done by a private laboratory. Currently, routine product monitoring is the prominent food safety activity of the FFVP. When a laboratory test reveals non-compliance, additional follow-up activities are initiated.

40. There are three levels of sampling activities for chemical and irradiation analyses in the FFVP - monitoring, surveillance and compliance based on historical produce data.

The monitoring phase is designed to gather data and provide information on the occurrence of chemical hazards and irradiated foods through random sampling of fresh fruit and vegetables. If the samples are found to be in violation, the product is put under the surveillance phase or follow up action is taken.

The surveillance (directed sampling) phase is conducted to confirm presumptive positive results and identify suspected problems. A specific commodity is targeted and samples from five shipments are collected and analysed. If all five samples are in compliance, the product is returned to the monitoring phase. If any one of the five samples is in violation, the product is placed on the compliance list.

The compliance phase is a regulatory control measure to prevent the sale and facilitate the removal from the marketplace of any product known to be contaminated and/or adulterated and/or in violation of standards. Shipments of the product on the compliance list are then tested at a recognized laboratory at the expense of the grower and shipper. The product will remain on the compliance list until five consecutive samples are in compliance. Then, the product is returned to the monitoring phase.

41. There are also three levels of sampling/inspection for microbiological contamination: monitoring, investigation and/or establishment inspection, and compliance.

42. The monitoring phase is designed to gather data and provide information on the occurrence of microbial hazards through random sampling of fresh fruit and vegetables. If the samples are found to be in violation, the product is put under the surveillance phase or follow up action is taken.

43. If a violation is found on an imported product, the FFVP will contact the foreign country's embassy to request that they carry out a follow up investigation in their country. In the case of an American or Mexican product, the FFVP will contact the USFDA or SENASICA directly.

44. If evidence leads the CFIA to believe that no action is taken in the country of origin or that the product still poses a food safety risk, an import policy is developed which may include a restriction on the imported product. Examples include the Import Policy for Guatemalan Fresh Raspberries and Blackberries and Import Requirements for Mexican Cantaloupes.

45. In the case of domestic produce an investigation/ establishment inspection is performed to determine the source of contamination and corrective measures are developed to minimize the risk.

46. The FSIP conducts sampling activities for sprouts to gather data on the occurrence of microbial hazards, confirm presumptive positive results, and remove unsatisfactory samples from the marketplace.

47. The outputs of conducting product sampling include product sampling reports and roll-ups as well as monitoring, surveillance and compliance lists. Such reports, roll-ups and lists are used to determine compliance rates and identify problematic producers or importers and high risk commodities. The data generated from these reports, roll-ups and compliance lists is used to conduct data/trend analysis and environmental scans, one of the FFVP's major activities. Compliance lists for fresh fruit and vegetables can be found on the CFIA website.

Conduct Establishment Inspections

48. Currently, FFV establishment inspections ¹⁶ take place only as part of investigations or through work specifications by the Food Safety Division. When a food safety risk is identified, the establishment must implement corrective measures to comply with CFIA's requirements. The FFVP does not require the industry to have HACCP systems in place or to be federally registered; however, some large retail and wholesale clients in the FFV industry require suppliers to have HACCP systems in place, as a condition of purchase.

49. Follow-up establishment inspections take place to verify Good Manufacturing Practices (GMPs) of the establishment. Following the inspection, effective corrective actions are requested by CFIA and sent to the establishment. CFIA does follow-up visits to ensure that the corrective actions are implemented.

50. According to the proposal to re-design the FFVP, food safety establishment inspections (domestic and importer verifications and audits) would be implemented. The proposed establishment inspection would consist of an in-depth evaluation of the establishment's food safety controls as well as its equipment and environmental conditions which would incorporate Good Manufacturing Practices (GMPs), Good Importing Practices (GIPs) elements and HACCP systems.

51. In the Areas, Food Safety Division inspectors conduct establishment inspections of production, packing and sorting establishments dealing with sprouts and pre-cut fresh ready-to-eat vegetables.

52. The outputs of this activity are establishment inspection reports and roll-ups. Such reports are used by FFVP to determine compliance rates and identify problematic importers, establishments and high risk commodities. Data generated from these reports and roll-ups are also used to conduct trend analysis and environmental scans, one of the FFVP's major activities. In the future, establishment compliance may also be used to determine inspection frequency.

Conduct Investigations and Non-compliance Actions

53. Food safety investigations are conducted as a result of product violations, food borne illness outbreaks, consumer complaints, and international advisories. Non-compliant test results or product violations (e.g., positive results for pathogens such as Salmonella spp and Listeria monocytogenes) lead to non-compliance activities such as follow-up sampling and testing, surveillance letters sent to establishments related to the non-compliant product, establishment inspections, and border lookouts. Follow-up inspections verify that corrective actions are implemented by the establishment and are effective. Repeated failure of an establishment to correct identified deficiencies can result in enhanced non-compliance actions including compliance letters and detentions until corrective actions are implemented.

54. International food safety issues, for example lettuce from the US found to be contaminated with perchlorate, also lead to food safety investigations. Likewise, a Hepatitis A outbreak in the US linked to Mexican green onions led to a food safety investigation by CFIA to ensure that similar products did not pose a health threat to Canadians.

55. When regulations or standards do not exist for a given food safety issue, (e.g., perchlorate in lettuce, cyclospora in raspberries) an investigation is conducted and FMCED's technical specialists may request health risk assessments (HRAs) from Health Canada. Further to HRAs, recalls of non-compliant products may be initiated by CFIA via the Office of Food Safety Recall (OFSR) or import directives may be issued.

56. The outputs of this activity include investigation and non-compliance follow-up reports, detentions, border alerts, HRA requests, import directives and recalls. Data generated from these reports are used to conduct trend analyses and environmental scans, one of the FFVP's major activities.

Develop and Provide Information and Advice for Consumers, Domestic Industry and Importers, Federal/Provincial/Territorial (F/P/T) Partners

57. The FFVP develops reference material for consumers, the domestic and import industry as well as for various government partners.

58. The FFVP at the HQ level, in collaboration with the Areas/Regions, stakeholders and partners such as Health Canada, develops fact sheets for dissemination to consumers. These include:

• Food Safety Facts for Fresh Fruit and Vegetables;
• Food Safety Measures for Cantaloupes;
• Sprouts Health Risks;
• Natural Toxins in Fresh Fruit and Vegetables;
• Oral Allergy Syndrome;
• Safety Tips for Black Widow Spiders in Grapes; and
• Why Sparks Occur when Cooking Vegetables in the Microwave

all of which can be found on the CFIA website.

59. In collaboration with Areas/Regions, stakeholders and partners, the FFVP develops food safety reference material for the industry such as:

• Code of Practice for Hygienic Production of Sprouted Seeds;
• Code of Practice for Minimally Processed Ready-to-Eat Vegetables;
• Submission Guidelines for the Acceptability of Novel Packaging Materials;
• Traceback Investigation Guidelines for Fresh Fruit and Vegetables; and
• Import Policy for Guatemalan Fresh Raspberries and Blackberries and the Import Requirements for Mexican Cantaloupes import directives.

60. In the re-designed FFVP, reference materials will be developed to assist industry in taking more responsibility for fresh fruit and vegetables' food safety.

61. The Agency also provides technical review feedback ¹⁷ to stakeholders on Good Importing Practices (GIPs) and On Farm Food Safety Programs (OFFSP). There are plans in the future to develop and provide information to better inform consumers, importers, and distributors on their responsibilities related to fresh fruit and vegetables.

62. Through establishment inspections, the Food Safety Division inspectors provide some training for the producers. In addition, Food Safety Division inspectors exchange information with their provincial counterparts (e.g., MAPAQ) in relation to sprouts.

63. In addition to publications such as fact sheets for consumers and reference material for industry (e.g., codes of practice), outputs can include GMPs and GIPs as well as information related to fresh fruit and vegetables provided through CFIA's Automated Import Reference System (AIRS).

16 Establishment includes packers and re-packers, minimal processors, wholesalers, importers and food services (see Appendix B)
17 Through Technical Review, CFIA's review team, comprised of commodity specialists, examine and assess the producer organization's on-farm food safety program to confirm that it is technically sound and adheres to Hazard Analysis Critical Control Points (HACCP) principles.

3.1.3 Reach

64. Although not included in the logic model graphic, reach is an important element of the model. Reach consists of the individuals and organizations affected by a policy, program or initiative, directly or indirectly. In the case of the FFVP, the reach includes partners and stakeholders such as Health Canada, provincial/territorial governments, CFIA's inspectors, industry associations, wholesalers, manufacturers, distributors, importers and Canadians who consume fresh fruit and vegetables.

3.1.4 Outcomes

65. The FFV outcomes are immediate (short-term), intermediate (longer-term) and ultimate societal (future) in nature. The immediate outcomes of the FFVP can be summarized as increasing the level of awareness and knowledge of food safety issues and best practices related to fresh fruit and vegetables by CFIA's Operations, industry, and domestic and import establishments as well as consumers. This increased level of knowledge and information will lead to the intermediate outcomes of better behaviour and practice related to the food safety of fresh fruit and vegetables.

3.1.5 Immediate Outcomes

66. The immediate outcomes are the intended results derived from key activities and outputs related to fresh fruit and vegetables, as stated in the logic model.

Inspectors have the necessary tools to carry out their activities

67. Inspection is the backbone of the FFVP system. To deliver inspection services, inspectors require technology to measure compliance of FFV with Acts/Regulations (including for example, effective performance measurement/management framework, timely data reports and analyses, decision-making capacity and tools). However, inspectors also require training (and associated tools); comprehensive, current standards (Acts/Regulations); manuals, procedures, directives, work plans; and appropriate resources (human, financial). Training provides inspectors the ability to perform their activities more efficiently and effectively.

Industry is aware of FFV food safety requirements

68. The development and update of key CFIA standards and manuals, monitoring product sampling, establishment inspections, surveillance and compliance lists, and other non-compliance actions are control measures expected to raise awareness amongst industry, of the fresh fruit and vegetables food safety requirements. The CFIA's intent is to develop a manual for the listing of requirements that the industry should be aware of.

Industry complies with requirements and adheres to good practices relevant to the Acts

69. Industry improved awareness subsequently leads to an industry that will assume more responsibilities toward the safety and compliance of fresh fruit and vegetables and one that adheres to food safety requirements and good practices relevant to the Acts described in the inputs section above. Industry compliance includes having a sound control system in place that includes important elements such as promoting compliance, identifying non-compliance and enforcing compliance.

Non-compliance enforcement activities occur to mitigate FFV food safety risk in the marketplace

70. When non-compliance actions do persist, enforcement activities are initiated. Enforcement activities include: warning letters, product detention, product removal from the marketplace and destruction of the product if necessary, issuance of a recall order when establishments do not cooperate, removal from Canada or disposal of the imported product, and prosecution(s). The FFVP has drafted an Enforcement Strategy which is currently under review and approbation by CFIA Enforcement and Investigation Services.

Consumers and partners have improved knowledge of fresh fruit and vegetables food safety

71. Establishment inspections, investigations and non-compliance actions contribute to the increase in food safety awareness of the industry. Also, information fact sheets for consumers, codes of practice, good manufacturing practices and good import practice for industry are expected to lead to industry awareness and compliance with FFVP's food safety requirements. Improved access to information for consumers and partners should improve knowledge of fresh fruit and vegetables food safety.

3.1.6 Intermediate Outcomes

72. All of the current and future activities (e.g., conduct product sampling ) conducted in FFVP, the outputs that they produce (e.g., product sampling reports and roll-ups) and the immediate outcomes (e.g., industry complies with requirements) that results are expected to lead to the following intermediate outcomes.

FFV Regulations reflect current international requirements and Canadian priorities

73. In Canada, the FFV regulatory approach is thorough and integrated such that the Canadian food safety system retains its current high standing on the world marketplace and that FFV stakeholders are actively engaged to provide safe FFV to the Canadian market.

74. With an enhanced compliance control system in place and better information sharing by FFVP on FFV food safety, Health Canada can develop and revise food safety standards, guidelines and policies related to fresh fruit and vegetables. Both organizations have the responsibility of ensuring the safety of food in Canada. Health Canada requires CFIA information related to FFV in order to develop food safety standards, policies and guidelines and in turn, CFIA will have access to these new and revised standards, policies and guidelines in a timely manner.

Strong stakeholders' confidence in the safety of FFV available in Canada

75. The totality of all of the above activities and outcomes related to fresh fruit and vegetables food safety will lead to consumers, industry and government being confident in the safety of FFV available in Canada. Overall, Canadians appear to be confident in the food safety system. Despite some food safety concerns, recent results indicate that this confidence has strengthened as a result of the way in which the Mad Cow Disease situation has been handled.¹⁸ Canadians were asked to name items that would make them feel better about the safety of food in Canada and suggestions included having more detailed food labels (34%), providing more information about food safety (10%), reducing/eliminating pesticide use (10%), and improving the regulatory system (8%)¹⁹.

Increased access to interprovincial and international FFV markets relative to pre-re-design of FFVP

76. The FFVP is seeking this outcome as part of its mandate to improve market access to Canadian FFV not related to food safety but rather quality grading. The rationale for this is that, increased industry awareness of food safety will lead to increased industry compliance which would allow more domestic and imported FFV products to be available for consumption. This in turn, would allow the public and industry to have increased access to diversified, nutritious and safe fresh fruit and vegetable products throughout the year.

Increased stakeholder implementation of FFV food safety practices relative to pre-re-designed FFVP

77. Increased industry awareness of and compliance with fresh fruit and vegetable requirements along with better information about FFV food safety provided to consumers and partners will lead consumers and industry to adopt effective FFV food safety practices. A recent study asked consumers about the food safety information available to them. Almost one quarter (23%) said they would like more information on food processing issues, such as food safety in general processing, and Escherichia coli and Salmonella spp. contamination. Supplying consumers' information on the processes and practices of establishments and individuals involved in food production would likely have a positive impact on their perceptions of food safety and quality, provided, of course, that the information demonstrates the positive actions taken to manage quality and safety risks.²⁰

18 Consumer Perceptions of Food Quality and Safety. Agriculture & Agriculture Canada. Ipsos Reid Corporation, November 2004.
19 Public Security Monitor, Wave VIII, EKOS Research Associates, March 2005.
20 Consumer Perceptions of Food Quality and Safety. Prepared for Agriculture & Agriculture Canada. Ipsos Reid Corporation, November 2004.

3.1.7 Ultimate Outcome

78. All elements in the logic model (activities, outputs, immediate and intermediate outcomes) are expected to lead to the ultimate outcome: Protection for the people of Canada from preventable health risks related to fresh fruit and vegetables. Along with CFIA, there are several other organizations that contribute to this ultimate outcome, however, the FFVP can play a major role in achieving this outcome by enhancing its food safety activities and the resulting impacts.

3.2 Logic Model

4.0 Evaluation Matrix

79. The evaluation matrix outlines the key evaluation questions, along with indicators, data sources and methodologies to address the questions. The components of the evaluation matrix are defined as follows (refer to section 4.2 for table):

Evaluation Question: A question that needs to be answered in order to address the broader evaluation issue. The major purpose of the evaluation question is to provide data on the characteristics and merits of a program (see section 4.1 for more detail). The evaluation questions in the framework will explore CFIA's risk management strategies and food safety activities, and possible linkages with human health outcomes, as well as Health Canada's policies and standards.

Indicator Construct: Provides a description of one aspect of the evaluation issue or concept to be pursued, for example, outputs in a logic model.

Indicator Variable: A quantifiable indicator under an Indicator Construct, which is the means for obtaining concrete measurement information on the concept, issue, or indicator construct, of interest. These indicators provide a quantitative or qualitative measurement of the extent to which the program is achieving its outcomes. Indicator variables must be simple, measurable, accurate, reliable (over time), relevant (to the audience and to what is being addressed), and timely.

Data source: The source of the information used to address the evaluation question. In evaluation, several sources of data are used to answer the same question to ensure reliability.

Methodologies: The methods used to collect and analyse the information to answer the evaluation question (see section 4.3 for more detail).

4.1 Evaluation Questions

80. The FFVP logic model provides an outline of CFIA's activities and outputs of the program as well as the expected outcomes. This logic model was used to formulate the evaluation questions identified in four categories that are relevant to the assessment of domestic and imported fresh fruit and vegetables.

• Rationale/Relevance: These questions focus on the relevance of the program or activities related to fresh fruit and vegetables, and their objectives related to CFIA's overall mandate. They ask whether the program or activities are still needed (e.g., to what extent do they continue to respond to federal and CFIA/ Health Canada priorities?). The questions also discuss the potential food safety risks and hazards which can affect human health.
• Program Design: This focuses on how the design of the program supports its performance. Questions include food safety activities, outputs (things that are tangible) and outcomes (longer term developments), their ongoing management and design, and linkages with science based and risk management strategies.
• Program Delivery: Questions consider food safety activities described in the design with a focus on the effectiveness of their implementation, (e.g., consistency of delivery with the Program design).
• Outcomes: These questions address the intended outcomes and effects that are identified in the logic model, with a focus on health and safety components. The questions also explore the extent to which the program or activity is achieving its intended outcomes (how successful it is) and whether there are any unintended outcomes.

81. For the purposes of the evaluation questions in section 4.2:

• FFVP staff includes: FFV inspectors, FFV Area Program Specialists, Inspection Managers, National Headquarters (HQ) Work plan Coordinator, National FFV Program Specialist, Import Chief.
• Partners include: Area Import Coordinator, Import Service Centre representative, National Import Operations Manager, Food Safety Investigation Program, Office of Food Safety and Recall (OFSR), Food Microbiology and Chemical Evaluation Division (FMCED) and the Science Branch.
• F/P/T partners include: Health Canada (HC), Canadian Border Services Agency (CBSA), Public Health Agency of Canada (PHAC) (National Microbiology Laboratory (NML)/ Laboratory for Foodborne Zoonoses (LFZ)), Pest Management Regulatory Agency (PMRA), Agriculture and Agri-Food Canada (AAFC), and Provincial/ Territorial counterparts.
• Stakeholders include: consumers, wholesalers, retailers, producers, manufacturers, distribution centres and importers and their respective associations.

82. The following nine evaluation questions listed in the Evaluation Matrix Table that follows (section 4.2) are proposed as possible evaluation questions for a future FFVP evaluation. Before implementing the evaluation, the BFSA evaluation team will assess the currency of the following questions to identify which ones can best address the evaluation needs and may add others in response to issues that may have arisen after the completion of this framework.

4.2 Evaluation Matrix Table

4.3 Evaluation Methodologies

83. The Evaluation Matrix Table outlines the evaluation questions, specific indicators, data sources and the methodologies that may be used to answer the evaluation questions. The matrix table provides a broad range of methodologies, both qualitative and quantitative in nature to be selected (based on information availability and feasibility) by the BFSA assessment team as part of a future evaluation of FFVP. Specific methodologies are chosen by the BFSA team in consultation with CFIA, as their priorities and available resources may influence the choice of methodologies to be used in the future evaluation. In collaboration with the FFVP, the BFSA team also used some of these evaluation methodologies in developing the FFVP logic model.

84. A range of methodologies could be undertaken as part of a future evaluation and may include:

• Document Review: This includes the review of documents that would be useful in exploring the evaluation questions. These documents could include the FFV program manuals, program output and data records, performance measurement reports, training manuals, and other pertinent information from the Agency. It also includes legislation, the CFIA Import Policy, FFV Products policy, FFV inspection procedures and FFV guidelines and standards that apply to the FFVP's import activities, including CODEX documents. The assessment team may also review documents related to the management of food safety risks which could include sampling plans, guidelines, laboratory methods, and risk assessments.
• Environmental Scans: This includes conducting scans on food safety hazards, emerging issues, and foodborne illnesses. Environmental scans can also obtain information from internet sites on the regulation of FFV imports.
• Literature reviews: This includes reviewing academic and scientific papers pertinent to the assessment.
• Interviews: The following groups have been identified for interviews: CFIA HQ's Managers, Area Managers, partners and stakeholders. The interviews at Headquarter levels are primarily with those staff who works in the FFV Program, the Import Sector, and the Office of Food Safety and Recall. It also includes consultation with HQ's Technical Specialists and the National Coordinator of Operations. In the Areas, interviews will be conducted with Import Service Centre Managers, Area Operations Coordinators, FFV Program Network Specialists, Regional Inspection Managers and Inspectors (including multi commodity inspectors with FFV responsibilities), and Program Network Directors and Chiefs. External to CFIA, interviews will be conducted with Health Canada, Agency partners with responsibilities for import control such as Canada Border Services Agency (CBSA) and Department of Foreign Affairs and International Trade, academia, FFV importers and industry including their associations, and possibly Provincial representatives.
• Focus Groups: This involves interview techniques and open discussion with a selected small group of individuals. The small group can be selected from within one of the larger groups or can be in any number of combinations from the following: CFIA HQ Managers; FFV specialists and inspectors; and industry representatives.
• Trend Analysis: This involves the assessment team's review of existing CFIA's trend analyses on such issues as the compliance rate of domestic and imported FFV, or the team conducting a trend analysis exercises to obtain information.
• Case Studies: The assessment team may conduct some case studies in order to obtain the domestic and importers' perspectives of CFIA's FFVP and related activities and to determine their understanding of documented guidelines such as Good Manufacturing and Good Import Practices. Focussed questions will be posed to these stakeholders and their responses will be summarised.
• Surveys: Industry and FFVP staff survey questions will be developed in collaboration with FFVP management and may include assistance of services from a recognized authority in survey design (e.g., from Health Canada or Statistics Canada). The method of survey delivery can be through mail or electronic means such as e-mail. Information gathered from the surveys will be used to answer questions related to program rationale / relevance, design, and impacts /success.
• Review of Databases: This involves the examination of any database information that is available and would benefit the assessment team in answering the evaluation questions. Some of the databases that are important to the team include: the Incidence Management System database which contains incident and recall information and the Import Retrieval System. The team will also review information from the disease surveillance database that is maintained by the Public Health Agency of Canada.
• Baseline Study: This involves the comparison of historical (longitudinal) data which has been collected before the re-design of the FFVP, with data collected after its re-design.

5.0 Evaluation Challenges

85. The following section outlines some challenges that may have an impact on the future evaluation of the FFVP. Many of these challenges relate to food safety data gaps and potential barriers or limitations in assessing the impacts or successes of the FFVP.

5.1 Food Continuum and Legislative Mandate of the Program

86. As shown in Appendix B, CFIA's roles and responsibilities primarily fall in the part of the FFV production continuum that deals with the FFV processing, import and distribution establishments. Although the pre- and post- processing steps are not under the jurisdiction of the CFIA, the Agency's desires to develop strategies to mitigate risks and reduce existing gaps in the FFV food safety continuum may result in a review and/or intervention of these areas.

Challenge:
Due to the multi jurisdictional nature of the FFV food continuum, the evaluation will require a thorough understanding of the dynamics between CFIA and its stakeholders.

5.2 Food Safety Risks

87. An evaluation of the FFVP will take into consideration both quantitative and qualitative data to reach food safety and evidence-based conclusions. Several factors impact on FFV food safety. The food safety risks vary between the FFV commodities, the growing environment and the domestic and imported production methods, all of which may require specific interventions to reduce the presence and level of hazards (e.g., on-farm food safety programs (OFFSP)).

Challenge:
Specific risk profiles for each FFV commodities and processing methods are currently incomplete and this is a challenge for the Program planning. Thus, the evaluation challenge would be in analysing the Program's rationale, logic, and successes in mitigating food safety risks.

5.3 FFVP Food Safety Data

88. Currently, the FFVP is mainly focussed on the collection of product quality inspection data based on Canadian and international trade requirements. Food safety data is limited to test results of sample analyses, complaint investigation reports, and ongoing specific food safety issues. In the re-design of FFVP, the proposed food safety activities include establishment inspections and increased collaboration between the industry and F/P/T partners. These changes would generate useful food safety data for evaluating the Program's effectiveness. In addition, the Program is establishing performance measurements as part of a CFIA wide performance framework implementation. This performance data will not only support food safety trend analyses and management decision-making, but will also provide useful evaluation information on the program's delivery and impacts.

Challenge:
Limited food safety data is currently available on chemical and microbiological testing. It is unlikely that an evaluation team could formulate useful food safety conclusions using the current level of data.

5.4 Capacity to Conclude on Industry Adherence to Food Safety Standards

89. Currently, the FFVP has limited documentation on the overall industry (processing establishments, distribution centres, wholesalers and importers) adherence to FFV food safety requirements. The Program's information is limited to non compliant products follow up investigations in domestic establishments. This results in data that does not represent the entire FFV industry.

Challenge:
It would be difficult to measure the impact of the Program without a complete and comprehensive set of food safety data that represents all of the FFV industry within CFIA's jurisdiction.

6.0 Evaluation Strategy

90. The following section proposes considerations for an eventual evaluation of the effectiveness of FFVP.

91. As FFVP is planning a re-design (more focus on food safety), BFSA recommends that an evaluation be conducted one to two years after the implementation of the re-design plan. The HC- CFIA Assessment Advisory Committee will review the feasibility of conducting the FFVP evaluation and the date of the evaluation in the next planning cycle. The evaluation's objective would include providing feedback to senior management about the impact of the re-design on the safety of FFVs available in Canada. Such evaluation could then utilize performance data collected by the Program and would focus among other things on comparing pre and post re-design food safety data and measure early outcomes of the re-designed FFV Program.

92. Findings and recommendations are then expected to be used to provide CFIA with evidence-based advice and guidance to help improve their food safety-related activities.

Appendix A - Terms of Reference

Purpose
1. The purpose of this terms of reference is to outline the work to be performed by the Bureau of Food Safety Assessment (BFSA), Health Canada, and to describe the scope and approach of the evaluation project of CFIA's food safety activities related to domestic and imported fresh fruit and vegetables.

Objective of Evaluation Project
2. The objective of this evaluation project is to develop an evaluation framework, which is to include logic models for both the current Fresh Fruit and Vegetable Program (FFVP) and the proposed / possible future FFVP food safety activities. Through the development of the evaluation framework this evaluation project will contribute to the re-design of the FFVP, a possible future evaluation of a re-designed FFVP, and provide advice and feedback to HC on its food safety policies and standards.

Background
3. In September 2003, the joint HC-CFIA Food Safety Assessment Advisory Committee identified domestic fresh fruit and vegetables as an evaluation priority for 2004 (see the original proposal in Appendix 1). BFSA began pre-evaluation work and environmental scanning in May 2004, paused during the summer due to mutual program resource pressures, and restarted work in October 2004.

4. In February 2005, CFIA requested that the Advisory Committee review the proposed evaluation due to an upcoming re-design of the FFVP. The Advisory Committee met on March 1, 2005 and decided that BFSA should adjust the evaluation project to take the re-design into account. The Advisory Committee also decided that the assessment should provide timely advice in the re-design of CFIA's domestic and imported fresh fruit and vegetable's food safety activities.

Program Description
5. Under the Canada Agricultural Products Act and the Consumer Packaging and Labelling Act, the FFVP ensures that fresh produce, which is imported, exported and marketed domestically under federal standards is safe, meets standards, and is packaged and labelled to avoid fraud and marketed in an orderly fashion.

6. The major activities undertaken by CFIA under this program include: sampling of fresh fruit and vegetables, inspecting of products for standardized container sizes, verifying of industry grading accuracy to established standards, and inspecting at points of destinations for defects in condition and appropriateness of the grade.

7. Outside of the FFVP, CFIA also conducts food safety activities related to domestic and imported fresh fruit and vegetables such as recalls, food safety investigations and enforcement actions, as provided under the Food and Drugs Act.

Scope of the Evaluation Project
8. The evaluation project will consider the relevance, design, implementation and food safety outcomes of the FFVP as well as all other CFIA activities related to domestic and imported fresh fruit and vegetables that are outside of the FFVP. Possible food safety outcomes for this project could include: effectiveness of processing and monitoring controls for microbial hazards (pathogens and viruses); effectiveness of processing and monitoring controls for chemical hazards (natural toxins, pesticides, food additives, heavy metals); and extent of CFIA's participation in on-farm food safety initiatives (as mentioned in Appendix 1).

9. Fresh fruit and vegetables for this evaluation project include: fresh fruit, fresh vegetables, fresh mushrooms, fresh herbs, sprouts and fresh-cut produce consumed in Canada.²¹ This includes irradiated and fumigated fruits and vegetables, and fruits and vegetables contained in modified atmosphere containers / packages. It does not include products which are processed (such as nuts) or where ingredients such as vinegar are used.

Methodology
10. The methodology for this project includes the following activities, which can be useful for program re-design and planning as well as future evaluation and performance measurement:

• document and literature reviews and interviews with CFIA staff from Headquarters, Areas and relevant stakeholders (e.g., provinces, industry associations, etc.) as required;
• the development of logic models (for current and proposed FFV food safety activities) in a collaborative workshop with HC and CFIA representatives;
• the development of an evaluation framework, which will include evaluation questions and issues, and performance measurement criteria and indicators (see appendix 2);
• the provision of advice, guidance and recommendations for FFVP and other food safety activities outside of FFVP that are related to fresh fruit and vegetables; and
• drafting of the final evaluation framework.

Roles and Responsibilities

11. The evaluation project will be conducted by BFSA in consultation with CFIA and relevant stakeholders as required. A joint HC-CFIA Food Safety Assessment Committee for this project was established in the fall of 2004. The committee facilitates the process during the course of the evaluation project and provides appropriate representation and input (see Appendix 3 for list of committee members).

12. The specific activities of the CFIA - HC Food Safety Assessment Committee include:

• reviewing the terms of reference (which includes the scope and approach of the evaluation project);
• supporting the evaluation project during its duration by expediting the process, providing guidance and resolving outstanding issues encountered;
• recommending the creation of working groups as needed;
• reviewing the evaluation framework, including the evaluation questions and strategy, and determining the appropriate evaluation option if several are presented; and
• recommending the approval of the logic model and evaluation framework to the Advisory Committee.

13. Given CFIA's re-design of the FFVP, the Advisory Committee will also review and approve the terms of reference for this evaluation project.

Time Frames and Deliverables
14. Work will commence upon acceptance of the terms of reference by the Assessment Committee and the Advisory Committee. An evaluation framework and logic models will be delivered in 2006, for approval by both committees and for publishing on Health Canada's website.

Meetings and Travel
15. BFSA will develop a Work plan for interviews and travel plans, and will coordinate with CFIA. These plans will allow CFIA to identify headquarters and regional area contacts and make appropriate arrangements well in advance.

Approved