Overview and Importance of Evaluation
About this guide
In order to ensure that consumer food safety education programs are effective in achieving their goals and preventing foodborne illness, it is important that these programs incorporate a rigorous and thorough program evaluation. In addition, planning of interventions must be strategic, evidence based, and tailored specifically for target audiences. This resource was created to serve as a planning and evaluation guide with tips, tools, and examples to help consumer food safety educators develop and evaluate their programs and activities.
Background: foodborne illness and food safety
Foodborne illness is a serious public health problem in the United States. There are approximately 48 million new cases each year, causing 128,000 hospitalizations and 3,000 deaths [5,6]. Cases of foodborne illness come with heavy economic costs, totaling approximately $51-$77.7 billion each year [6]. These costs include medical and hospital bills, lost work productivity, costs of lawsuits, legal fees, and a loss of product sales [1,3]. Foodborne illness also causes emotional tolls and burdens on family members when caring for friends and relatives or when experiencing the loss of a loved one [3].
To prevent microbial contamination at various stages of food production, the U.S. Food and Drug Administration’s (FDA) Food Safety Modernization Act (FSMA) was signed into law in 2011. The new law provides mandates that require comprehensive and science-based preventive controls across the food supply. Through FSMA, the government and food industry are working together to create a new prevention-based food safety system.
Microbial risk due to improper food preparation and handling by consumers in the home is another preventable cause of foodborne. Health educators across the Nation have engaged in food safety education activities focused on the consumer. Health and consumer food safety educators work to increase the public’s awareness and knowledge about food safety practices and the risk of foodborne illness, and to promote safe food handling practices. However, there are several areas of concern related to consumer food safety education interventions. Gaps include a lack of rigorous and evidence-based evaluation of educational program activities and a need to improve research designs by ensuring that interventions focus on the specific needs of the target audience and address common influencers of consumer food safety practices, such as specific knowledge, perceived susceptibility, and access to resources [2]. Consumer food safety educators can use this guide to learn about how to increase evaluation efforts and adopt more rigorous and thorough evaluation approaches that will help them improve their programs.
Why evaluate?
Evaluating your program is important and beneficial to its overall success. For example:
- An evaluation can help you identify the strengths and weaknesses of your program, learn from mistakes, and allow you to continuously refine and improve program strategies.
- By using evaluation data to improve program practices you can ensure that resources are expended as efficiently and effectively as possible.
- Evaluation data can provide program staff with valuable insight to help them understand the impact of the program, the audience they are serving, and the role they can play to contribute to the program’s success.
- There is no way to really know what kind of effect your program or activities are having without a program evaluation.
- Evaluation data can show you how successful your program is in preventing foodborne illness and promoting safe food handling practices.
- Conducting an evaluation can help you monitor the program and ensure accountability.
- Sharing what you learned from the evaluation with other consumer food safety educators can help them design their own programs more effectively.
- Having documentation and data that show how your program works can help you receive continued or new funding. Evaluation data is usually expected when applying for grants.
- Demonstrating that your program has an impact can help increase support of activities by other researchers, educators, and the greater community.
- Showing the target audience how effectively your program works can help increase interest and participation.
Evaluation standards
The Joint Committee on Standards for Educational Evaluation (JCSEE) has identified five attributes of standards to ensure that evaluations of educational programs are ethical, feasible, and thorough. The five attributes, Utility, Feasibility, Propriety, Accuracy, and Evaluation Accountability, include a list of standards that are important to consider when evaluating your program.
Below are brief descriptions of each attribute and examples of the standards [4,7,8] –
Utility standards help ensure that program evaluations are informative, influential, and provide the target audience or stakeholders with valuable and relevant information. These standards require evaluators to thoroughly understand the target audiences’ needs and to address them in the evaluation.
– Examples of standards related to Utility are – Attention to Stakeholders, Explicit Values, Timely and Appropriate Communicating and Reporting, and Concern for Consequences and Influence.
Feasibility standards are intended to ensure that evaluation designs are able to function effectively in field settings and that the evaluation process is efficient, realistic, and frugal.
– Examples of standards related to Feasibility are – Project Management, Practical Procedures, and Resource Use.
Propriety standards protect the rights of individuals who might be affected by the evaluation and support practices that are fair, legal, and just. They ensure that the evaluators are respectful and sensitive to the people they work with and that they follow relevant laws.
– Examples of standards related to Propriety are – Formal Agreements, Human Rights and Respect, Clarity and Fairness, and Conflicts of Interest.
Accuracy standards ensure that evaluation data, technical information, and reporting are accurate in assessing the programs worth or merit.
– Examples of standards related to Accuracy are Justified Conclusions and Decisions, Valid information, Reliable Information, and Explicit Evaluation Reasoning.
Evaluation Accountability standards encourage sufficient documentation of evaluation processes, program accountability, and an evaluation of the evaluation, referred to as a meta-evaluation. This provides opportunities to understand the quality of the evaluation and for continuous improvement of evaluation practices.
– Examples of standards related to Evaluation Accountability are – Evaluation Documentation and Internal Meta-evaluation.
For a full list of the program evaluation standards and to learn more about them: http://www.jcsee.org/program-evaluation-standards-statements.
Keep these standards in mind as you plan your program and the evaluation. Remember, it is not only important to evaluate your education program, but also to think about the quality of your evaluation and to ensure that it is ethical, accurate, and thorough. In addition, a key factor for having a successful evaluation is to think about these standards and plan your evaluation as you design and before you implement your program, not after.
In Summary,
when thinking about evaluation and how to apply what you learned in this chapter to your program you may want to ask:
- How would a program evaluation benefit or help improve my program?
- Can I improve or increase current program evaluation efforts?
- What can be done to ensure my program evaluation is as ethical, feasible, and as thorough as possible?
- How will I ensure that evaluation planning is addressed during program development and before program implementation?
References
- Buzby, J., Roberts T, Jordan Lin CT, Roberts, T, & MacDonald, J. (1996). Bacterial foodborne disease: medical costs and productivity Losses. United States Department of Agriculture –Economic Research Service (USDA–ERS). Agricultural Economics Report No. AER741. Retrieved from: http://www.ers.usda.gov/media/755624/aer741_1_.pdf
- Food and Drug Administration (FDA). White Paper on Consumer Research and Food Safety Education. (DRAFT).
- Nyachuba, D. G. (2010). Foodborne illness: is it on the rise? Nutrition Reviews, 68(5), 257-269.
- Sanders, J. (1994). The program evaluation standards: how to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage.
- Scallan, E, PM Griffen, FJ Angulo, RV Tauxe, & RM Hoekstra. (2011). Foodborne illness acquired in the United States-unspecified agents. Emerging Infectious Diseases. 17, 16-22.
- Scharff. RL. (2012). Economic burden from health losses due to illness in the United States. Journal of Food Protection. 75, 123-131.
- The Joint Committee on Standards for Educational Evaluation (JCSEE). (2016). Program evaluation standards statements. Retrieved from: http://www.jcsee.org/program-evaluation-standards-statements
- Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.