Community Action Resources for Inuit, Métis and First Nations: Evaluating

1998

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Table of Contents

• Introduction to the Kit
• I. Introduction to Evaluating
  • Evaluation is a natural process
  • A step-by-step model
  • Programs and puzzles
  • Training goals
  • Training objectives
  • Approach
• II. What is Evaluation?
  • Definition
  • Two types of evaluation
    1. Outcome evaluation
    2. Process evaluation
  • Evaluation in the life of a program
• III. Why Evaluate?
  • Reasons why we avoid program evaluation
  • By not evaluating, you may be missing out
  • Reasons why we should evaluate
• IV. How to Evaluate?
  • Evaluation can be easy to do, yet systematic
  • Common sense rules
  • A five-step model
    1. Setting the context of the evaluation
    2. Preparing an evaluation plan
    3. Gathering the information
    4. Making sense of the information
    5. Using the results
  • An effective evaluation
• V. Human Aspects Of Evaluation
  • People's feelings
  • Resistance
  • Confidentiality
  • The ideal evaluator
• Appendices
  1. Evaluation goals and objectives chart
  2. Methodology chart
  3. Evaluation timetable chart
  4. Ongoing evaluation report
  5. Five-step model

Introduction to the Kit

When asked "What is a community?", Aboriginal people often answer "A community is a group of people sharing and caring."

This series of manuals is about just that: sharing information and skills to empower people so they can better care for their communities.

Training in community development and program evaluation has been identified by Health Canada and by national Aboriginal organizations as a priority need.

In order to fulfil this need, training workshops were offered between 1993 and 1995 to approximately 300 Aboriginal people across Canada, most of them front-line health workers. The training was based on a generic package of resources on health promotion and community development called the Community Action Pack.

Experience and feedback received from the participants during the workshops and after they returned to their communities led to the development of this new kit.

Community Action Resources for Inuit, Métis and First Nations was developed specifically for Aboriginal people, taking into consideration their values, culture and way of life.

This kit is a self-help tool for Aboriginal people who want to get a community development project off the ground.

The series of six manuals contains information, tips, examples and ready-to-use charts that you can copy and use for yourself or to train others in your community. It is intended to be user-friendly, emphasizing questions such as what, why, when and how to accomplish the different steps of a community development project.

Community Action Resources for Inuit, Métis and First Nations uses a holistic approach, taking into consideration community development know-how and context as well as the human aspects that are part of the entire process.

Look for the following symbols. They will help you apply your skills and knowledge.

tips

examples

exercises

What is a community?

Let's take a moment to think about the word "community". Communities are not just a question of geography. People in a volley-ball league can be a community. A support group can be a community. Members of a church can be a community. Communities are people with something in common, something that is important to them. The members of a community depend on each other as they work together to accomplish challenging tasks. A community can be seen as a group of people:

• living in the same geographical area;
• sharing the same culture;
• with a common interest or concern; or
• with a shared sense of identity.

Community Development: Making Communities Healthier

The World Health Organization and Health Canada have defined health as more than the absence of illness and disease. Health is a sense or state of physical, emotional and psychological well-being. Health is an individual or group's ability to reach goals, to satisfy needs, and to cope with or change their environment. In a community context, it is the ability of a community to sustain itself in a caring and fulfilling way. In other words, health means looking after ourselves and others. It is promoted through having access to services appropriate to our needs. It is enhanced by living in a clean environment and by a spirit of community (feeling part of a group of people who care about each other).

This means that groups that are concerned with the kind of housing available in their community, reducing violence, developing the local economy or starting a self-help group for bereaved parents can be seen as improving and promoting the health of their community.

A step-by-step Process

Is there a secret to turning a good idea into a successful project? Actually, there is no secret. Groups follow different paths to action, yet experience shows that successful efforts use some "tried and true" steps.

Most groups go through the steps of identifying needs, planning activities, finding volunteers and funding, evaluating and helping people work together through the normal ups and downs of an activity. Community Action Resources for Inuit, Métis and First Nations is built around these steps.

Starting a new activity is like planning a trip to a place you have never been.

Just as you need a map for your trip, you also need a map for your community activity. This map will help you figure out where you are, what has been accomplished, and point out what still needs to be done.

The following diagram is your map to the steps for community action that is described in Community Action Resources for Inuit, Métis and First Nations. Each circle represents a step in the process of developing an activity.

The interpersonal aspects of community action and the techniques needed to carry out a project, such as taking minutes in meetings, have been gathered together in the "Toolbox". This circle has been placed in the middle of the diagram because the "Toolbox" can be useful in all steps of the process.

It is important to remember that every group follows its own path, which may be different from the one shown here. There is no absolute "right" way of doing things. You need to go with what works best for you.

The process described in Community Action Resources for Inuit, Métis and First Nations is intended to help you plan activities, not to tell you what to do. The diagram is like a map and you can take as many side trips as you like. There are no rules!

I. Introduction to Evaluating

Evaluation is a natural process

Have you ever felt very proud of a program, without really being able to answer the question: "Why did it go so well?"

Or have you ever wondered: "Although everyone involved in the program seems to be doing their best, why is it just not doing as well as it could?"

If you have asked yourself these or similar questions, you have already taken a first step toward evaluation. Evaluation comes naturally. You do it every day!

Evaluation means finding out why a program is successful in order to have the ability to repeat positive experiences. It also means trying to understand what needs to be improved to avoid negative experiences.

A step-by-step model

Although evaluation comes naturally, it is helpful to structure it using a step-by-step model. This way, all the important aspects are included, the results are accurate and the conclusions are relevant to your organization.

In this training, we will introduce you to a five-step model. It can help you evaluate effectively to meet your needs. The model is flexible and can be adapted to your working environment.

Programs and puzzles

This training will show that evaluation can help you piece together the overall program.Sometimes, we lose sight of this picture due to the demands of our everyday activities. When you evaluate, you step back from your everyday concerns to get a clearer picture of what is happening. Evaluation helps keep the program on target. It also helps you make informed decisions.

We feel that programs and puzzles have more in common than the letter "p". In this training, we will look at how programs and puzzles are alike.

Getting started

When you start to piece together a puzzle, you know your goal: to copy the picture on the box. In the same way, when you begin to plan a program, you need to know its goalsandobjectives.

Along the Way

When working on a puzzle, you try to put the pieces together correctly, one by one. You check the picture regularly, to make sure the two match. For a program, it is also useful to refer back to the objectives regularly to make sure you are on the right track. Comparing the "ideal picture" with the "real picture" is referred to as "ongoing" or "process" evaluation. It helps you understand what is going on in the program and how it is operating.

Finishing

Once you have put all the puzzle pieces together, you compare the result with the picture on the box. Congratulations, you did it! They are the same.

At the end of a program, you can also compare the results with your goal. This is called "outcome" evaluation. It looks at the outcome or how things progressed.

This training will look at both process evaluation and outcome evaluation.

Training goals

This training will help you to feel more comfortable evaluating programs in your community and to help you use a model that is easy to use.

Training objectives

By the end of this workshop, you should:

• understand the importance of evaluating and the consequences of not evaluating;
  
• be able to use a five-step program evaluation model;
  
• be able to develop and use simple evaluation tools;
• be able to prepare an evaluation plan to evaluate a program in your own working environment; and
  
• be sensitive to the interpersonal relations involved in program evaluation and know how to deal with them.

Approach

In this training, program evaluation is seen as a positive, constructive process. It can help improve a program.

With this approach to program evaluation,a miscalculation can become an opportunity for making improvements. It can also become an opportunity for not making the same error again.

Our approach is holistic. We will look at the requirements needed to evaluate a program, as well as the environment and human aspects involved in program evaluation.

II. What is Evaluation?

Definition

Many people have negative feelings regarding evaluation.They immediately think of times when they felt they were judged unfairly. But what exactly is program evaluation?

It means asking questions and gathering information in order to:

• assess how a program is coming along (process evaluation); and
  
• compare the program objectives with the actual results (outcome evaluation).
  

Evaluation is a way of measuring whether a program is doing what it is supposed to do.

Evaluation provides an opportunity to develop and improve a program.

The purpose of evaluation is to understand why things work well or why things don't work. Evaluation also allows you to consider possible improvements.

When you evaluate, you are trying to get an overall picture. You step back from your daily concerns to try to find the links between pieces. You also want to determine whether the picture you are piecing together is similar to the one you planned on.

Two types of evaluation

Evaluation can be approached in two ways: it can be a one-time effort (outcome evaluation) or it can be ongoing (process evaluation). Both can help in decision making. However, the approaches and results differ.

1. Outcome evaluation

Outcomeevaluationsummarizes the results of the program and looks at its outcome. It attempts to answer questions like:

• Have we reached our goals?
  
• Should we continue the program, modify it or end it?
  

For an outcome evaluation's analysis and recommendations to be useful, a program has to have been operating for some time. Analysis of the outcome is often completed toward the end of the program. However, it needs to be planned from the beginning and information has to be collected throughout the program or else there will be no data to use in the evaluation.

You can't start planning the evaluation when the program is almost finished.

2. Process evaluation

Process evaluation, on the other hand, is carried out as the program is operating. It can be informal and simple,but provide interesting results. It focuses primarily on the process, asking questions like:

• What aspects of the program are working well? Why?
  
• What aspects of the program could be improved? How?

While outcome evaluation is done at a specific point (usually toward the end of the program), process evaluation is carried out throughout the course of activities. Process evaluation helps you make adjustments along the way, rather than discovering the program's strengths and weaknesses when it's too late to do anything about it! It also provides you with quick answers to questions that might be asked about your program.

Evaluation in the life of a program

As shown in the diagram, evaluation can occur at any stage in a program. The information obtained can be used as it becomes available, either to improve a program or help plan others.

III. Why Evaluate?

Reasons why we avoid program evaluation

Evaluation is natural and satisfying. It is also necessary if we want to make improvements. Yet, we often tend to avoid evaluation in a more formal setting.

Generally speaking, people tend to associate evaluation with unpleasant feelings. Evaluation triggers all kinds of memories.

There are usually three main reasons why people do not evaluate programs:

• They don't know how. They lack the necessary knowledge, expertise and techniques.
• They can't evaluate. In their working environment, there is no time, no budget, etc., for evaluation.
• They don't want to evaluate or are reluctant to. They may have a negative view of evaluation.

By not evaluating, you may be missing out

Evaluation can help you see whether a program has reached its objectives and how it has done so. It also provides information about how the program is coming along and allows programs to be improved. Improvements keep programs on track and make sure they reflect the goals and objectives set.

Reasons why we should evaluate

Evaluation can make people feel anxious and defensive. However, it can also empower them. An evaluation that suits your needs and interests can be very useful.

The reasons why organizations evaluate vary. Most likely, you will have more than one reason for carrying out your own evaluation.

Generally, there are four main reasons for evaluating:

1. to help you understand how your program is coming along;
   
2. to show your funders what you aredoing;
   
3. to show other communities what has worked for you; and
   
4. to show your staff how they are doing.

1. To help you understand how your program is coming along

At any point in the life of a program, you should be able to say whether it is doing what it set out to do.

2. To show your funders what you are doing

Since funding for programs is limited, you may need to show a funder what you are doing.

3. To show other communities what has worked for you

Your program may have had a very positive impact on young people in your community. Other communities may want to know what you did and how.

4. To show your staff how they are doing

Staff working with participants on a daily basis need to see their work within a larger context.

IV. How to Evaluate?

So far, you have looked at why an evaluation should be done. You have thought about the advantages of evaluation.You have even decided on a program you would like to evaluate. This section will guide you through a five-step process to do so. But first, let us look at some basic information about how to carry out an evaluation.

Evaluation can be easy to do, yet systematic

Evaluation is a natural process. However, when evaluating a program, it is useful to use a systematicapproach.

"Systematic" does not mean "complicated". In fact, evaluation rules stem from "common sense". Being systematic simply means planning and being consistent (doing things the same way all the time).

Common sense rules

Here are some common sense rules to keep in mind while evaluating:

• The evaluation must be planned.
• You must have decided in advance which questions or topics will be covered in the interviews or questionnaires. Making them up as you go along would not be systematic.
• You must get the same type of information about the events/ services/actions you want to compare. If not, you will be comparing different things.

A five-step model

This model will help you carry out a program evaluation systematically and easily.

The model is divided into five steps, as shown in the diagram below.

The diagram presents evaluation as a five-step process. The steps are:

1. setting the context of the evaluation;
2. preparing an evaluation plan;
3. gathering the information;
4. making sense of the information; and
5. using the results.

As you can see, evaluation is a dynamic, ongoing process. No step can be carried out unless the previous one has been completed. For example, you cannot gather relevant information until the context of the evaluation has been set.

The diagram also shows how the steps in program evaluation make up a cycle in which findings are integrated to improve the program.

Moreover, "taking human aspects into consideration" is an important element in program evaluation. It must be dealt with throughout the process. Since it is not a step, however, it has been placed in the middle of the diagram (6).

Because of the importance of considering human aspects in program evaluation, a separate chapter of this manual is devoted to the topic.

In this training, we will use the cycle diagram as a guideline. It is not set in stone. You can, and should, adapt it to your own program or situation.

Let's take a closer look at each step now.

1. Setting the context of the evaluation

When you evaluate a program, the first thing to do is set the context of the evaluation.

This step can be compared to what sculptors do when they begin a wood carving. While whittling down the piece of wood, they think about what they want to do and what the carving will look like. They might think about the person the carving is intended for.They also star tthinking of how they will work the wood.

In program evaluation, this preparatory step lets you define your evaluation needs. It ensures the evaluation's goals and objectives are set from the beginning. By completing Step 1, you clarify what you need and expect from the evaluation process to ensure the evaluation will be useful to your organization.

Step 1 in the evaluation process is based on six elements:

1. knowing your organization;
2. deciding why you want to evaluate;
3. deciding who the evaluation is intended for;
4. deciding what you want to evaluate;
5. deciding how the evaluation will be carried out; and
6. deciding who will carry out the evaluation.

All of these elements could be discussed in a meeting with board members, management and staff.

Choose someone to chair the meeting. That person should write ideas on a blackboard so participants can share their ideas more easily. A record of the results should be kept because it will be needed later in the evaluation.

a) Knowing your organization

Before you can evaluate, you need to set the context of the evaluation.

At this stage, you would think about your organization's philosophy, goals and structure. You would also look at the programs and resources offered. You would consider the program environment and any other aspect that might be important for an accurate overall picture of your organization. You would also consult long-time members.

Don't get discouraged by this exercise. It can be done quite simply and spark interesting discussions. You will find that it may also be handy in other situations when basic information about your organization is needed.

b) Deciding why wou want to evaluate

First, you must decide why you want to evaluate and what you hope to gain from the process.

On the board, write all of the reasons people give for wanting to evaluate the program. These reasons will help you set your evaluation goals later on.

Here are some reasons an organization might decide to evaluate:

• to show others that a program is effective;
• to find out whether or not a program is heading in the right direction;
• to find out whether the program is meeting the needs it was designed to meet;
• to justify past or future spending;
• to assess program costs in terms of time and money;
• to gain support for expanding a program;
• to improve or "fine tune" a program;
• to provide information to someone who wants proof that the program is effective; and
• to identify problems within the program and ways of dealing with them.

c) Deciding who the evaluation is intended for

Why you evaluate is sometimes related to who you are evaluating for.

Evaluation can be useful to many parties. For example:

• the organization responsible for the program (board of directors, executive committee, general management, etc.);
• front-line workers (paid staff or volunteers);
• executives;
• members of similar projects;
• funders;
• people participating in or using the program;
• people who once used the program;
• participants who have left the program;
• Elders; and
• community leaders.

It is important to identify why you want to evaluate and who will use your findings. This information will determine how many parts you will want to evaluate, and in how much detail. It will also help you decide who will carry out the evaluation and what type of evaluation report should be written.

For example, an evaluation carried out to pinpoint and deal with a specific problem within the program won't be set up along the same lines as an evaluation to gain support for expanding a program.

d) Deciding what you want to evaluate

Why you are evaluating and who you are evaluating for will influence what you evaluate and what type of evaluation you carry out.

If you want to evaluate how the program is coming along, you will choose a process evaluation.

On the other hand, if you want to evaluate to what extent the objectives of the program have been reached, you will choose an outcome evaluation.

Outcome evaluation will show whether you have achieved your goals and objectives. If you want to know how and why certain results were achieved, you will have to evaluate the process. These two types of evaluation go together.

e) Deciding how the evaluation will be carried out

At this point, you will also want to explore how the evaluation will be carried out, without going into detail - much like sculptors who prepare their piece of wood without knowing exactly what the finished work will look like.

f) Deciding who will carry out the evaluation

There are many ways to carry out the evaluation. You can take the responsibility yourself, you can give it to other members of the group or you can create an evaluation committee. You can ask everyone involved in the program to give their views at a meeting or you can hire outside experts.

When deciding who will carry out the evaluation, you need to consider the scope of your evaluation project. You also need to look at how much experience the members of your group have before deciding.

Deciding who will carry out the evaluation may also depend on the program and type of evaluation you want done.

For an outcome evaluation, it is preferable to ask for help from a person who is not involved in the program. Why? Because it is very difficult to be involved yet objective when judging whether the program's goals were reached.

The information you have gathered at meetings will give you an idea of who should be involved in the evaluation. Being involved can include anything from gathering information to attending evaluation review meetings to helping write the evaluation report.

One solution is to set up an evaluation committee.This committee could include the people who will eventually use the evaluation results. You need to find interested people and people who will be able to use the results of the evaluation.

Generally speaking, the evaluation committee directs and supervises the process and the evaluator or evaluation team carries out the evaluation. To function effectively, the group must not be too big: five or six people, never more than ten.

The most effective evaluation occurs when everyone measures the progress made. This way, everyone comes to understand the program better and wants to improve it. This is more effective than when some higher or outside authority carries out the evaluation.

The evaluator does not need to be a specialist. Using this guide, someone who knows the program and has the support of an evaluation committee could be able to carry out an effective evaluation.

The following table provides some examples of the people who could participate in either a process evaluation or a program evaluation.

Deciding who will carry out the evaluation

2. Preparing an evaluation plan

Step 1 lets you set the context of your evaluation. Next, you will want to prepare a more specific evaluation plan.

Step 2 of the evaluation process will allow everyone involved to see what the process entails, how they can participate and how the pieces fit together to make up a program evaluation.

Step 2 of the evaluation process is based on four elements. You will use what was decided in Step 1 for:

1. finalizing your evaluation goals and objectives;
2. preparing the evaluation questions;
3. deciding on the evaluation method; and
4. preparing a timetable.

Don't let these guidelines scare you. Take your time and you'll be fine! These steps are very close to what you do naturally.

a) Finalizing your evaluation goals and objectives

The first thing to do is to finalize your evaluation goals and objectives. They will form your general evaluation guideline.

b) Preparing the evaluation questions

You may notice that we are getting more and more specific as the model unfolds.

Now that you have set the evaluation objectives, you need to prepare the evaluationquestions.These questions will be used to gather the information you will analyse to prepare your recommendations for the program. These questions can be decided in a working session.

Here is an example of evaluation questions which were developed from a set of evaluation objectives from Maternity House.

c) Deciding on the Evaluation Method

Now that you have the evaluation questions, it is time to choose your method. This means deciding how you intend to get the answers to yourquestions.

Once again, you will be building on the previous stage. Looking at each evaluation question, try to find out how you can answer. Where is the information and how can you get it?

The following chart gives you an idea of the method used to evaluate the Maternity House Volunteer Program.

d) Preparing a timetable

You now need to make up a timetable for the work to be done. This means assigning tasks to different people and setting deadlines.

It might be a good idea to think about the costs related to your program evaluation. Think about the possible expenses: photocopies, stamps, long-distance telephone calls, a specialist's wages, coffee for a meeting, etc.

In this section, we have seen how an organization might transform its evaluation plan from an initial broad idea to a detailed methodology and timetable.

Step 3 of our model will get people involved in a different way. It is time to gather the information used to answer the evaluation questions. This step can be very interesting because you finally feel you are getting close to the goal.

3. Gathering the information

After completing Steps 1 and 2, you have a precise idea of what you need to know. You also know how you will obtain this information.

Step 3 includes three elements:

1. reviewing existing information;
2. creating tools; and
3. locating new information.

a) Reviewing existing information

You will begin by reviewing existing information which might help answer your evaluation questions.

The first place to look for information to answer your evaluation questions is the organization or program you are studying. It is wise to review existing information before trying to locate new information.

There are generally two information sources for an evaluation: people anddocuments.

Here is a list of people who could provide information:

• people using the program;
• program manager;
• board members;
• program staff;
• program volunteers;
• community Elders;
• community leaders;
• organization staff;
• funders;
• managers, staff and clients from other related agencies; and
• outside consultants.

Documents fall into various categories.Recording and reporting systems give information on program events as they occur.

These documents often contain much of the information you need to know about a program.

Here are examples of possible sources of information, both inside and outside your organization:

Internal sources

• original program proposal for funding;
• studies or needs assessments done in support of the program;
• budget and financial statements;
• organizational charts;
• monthly activity forms;
• policy or operations manual;
• log books;
• caserecords;
• training or orientation guides;
• memos;
• minutes of meetings;
• committee reports;
• annual reports;
• newspaper articles and publicity materials;
• previous studies or evaluation reports; and
• statistics sheets, work reports, time sheets, etc.

External sources

• census data;
• community studies;
• needs assessments/studies done by other agencies;
• local historical society;
• literature and other research in this field of study;
• governmentministries: publications, reports and consultants; and
• funding source(s) of the program under study.

b) Creating tools

Your review of existing materials will probably not answer all of your evaluation questions. You may need to gather new information to get a complete picture.

You need tools to gather this information. A wide variety of information-gathering tools exists. The most common are:

• written questionnaires;
• interviews (face-to-face, telephone or group interviews); and
• observation.

You will find more information about these techniques in the "Toolbox" under the headings "Interviews", "Questionnaires" and "Questions".

c) Locating new information

The tools you have created will help you locate new information. The next thing to do is to send out questionnaires, interview people or observe pre-selected people.

The timetable you prepared earlier will ensure you don't forget any tasks.

This new information, added to what you already have, should help answer your evaluation questions. The pieces are falling into place! Don't stop now. Step 4 is making sense of this information.

4. Making sense of the information

By now, you have gathered all the information you need to answer your evaluation questions. Let's try to make some sense of it!

Step 4 of the evaluation process includes three elements:

1. compiling the information;
2. analyzing the information; and
3. drawing conclusions.

a) Compiling the Information

Start by compiling all the information you have gathered.

It may be useful to photocopy original documents. You can then write on the copies if necessary. The originals should be filed.

Make sure to pay special attention to confidential documents such as clients' personal files.Photocopying this type of document is not recommended.

b) Analysing the information

Once all of the information is compiled, you need to analyse it. Take some time to look back at the evaluation questions. Then, try to find trends or patterns that could help pinpoint an answer.

It is important to write down your ideas immediately, while they are fresh in your mind.

As you read the material, look for patterns in the answers.

While you are looking for patterns in the answers, you also need to look for differences.

c) Drawing conclusions

Probably the most important stage is interpreting the information you have gathered. This is called drawing conclusions.

As you are forming yourconclusions, ask yourself whether all the information is consistent. Is what is written in the documents the same as what is done in practice? When there is a difference, it is helpful to note why.

Drawing conclusions requires careful thought. Give yourself some time to let the information you have gathered and analysed become clear.

Keep the evaluation questions in mind. For each question, try to determine the conclusions that are taking shape in your own mind and justifying them. For example, is it because:

• certain numbers or answers show a pattern;
• there are a number of similar answers; or
• there is a logical chain of evidence?

The conclusions may deal with the process (how things are going in the program). They may also deal with the outcome (to what extent the expected results or program objectives were achieved).

Step 5 uses the results of your evaluation. Things are going well! You have almost completed the step-by-step model.

5. Using the results

"Using the results" is the last step in the model, but don't be misled by appearances. In fact, the results should be used as soon as they become available. This is easier to do for a process evaluation and when the decision makers and program participants are involved from the beginning of the evaluation process.

The following diagram shows how evaluation findings can be integrated into the program to help you reach your goals and objectives.

Let's now move to "using the results"

You have your results. Now, you want to use them. Step 5 includes foure lements:

1. making recommendations;
2. writing the report;
3. making the results known; and
4. taking appropriate action.

a) Making recommendations

Recommendations naturally follow conclusions.Your conclusions were designed to improve the process or to determine to what extent program goals and objectives have been met.

If the program has achieved its goals and objectives, it should probably remain unchanged. If not, maybe it should be modified or cancelled altogether. Or maybe the objectiveswereunrealistic;they might need to be adjusted.

b) Writing the report

Your recommendations need to be written down in a report. But the evaluation report does not only consist of recommendations.

Here are the main elements that are usually included in an evaluation report:

1. summary;
2. introduction, goals and objectives;
3. evaluation questions and method;
4. results and conclusions;
5. recommendations; and
6. appendices.

The summary gives the reader key facts and information. Often, if there is no summary, people will not read the report. It is nice to send a copy of the summary to people who helped in the evaluation process as a way of saying "thank you".

The introduction briefly describes the program and outlines its goals and objectives. Next is a description of the methodology used in the evaluation. Generally, you will explain the procedure and tools used to gather information. This section also includes any other relevant details about your overall evaluation strategy.

Then, the report presents the results as objectively as possible. Conclusionssynthesizetheresults and interpret their meaning.

The report ends with recommendations for the future. Any other information of interest to the reader may be placed in the appendices (for instance, a copy of a questionnaire used to gather information).

It is a good idea to write a first draft of your report and then put it aside for a few days. When you reread it, you can do the final editing.

A report is a tool to help people make decisions. It may be used to:

• find out the strengths and weaknesses of a program to help improve it;
• compare how effective different types of programs are;
• help people decide whether program goals and objectives are being reached or whether readjustments are necessary;
• recognize work performed;
• encourage other communities to take similar action; and
• gain support for continuing or expanding a program.

c) Making the results known

Evaluation does not end with the report. It is important to make your conclusions and recommendations known.

The people involved in the program will be interested in the results. They might also be interested in talking about them. A feedback session could be organized to give them the chance to comment on the findings.You could present your report and then have a group discussion.

Concentrate on positive information. Describe the program's strengths and weaknesses and discuss them openly.

The report can also be used to tell others about the work the group has done or specific issues you are working on. The community can be involved in providing feedback or comments that will be incorporated into the report. Parts of the final report can be used in your annual funding request.The evaluation report can also be a way to ensure all aspects of your group's work are recognized as useful and important.

d) Taking appropriate action

Evaluation results are meant to be used. Talk about them. Use them to plan your next program. Use your evaluation report or summary to inform new members of the group.

An evaluation is an investment for the future. It should be a starting point for taking action. Action is based on people's decisions, so make sure people who are in a position to make decisions get your report. The evaluation process will be complete when informed decisions based on evaluation results produce new actions.

Now you have finished! All of the puzzle pieces are in place. You can be sure the picture you have of your program is accurate because your methods were systematic. Your group can rest assured the decisions it makes will be in line with the program's specific needs and expectations.

Good luck with future plans!

An effective evaluation

Here are some basic principles that will guarantee the effectiveness of your evaluation. Make sure that yourevaluationis:

• useful

  It should be based on the needs, goals, objectives and decisions of your organization, staff and clients.

  The evaluation should lead to decision making and taking actions.

  The evaluator should present the results in such a way that the organization can use them to improve the program under study.

• planned

  To be constructive, the evaluation should be planned from the beginning of the project or program. Both participants and decision makers should be involved from the outset so they have a good understanding of the process and can give you support when needed.

• conducted ethically

  Evaluation techniques should be simpleandnon-threatening.They shouldrespectindividualconcerns and needs. Special steps should be taken to protect the confidentiality of the data.

• systematic

  The evaluation should be systematic, even if it is simple.

  It should use a variety of methods to investigateandgatherinformation.

  In this way, it will yield results that are true to reality - your reality.

• ongoing

  Try to get into the habit of continuously gathering information on your program and carrying out an ongoing evaluation. This will help you continue to work toward your initial objectives.

V. Human Aspects of Evaluation

You now know how to use a step-by-step program evaluation model. Managing interpersonal relations while performing the evaluation is also important. That is why we have included this section.

People's feelings

An evaluation can be hard on people's feelings.

One of the reasons is the value judgments people generally associate with evaluation. They are afraid of being put "on the spot". Even if they are very good workers, they may be afraid they will lose their jobs as a result of the evaluation process.

Remember the list of emotions you developed at the beginning of the manual on why people don't evaluate? This exercise should help both you and them better understand these feelings.

It is very important to take time to tell everyone you intend to conduct a constructive evaluation. Stress that you will be looking at facts, not at people.

Try to involve people right from the beginning of the five-step model, so they will understand how you will be proceeding and why.

Celebrate your successes and progress. Pay particular attention to them. Some of your most useful lessons will be learned from analysing how and why the positive aspects were achieved.

People may feel the evaluation process does not accurately represent their efforts. This may be true, especially if you try to evaluate a program that has been in use for a long time. People may feel the five-step model makes a program look like a skeleton without a soul.

Resistance

The evaluation process might change your working environment. Some people like change, others don't. Be prepared to face resistance to the evaluation. Some people will be willing to help you and some will not. Why?

Resistance to change is normal. In a new situation, two opposing forces are at work. There are "driving forces" which initiate change and keep it going. And there are "resisting forces" which put on the "brakes" and try to stop the "driving forces".

While you can't increase the driving forces, decreasing the resisting forces will ensure the changes brought about by your evaluation are more easily accepted. Here are six strategies to help:

1. Look at the past. Try to see how resistance was handled in the past.
2. Communicate. To gain support, provide adequate information.
3. Have people participate. As early as possible, involve everyone concerned in the process. It is much easier to support something you have been involved in.
4. Be supportive. Be prepared to spend extra time with people who have trouble accepting change.
5. Negotiate. Work out a situation where all parties involved feel they are gaining something. Match people's goals with the evaluation goals and objectives.
6. Always have a positive attitude.

Confidentiality

Participants and staff involved in the programs you are evaluating have a right to privacy. Always remember an evaluation must keep information about their lives and jobs private. This is called confidentiality. There are two rules of confidentiality:

• Evaluators need an individual's permission to have access to private information about him or her.
  
• Evaluators should not tell other people private information they have learned from an evaluation.

Keeping information confidential shows respect for those who answer your questions and is especially important if your group wants to use the evaluation findings to continue the program or improve services.

Protect confidentiality at all times during and after the evaluation process. For example, if someone talks about places, people and things that could identify them, remove these details when you write about them in the final report.

Decide on how to "code" people. There are a number of options. You can make up names or use numbers, letters or a combination of letters and numbers.

Finally, you may need to get a signed letter of consent to see a person's file or observe him or her in a program. If the person is under 18, a parent or guardian must give this permission.

In the evaluation process, confidentiality is very important and should be taken very seriously. People's privacy must be respected.

The ideal evaluator

After all that has been said thus far, we can conclude that the ideal evaluator:

• dares to question situations and facts;
• doesn't take anything for granted;
• respects people;
• never loses a chance to gather useful information and keeps track of that information;
• uses both common sense and a systematic approach; and
• is willing and prepared to deal with resistance.

Appendix A

Evaluation goals and objectives chart

Program:

• Reason for evaluating
• Evaluation goals
• Evaluation objectives

Signature
Date

Appendix B

Methodology chart

Program:

• Evaluation questions
• Evaluation questions
• Who has the information or where can we find it?
• How will we get the information?

Signature
Date

Appendix C

Evaluation timetable chart

• Task
• Who will do it?
• When will it be done?
• Comments

Signature
Date

Appendix D

Ongoing evaluation report

Program:

Did the activities planned during the last meeting take place?

yes
no

If the activities took place as planned

1. What worked especially well?
   1. Why did it work so well?
   2. How could the same success be achieved in other activities?
2. 1. What difficulties were encountered?
   2. What solutions can you think of?
   3. Which solution would you recommend?
   4. Who will be responsible for implementing the solution?

If the activities did not take place as planned

1. Why didn't they?
2. What will be done about it?
3. Who will be responsible?

Signature
Date

Appendix E