Health Canada First Nations and Inuit Home and Community Care (FNI HCC) Program - Evaluation Guide

Minister of Public Works and Government Services
2002
ISBN: 0-662-66913-4
Cat. No.: H35-4/31-2002

Help on accessing alternative formats, such as Portable Document Format (PDF), Microsoft Word and PowerPoint (PPT) files, can be obtained in the alternate format help section.

Table of Contents

• Guiding Principles of the FNI HCC Program
• The Definition of Evaluation
• The Purpose of Evaluation
• Getting Started: Developing an Evaluation Plan
• Determining the Method to Collect Information
• Conducting the Evaluation
• Analyzing the Information
• Reporting the Information
• FNI HCC Examples

An evaluation is not conducted to prove that a program works but more importantly to improve the way it works.

Source: Unknown

This section provides the background for the First Nations and Inuit Home and Community Care (FNI HCC) program, outlining the vision, principles and objectives.

Vision
In response to the unique health and social needs of the First Nation and Inuit people, the FNI HCC Program provides basic home and community care services that are comprehensive, culturally sensitive, effective and equitable to that of other Canadians. The program is a coordinated system of home and community-based health-related services that enable people with disabilities, chronic or acute illnesses and the elderly to receive the care they need in their homes and communities.

This document has been developed as a reference source for First Nations and Inuit communities evaluating the FNI HCC program.

If you would like more information on FNI HCC evaluation/comment on this document or if you require evaluation support, please contact:

Satish Seetharam
Program Evaluation Project Manager
Health Programs Support Division
Community Health Programs Directorate
First Nations and Inuit Health Branch
Health Canada
Postal Locator 1920A
Jeanne Mance Bldg., Room C 2076
Tunney's Pasture
Ottawa, ON K1A 0L3
(613) 952-4230

Published by the authority of the Minister of Health Canada.

This publication can be made available in/on computer diskette/large print/audio
cassette/Braille upon request.

© Minister of Public Works and Government Services, 2002
Cat. H35-4/31-2002
ISBN: 0-662-66913-4

Guiding Principles of the First Nations and Inuit Home and Community Care Program

• Respect traditional and contemporary First Nations and Inuit approaches to healing and wellness;
• Planning is community-based and community-paced;
• Programs are available to individuals of all ages with an assessed need;
• Services provided are at least as equitable, effective and equivalent to those received by the general population and they are supported by quality assurance measures; and
• Supportive to family and community involvement.

Program Objectives

• To build the capacity within First Nations and Inuit to develop and deliver comprehensive, culturally sensitive, accessible and effective home care services;
• To assist First Nations and Inuit living with chronic and acute illness in maintaining optimum health, well-being and independence in their homes and communities;
• To facilitate the effective use of home care resources through a structured, culturally defined and sensitive assessment process to determine service needs of clients and the development of a care plan;
• To ensure that all clients with an assessed need for home care services have access to a comprehensive array of services within the community, where possible;
• To assist clients and their families in participating to the fullest extent and to utilize available First Nation and Inuit communities support services where available and appropriate in the care of clients; and
• To build the capacity within the First Nations and Inuit to deliver home care services through training and evolving technology and information systems to monitor care and services while developing measurable objectives and indicators.

Support Elements might include but are not limited to:

• Facilitation and linkages for rehabilitation and therapy services;
• Respite care;
• Adult day care;
• Mental health home-based services for long-term psychiatric clients and clients experiencing mental emotional illness. These services might include traditional counseling and healing services, medication monitoring;
• Support services to maintain independent living which may include assistance with special transportation needs, grocery shopping, accessing specialized services and interpretive services;
• Home-based palliative care;
• Social services directly related to continuing care issues; and
• Specialized health promotion, wellness and fitness.

FNI HCC Program Logic Model

Activities

• Provide Homecare Services to Eligible FNI populations
• Create/Develop Partnerships in Support of the HCC Program
•  Educate, Communicate, Share and Listen to/with Community Leaders,
  HCC Eligible Populations and Service Providers
• Administrate/Manage the FNI Home and Community Care Program

Outputs

• HCC Services for Eligible FNI populations
• Partnerships, Service Delivery Agreements, HCC Best Practice Models
• Communications and Educational Products for FNI Populations, Community Leaders, F/P/T Governments, Certified Training Program for Care Providers etc.
• Needs Assessments, Service Delivery & Training Plans, Knowledge Management and Information Systems (financial, ongoing monitoring evaluation)

Immediate Outcomes

• FNI Access to HCC Services - Essential and Supportive Elements
• Increased Community Capacity - Human Resources to Administer and Deliver the HCC Program
• Increased Service Delivery Awareness and Accountability - Informed Users of HCC Services and Informed Community Leaders
• Infrastructure in FNI Communities to Deliver the HCC Services - Capital and Program

Intermediate Outcomes

• Responsive, Flexible Use and Provision of HCC Services
• FNI Delivered and Administered Homecare Services
• Sustainable, Culturally Sensitive HCC Services

Ultimate Outcomes

• HCC Clients Maintaining Optimum Health
• HCC Clients Remaining in Their Homes and Communities

The Definition of Evaluation

This section discusses what an evaluation is and what it is not.

Program Evaluations

A program evaluation involves gathering information about a program in order to make decisions about the program.

You may hear references to "goal-based evaluations", "process-based evaluations", "outcomes-based evaluations" or "other type of evaluations". The name or type of evaluation is less important. What is important is that you get the information needed in order to determine what is working well and what areas could be changed for improvement.

What a Program Evaluation SHOULD BE

• A chance to find out what is working and what is not;
• A chance to make changes to a program;
• Useful, practical, and relevant.

What a Program Evaluation SHOULD NOT BE

• An opportunity to point fingers or lay blame;
• An activity that produces fear that a program will be reduced or eliminated;
• A one time activity

The Purpose of Evaluation

This section discusses why evaluations are important, both in general and in context of the First Nations and Inuit Home and Community Care Program.

Reasons to Evaluate

While there are many reasons to undertake a program evaluation, the following are three key reasons:

1. To understand the impact a program and its components are having on clients, families, and the community

   These "outcomes" evaluations are necessary to see the program's impact on the community. It is better to rely on feedback from individuals to draw conclusions. A more formal evaluation can more accurately and reliably understand where things are working well and where there are issues that need to be addressed.

2. To improve the delivery of services to increase quality and improve spending of resources

   Public, private and non-profit organizations all share a common goal - to maximize their effectiveness. It is only natural that all involved look to be as efficient as possible in delivering the program at the national, regional, and community level.

3. To confirm that you are doing what you think you are doing

   Evaluations can verify if a program is really running as originally planned. These are sometimes referred to as "process" evaluations. There may be lots of good reasons why it might not be, but those gaps must be identified for future planning.

Evaluating FNI HCC at the Community Level

It is not an excuse to point fingers.

Some organizations, or individuals within organizations, may not embrace a program evaluation because they think it will be used as a reason to position individuals or departments against one another. (If this occurs, the issue is not the program evaluation but signals there are likely other problems within an organization.)

It is not an excuse to cut funding.

This is a common fear - that if program sponsors find out that the program is not working as well as it could be in the community, it will result in a decreased financial commitment in the future. The goal is to make the best use of the financial and human resources dedicated to the FNI HCC Program for the benefit of the community - not to see what can be cut or eliminated.

It is meant to see if the plan is right and if not, how it should be changed.

It sounds simple because it is simple. If everything is working perfectly, then no changes need to be made. While that would be wonderful, it is more realistic to expect that some areas are working better than others. Based on this information, one can continue to do the things that are working and modify the areas that require changes.

National and regional evaluations cannot gather the same information that a community evaluation obtains.

The reason for evaluating the FNI HCC Program at the national, regional, and community level is that each plays a unique role in the program, has a unique perspective, and requires different information to understand what is working and what modifications may be needed.

The information at the community level is conducted by community representatives for community purposes.

It allows for ongoing program information about program performance.

By communicating, educating, sharing and listening to/ with community leaders, HCC eligible populations, and service providers on a regular basis we can act as a medium of information. Therefore the cycle of information from planning, implementing, and evaluating will be continuous.

Too Many Evaluators Not Enough Doers!

In both the public and private sectors, some organizations appear to struggle with the importance of planning versus the importance of implementing (the "doing"). Both are clearly important. And yes, one can spend too much time planning and evaluating.

For the FNI HCC Program, evaluations are being conducted at the national, regional, and community level. Why so many evaluations by so many different people? The FNI HCC Program represents a huge commitment, not only on the part of the federal government but also other stakeholders including First Nations and Inuit communities and individuals within these communities. Again, all of this learning can help communities and administrators be more effective in meeting the goals of the program.

The national, regional, and community evaluations review different components from different perspectives. All are relevant and all are necessary. From the community perspective, the goal is that community stakeholders will take the information obtained from the community evaluations to understand what is working well and should be continued and to identify where changes could be made.

Getting Started: Developing an Evaluation Plan

It can sometimes be a difficult task. "Where do I begin? "

Developing an evaluation plan is a critical step because it will save you time and frustration along the way.

Defining Goals and Objectives

During the initial stages of an evaluation project, the list of issues and topics to evaluate can quickly turn into a lengthy one. While it is excellent to have many areas to evaluate, it is easy to lose sight of the main reason you began the evaluation.

Try to develop one sentence that describes the purpose of the evaluation project. This statement (sometimes referred to as a "purpose statement" or "statement of objectives") should serve as a reference point in developing your questionnaire or discussion guide. There can always be secondary objectives but this single sentence will ensure that the primary focus of the evaluation does not get lost in the process.

Examples of Purpose Statements

To measure the level of satisfaction which care providers have towards the home care support services.

• To understand the perceptions and opinions among community health care workers towards the FNI HCC Program.
• To determine the level of awareness of the home and community support services available among adults in the community.
• To elicit the reactions of current users of home care services to changes made to services in the last year.

Tip!
Consider referencing the FNI HCC logic model (page 3) when developing your evaluation plan. The activities, outputs, and outcomes cited can help focus your evaluation and make it as relevant and meaningful as possible.

Tip!
No evaluation design is perfect. While planning an evaluation is very important, you can spend too much time planning and not give yourself enough time "doing" the evaluation.

Determining Resources

When planning an evaluation, there are typically two types of resources to consider:

1. Financial Resources: Preparing a Budget
   • What money do you have to work with? What are your sources?
   • Are you going to need to hire anyone for any aspects of the project?
   • If you are doing focus groups, do you need food and refreshments?
   • Are you offering focus group or interview participants an incentive?
   • Brainstorm as many potential costs that you foresee so they do not come as a surprise in the end!
2. Human Resources: Developing a Project Team
   • Who do you have to use as a resource?
   • What are their skill sets?
   • Who is going to be the project manager that ensures it all happens?

Determining Timelines

A rule of thumb is that however long you think it will take - it will take twice as long! That said, it is important to map out a realistic time frame and stick to it as much as possible.

Twenty Questions to Get You Started

The 20 questions listed below are only one place to start. They are not an extensive list by any means, but are designed to be inspiration for thinking of other relevant questions and issues.

The Program

• 1. Did we do what we set out to do in the plan?
• 2. Why has it worked or not worked?
• 3. What activities were carried out?
• 4. How do these activities link to reaching the program objectives?
• 5. How realistic and relevant are the project objectives?
• 6. What has changed as a result of the program for users, service providers, caregivers, staff, the community, and other stakeholders?

The Implementation Process

• 7. What was the process of implementation?
• 8. What obstacles have occurred along the way?
• 9. How has this impacted intended outcomes?
• 10. What lessons have we learned about implementing this program?

The Community

• 11. Have the objectives of the program met the needs of the community?
• 12. What has been the feedback from the general community about the program and its components?
• 13. What has been the feedback from users, their families, and care providers about the program?
• 14. What new partnerships developed from this project and what has been the contribution to home and community health care?
• 15. How has the program been culturally integrated into the community?
• 16. In what ways has this program contributed to health care in the community?

The Resources

• 17. Are the human and financial resources able to sustain the activities?
• 18. What has been the feedback from health care providers about the project?
• 19. What has been the feedback from other staff about the project?
• 20. Has the quality of service met expectations of those involved? Why or why not?

Determining the Method to Collect Information

There are some basic rules of thumb when trying to determine how you are going to collect the information. This section tries to take the guesswork out of determining which method will work best for you.

Key Methods of Evaluation

Following are three key methods for evaluation, the main reasons for using each, and some associated advantages and disadvantages.

• Method: Surveys (Questionnaires)
  • Reasons to Use:
    • Gather measurable information if a relatively large sample is surveyed
  • Advantages:
    • Anonymous
    • Able to attach numbers to the evaluation
  • Limitations:
    • Do not tend to reveal underlying perceptions and opinions
    • Reliable data very dependent on wording of questions, sampling, and data analysis
• Method: Surveys (Questionnaires)
  • Reasons to Use:
    • Uncover underlying opinions, beliefs and attitudes that a survey cannot obtain
    • Respondents might be more likely to "open up" in an interview than in a focus group
  • Advantages:
    • Can probe for more information
    • Timing can adapt more easily to respondent's schedule
    • Sensitive topics often lend themselves better to interviews than focus groups
  • Limitations:
    • Time consuming
    • Analysis is subjective to individual interpretation
    • Do not have the advantage of focus groups where discussion spurs ideas and opinions
• Method: Focus Groups
  • Reasons to Use:
    • Enable in depth discussion in a group environment to get at perceptions, opinions, beliefs and attitudes
  • Advantages:
    • Provide directional information about programs and issues
    • Can be a good way to get at a range and depth of information in a short time
    • Respondents can spur dialogue, generating more discussion than an interview
  • Limitations:
    • Analysis is up to individual interpretation
    • Moderator/facilitator plays a critical role to generate discussion and uncover opinions
    • Logistics required to organize a group

Conducting the Evaluation

This section provides suggestions for conducting surveys, interviews, and focus groups.

Conducting Surveys

If you have decided to use surveys as part of your evaluation, there are a number of ways to collect the data.

There are four main ways to conduct surveys:

1. Telephone
2. In-Person
3. Mail
4. Email/Internet

• Survey Method: Telephone
  • When to Use:
    • Target population is dispersed over a wide geographic distance
    • In-person (face to face) interviews are not possible
  • Advantages:
    • Convenient for interviewer
    • Respondents might feel more anonymous
    • Can cover a wide geographic area
    • Can clarify questions if respondent has difficulty
  • Limitations:
    • Not everyone has a telephone
    • Respondents might be more comfortable discussing in person
    • Respondents must be at home
• Survey Method: In-Person
  • When to Use:
    • Geography and/or time is not an issue
    • If you think you might get more information than over the telephone
  • Advantages:
    • Intimacy
    • Can clarify if something is confusing
  • Limitations:
    • Not everyone has a telephone
    • Respondents might be more comfortable discussing in person
    • Respondents must be at home
• Survey Method: Mail
  • When to Use:
    • Resources to conduct the surveys over the phone or in-person are unavailable
  • Advantages:
    • Respondent can do it at his/her convenience
    • Fewer human resources required
  • Limitations:
    • Need a way of getting surveys returned
    • Requires a certain level of language comprehension
    • Response rates tend to be low
• Survey Method: Email / Internet
  • When to Use:
    • Participants are geographically dispersed
    • Participants have access and a high level of comfort with email or Internet
  • Advantages:
    • Surveys can be administered in a timely and cost-effective manner
  • Limitations:
    • Target audience needs to be comfortable with the technology
    • No human interaction

What kinds of questions should you ask?

Now that you know how you are going to survey respondents, you need to come up with the questions.

Closed Ended vs. Open Ended Questions

These are two technical terms to distinguish the two main types of survey questions. However, it is worth noting the distinction between the two types.

For closed-ended questions, survey respondents must choose from a pre-determined list of responses. In the example below, respondents must pick one of four responses provided.

Example: How long have you been a volunteer care worker?

Less than 1 year
1 to 3 years
3 to 5 years
More than 5 years

As the name implies, open-ended questions allow for an unlimited (open) number of responses. There are no lists or pre-determined answers.

Example: Why do you continue to serve as a volunteer care worker?

There are reasons to use both types of questions but keep in mind that closed-ended questions are easier to administer and analyze. Open-ended questions require coding (classifying similar responses into groups) and are more labour-intensive. As such, the fewer open-ended questions you have the more manageable your survey will be.

Tip!
For every question you come up with ask yourself "How will this information be used? " If it is not critical to the evaluation, consider removing it from the survey. It is easy to get caught up in asking the "nice to know" questions. Focus on the "need to know" issues.

Demographic Questions
These questions relate to a respondent's gender, age, language, occupation, family composition, household income, etc. They are used to see if there are any patterns among the different variables used. It is always a good idea to reinforce that information is confidential. Ensure that the questions are culturally sensitive and test them by obtaining feedback from others.

Length of Surveys
In general, the shorter the better. However, you must ensure that the survey gets all of the information you need for your evaluation.

For each question, ask yourself "What will I do with this information? " If it is more "interesting" than "necessary" consider removing that question.

Test it!
Have a couple of people who are not close to the project complete the survey. You can observe them to see any trouble spots as they go. Alternatively you can give them a written copy and get their feedback after. As well, you can visually see by their answers if there were questions they skipped or did not complete correctly.

Who do you survey?
This relates back to the objectives of the study and the information you are trying to obtain. Some of the different groups that might be the subject of a FNI HCC survey include:

• Clients
• Care providers
• Employees/volunteers
• Community leaders
• General community

How many completed surveys do you need?
There is rarely a magic number and the purpose of this document is not to get into the actual math process involved. A good rule of thumb is that the more you get, the more confidence one can have in the results. Try for as many as you can obtain and manage, while ensuring that you keep within the target audience you want to survey.

Conducting Interviews and Focus Groups

If interviews or focus groups are part of your evaluation, this section provides some suggestions designed to make the process go smoothly and efficiently.

Types of Interviews & Focus Groups

• Method: In-Person Interviews
  • When to Use:
    • The topic is not appropriate to discuss in a group
    • Geography is not an issue
  • Advantages:
    • Respondents may feel more comfortable than in group environment
    • Personal
    • Can probe to obtain in-depth perceptions and opinions from individual
    • Can observe body language
  • Limitations:
    • Take comparatively more time
    • Do not obtain the dynamics associated with groups
• Method: Telephone Interviews
  • When to Use:
    • The topic is not appropriate to discuss in a group
    • Geography IS an issue or coordinating times for personal interviews is too inconvenient
  • Advantages:
    • Convenient for both interviewer and respondent
  • Limitations:
    • Interviewer is "only" hearing respondent's voice (not able to observe body language)
    • Less personal
• Method: Focus Groups (6 to 10 participants)
  • When to Use:
    • When the benefits of group dynamics outweigh the need for private communication
  • Advantages:
    • Opinions and beliefs tend to be more spontaneous in groups; lead to other valuable discussions
    • Can talk to more people in shorter period of time
  • Limitations:
    • Some people may not be comfortable revealing opinions with strangers or other community members
    • It can be challenging keeping a group on track, getting quieter people to speak up or more outgoing ones to moderate the amount they speak
• Method: Mini Groups (Approximately 4 participants)
  • When to Use:
    • When you are looking for more in-depth feedback from individuals but want some group dynamics
  • Advantages:
    • Can be easier to control smaller group
    • Each person has more "air time" for opinions
  • Limitations:
    • You may need to do more groups to obtain enough opinions

What is a Focus Group?

• A group discussion where typically 6 to 10 people are brought together in a single session for 1 to 2 hours to discuss issues and share opinions;
• Facilitated using a specific agenda of structured questions, similar to the interview guide, which focuses the discussion in the meeting.

Finding a Location

Regardless of the topic, format, or number of participants, you are going to need a location to hold the focus group(s).

• Make it central for participants.
• Ensure it is private, quiet and free from interruptions.
• If you are sitting around a table, ideally a round or oval table is best.
• Using a "living room" setting with sofas and/or chairs can work very well too.
• Community centres, hospitals, churches, schools, or an office boardroom are often good sources.

Recruiting Participants

This is a relatively straightforward yet key part of the focus group process.

Develop the profile of the participants.

Similar to defining your target for a survey, you will need to identify "the type of person" you want to see in the group(s). This could include information related to:

• Age (e.g. adults 19 to 34, seniors aged 65+, etc.)
• Gender (male only, female only, or mixed)
• Lifestyle (e.g. smokers)
• Health-related (e.g. parents of children with a mobility disability)
• Occupation (e.g. health care professionals, stay-at-home mothers, retirees, etc.)
• Household (e.g. single mothers, parents with children under 5 years of age, seniors living with adult children, etc.)

Determine how you are going to contact (recruit) prospective participants.

• Do you have a list to work from?
• Who is going to do the telephone calling?

Determine how you are going to contact (recruit) prospective participants.

• Do you have a list to work from?
• Who is going to do the telephone calling?

Follow up with a reminder telephone call the day before.

• If a participant cancels, try to find a replacement if there is time. It is not unusual for one or more "confirmed" participants not to show.

Offering Participants Incentives

For focus groups, participants are often provided with something for taking the time to discuss their opinions and as an incentive to participate during the recruiting process. It can be in the form of money, gift certificates, or small gifts. A letter from a community leader also may or may not be appropriate.

You know your participants best and therefore you know what is appropriate to give. The most important thing is to ensure it is fair, appropriate, and consistent. You also may decide that a gift or incentive is not appropriate or required.

Tip!
There is no magic number of focus groups to conduct. Use your judgment based on time, number of potential participants, budget, etc.

Writing a Discussion Guide

What is a discussion guide?

It is a few pages of notes that the focus group moderator prepares that "guides" him or her through the discussion. The more you know your discussion guide, the more comfortable you will be in conducting the groups. The guide is not intended to be a set list of questions like those of a survey. It is meant to ensure that you have a list of the different topics, issues, and types of question you want to have discussed.

Keep Participants Happy

Why do you want to keep participants happy? Because if they are not happy, they are not going to talk!

Consider some of the following:

• Provide them with some food and something to drink before the group.
• Keep food available outside the room while you are discussing. It can be too distracting during the conversation.
• Consider having beverages such as water, tea, coffee, juice and/or soft drinks on hand.
• Let participants know that they can get up at any time to use the washroom.
• Make sure the room is at a good temperature. It is not unusual for people to be preoccupied with the temperature if it is too hot or too cold.

Moderating Focus Groups

There is a saying that "A good focus group moderator makes it look so easy that anyone can do it."

Who can moderate?

Focus group moderators (also known as facilitators) come in all shapes, sizes, and from diverse educational and professional backgrounds. While some are outgoing and extroverted, others are more reserved and soft spoken. However, some common traits of an effective moderator include:

• Shows warmth and understanding, and creates a non-threatening environment;
• Does not reveal his/her own opinions in a discussion;
• Probes the comments of participants to understand meanings;
• Controls the group so discussion stays on track without being severe;
• At the same time, is flexible enough to let conversations flow "off schedule" to get at insightful comments;
• Encourages and gets participation from all;
• Respects all participants and all opinions;
• Has a sense of humour;
• Sorts information quickly and efficiently; and
• Uses eye contact.

Moderating takes practice but it does not have to be an intimidating task. If you relax as a moderator, your participants will be relaxed and likely to open up and share their opinions. That is half the job.

Tip!
A focus group moderator must remain neutral, non-judgmental and have the skills to keep the discussion moving and focused. You may want to ask an Elder from your community to help in this task since they can usually more easily resolve problems because of their wisdom and experience.

The Flow of a Focus Group

While there is no "right way" to moderate a group, there are typically four main components of a group. Use them as a loose guide when planning your groups.

Introduction

• Approximately 10 minutes
• Explains the general purpose of the discussion (e.g. "to discuss health care")
• Sets the framework for the group (see table)
• Allows people to introduce themselves (typically first name, where they are from)
• During the introduction, it is often a good idea to get people to say something "easy" and "non-threatening" as a way to break the ice and ease people into discussion (e.g. favourite food or TV show, dream vacation, etc.)

Developing Rapport and Easing into Discussion

• 10 - 15 minutes
• Typically includes general questions that help ease respondents in revealing more detailed, and sometimes, more personal opinions

Discussion of Key Issues

• 60 - 75 minutes
• This is the heart of the investigation of issues
• Focus is on the most important, in-depth topics

Wrap-Up

• 5 minutes
• Summarize overall opinions; seeks any final information
• Thank them for their time and sharing
• If an incentive is offered you can distribute it or tell them how they will receive it

Tip!
Make people - not paper - your focus. It is tempting to take notes but this will distract you and the respondents. Having a note taker and/or a tape recorder is much more preferable.

Ensuring Good Communication

• Encourage differences of opinion.
• Do not tolerate respondents putting down the opinion of others.
• Watch what people do in addition to what they say.
• Use open-ended questions when possible.
• Resist the temptation to always "keep the conversation going". Sometimes a long pause is a great way to have someone come up with something to say (as many people feel uneasy when there is a lull in conversation).
• Use eye contact and show sincere interest.

Tip!
Keep probing. That is to say, look for opportunities to ask additional questions on a topic. Often this will reveal more detailed underlying perceptions and opinions.

Setting the Framework

These are some typical items you may wish to cover off in your introduction. They are sometimes referred to as "disclosures", "rules", or "housekeeping items".

• Introduce the note taker if you have one.
• You will keep all information confidential (i.e. only report in general with no names attached). Ask that they keep what is said during the groups inside the room.

Sample Rules

• Please talk one at a time.
• Please talk loud enough so everyone can hear.
• Avoid side conversations with your neighbours.
• You do not have to answer every question but everyone participates.
• You will not be called on to answer each question so jump in at any time.
• There are no wrong answers.
• Differences of opinion are encouraged.
• Say what you believe even if everyone else disagrees.
• You do not need to raise your hand before speaking.

Tip!
While there is no minimum time for focus groups, most groups run from 1 1/2 hours to 2 hours in length. Any longer starts to become counter productive as respondents get tired and restless. Of course, if you do not have enough to discuss for more than an hour, that is fine too!

Analyzing the Information

You have developed a plan, chosen your method of collecting the information, and gone out and gathered the information. So what does it all mean? This section is divided into two parts. The first provides suggestions for analyzing survey data while the second provides tips for analyzing feedback from interviews and focus groups.

Tip!
Consider organizing the analysis into a Conclusion- Recommendation link. Determine what you can conclude from the results and link those conclusions to what you would recommend.

Review your project purpose
Regardless of the method used, you will save yourself considerable time and effort by reviewing your original project purpose - the reason you did the evaluation - before you begin your analysis. This will help to focus your thoughts before getting into the data.

Analyzing Surveys

You have conducted the survey and now you need to determine what the results mean.

Coding Questions

As outlined on page 13, there are two main types of questions you will have asked in your survey:

• Closed-ended Questions
• Open-ended Questions

It is worth mentioning these two types again because closed-ended questions are easier to work with because you have a defined number of choices.

While open-ended questions are useful, they do require more work in analyzing. Coding is a technical term used to describe the grouping of similar responses. Unless you are familiar with coding or have someone at your disposal that is, you might want to try to structure your survey to have closed-ended questions only.

Data Entry
In order to analyze the data from surveys, you can compile the information by using a computer program. If you don't have accessible computer capacity on hand, you may want to ask someone who has experience with computers. Several good computer programs (spreadsheets) such as Microsoft Excel and Lotus are available to accomplish this task.

Again, it helps to have someone who is comfortable with numbers and data entry with spreadsheets.

Numbers, numbers, and more numbers!

Not everyone is a number cruncher. Some people don't like it while others do not find it comes easy. Find someone who likes it and has the skill set to conduct the analysis.

Tip!
Organize your analysis by the sections used in your survey or discussion guide. Quite often there is a natural link of topics, themes or categories. This will help make your analysis more workable.

Analyzing Focus Groups and Interviews

This section provides suggestions for sorting through your notes from focus groups or interviews.

Organizing Your Information

Unlike analyzing numeric data, analyzing the information from focus groups or one-on-one interviews is more subjective. That said, there are some ways to make the process easier and more reliable.

• Start with making your own "top of mind" notes following each group or interview. No matter how much you may think "I won't forget", the longer you wait the less you will tend to remember.
  
• If you were fortunate enough to have someone take notes for you, review their notes following the project.
  
• Organize comments into similar categories (e.g. concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, recommendations, outputs, outcome indicators, etc.)

Tip!
Know when to quit. You can often keep looking at the data and crunch more numbers. Once you feel you have the most important information, do yourself a favour and begin writing the summary report.

Know Your Limitations

Remember that information from focus groups is not meant to be calculated. Avoid trying to create percentages or attach numbers.

Typically, your evaluation is based on feedback from a relatively small number of people. Keep in mind that the information should be used to identify trends. This is not to suggest it is not valuable. It is.

Tip!
Get someone who takes good notes to record the feedback during the focus groups or interviews. For a number of reasons, it is not recommended that you take notes if you are the focus group moderator or interviewer.

Reporting the Information

Everyone has a unique style and preference when either writing or reading a report. The best tip is to write in a way that is comfortable for you. This section provides you with some ideas for consideration.

Sections to Consider for Your Report

Background
You never know who is going to read your report. A short background of the project can be very helpful. Keep it as short as possible, asking yourself "What information would I want to know about the background of this project if I was not familiar with it? "

Objectives
List the objectives of your evaluation. This lets the reader know what the purpose was for the evaluation.

Methodology
This describes what you did. In other words, did you do a survey? Focus groups? Interviews? A combination of these? When? Who did you speak with? (Not the actual names but the profile of your respondents)

Conclusions & Recommendations
This summarizes what you learned from the evaluation and the impact this learning has on the program.

Detailed Findings
This section is for those who want all of the information gathered from the study.

Executive Summary
The Executive Summary is typically anywhere from 2 to 5 pages in length. It provides a quick summary of the background, objectives, and methodologies but focuses mainly on the conclusions and their implications.

Tip!
Less is more. Avoid extra details if they are not required. Consider using bullet points, charts, and tables to make it easy to read and interesting for the reader. Find a report which has a style and format you like and try to use that as a guide in creating your report.

FNI HCC Examples

Based on the FNI HCC logic model, the following examples demonstrate parts of the model in a modified format, providing a number of examples of questions that might be considered if evaluating some of the associated activities, outputs and outcomes.

This is not meant to be a long list of outputs, outcomes or related questions. Rather, it is intended to help serve as an example of how the logic model can be used in a systematic way to identify key issues to explore in evaluation.

Tip!
While many people might read your evaluation report, the most important issue is to make sure it is relevant for you and your community's needs.

Example 1

Activity
Provide Homecare Services to Eligible FNI populations

Output
HCC Services for Eligible FNI populations

Immediate Outcome
Increased FNI access to HCC Services - Essential and Supportive Elements

Intermediate Outcome
Responsive, Flexible Use and Provision of HCC Services

Ultimate Outcomes

• HCC Clients Maintaining Optimum Health
• HCC Clients Remaining in Their Homes and Communities

Evaluation Questions to Consider

• Which home care services have been delivered as a result of the program, that were not available before? Which have increased? Which have become of higher quality? Has anything been affected negatively in terms of type, amount or level of service? What are the reasons behind these changes?
• What statistics have been reported in the service delivery and reporting template? What are the reasons for any change (or no change) in the numbers?
• Has the program been implemented as intended? How does what was planned compare to what has actually been done?
• Is the HCC program meeting general home health care objectives?
• Does the FNI HCC Program meet the needs of the community?
• Are clients satisfied with the amount and quality of care?

Example 2

Activity
Create/Develop Partnerships in Support of the HCC Program

Output
Partnerships, Service Delivery Agreements, HCC Best Practice Models

Immediate Outcome
Increased Community Capacity and Human Resources to Deliver and Administer the HCC Program

Intermediate Outcome
FNI Delivered and Administered Homecare Services

Ultimate Outcomes

• HCC Clients Maintaining Optimum Health
• HCC Clients Remaining in Their Homes and Communities

Evaluation Questions to Consider

• Does the community have the human resources to deliver these services?
• If no, what are the gaps? Can these be overcome with training?  What are the current skill sets?
• What about administrative resources?
• Have the Home Care services been integrated effectively with Primary and Acute care services?
• How many nurses, support service providers, and administrative staff are there according to the Service Delivery and Reporting Template and other studies? What are the reasons behind these numbers?
• What partnerships have been developed with government, health care authorities, and/or health care providers?

Example 3

Activity
Educate, Communicate, Share and Listen to/with Community Leaders, HCC

Eligible
Populations and Service Providers

Output
Communications and Educational Products for FNI Populations, Community Leaders, F/P/T Governments, Certified Training Program for Care Providers, etc.

Immediate Outcome
Increased Service Delivery Awareness and Accountability - Informed Users of HCC Services and Informed Community Leaders

Intermediate Outcome
Sustainable, Culturally Sensitive HCC Services

Ultimate Outcomes

• HCC Clients Maintaining Optimum Health
• HCC Clients Remaining in Their Homes and Communities

Evaluation Questions to Consider

• Are HCC users aware of services? Why or why not?
• Are community leaders aware of services? Where do they feel there are gaps?
• What issues are there surrounding training for care providers? Is the training sufficient?
• What do service providers have to say overall about the program?
• What areas are working well and what areas could use improvement?
• What do users/clients of the service think?
• What about community leaders?
• Have communications documents been developed and delivered to relevant audiences in the community?
• Are the services in line with the culture within the community? If not, what areas are not compatible?

Example 4

Activity
Administrate/Manage the FNI Home and Community Care Program

Output
Needs Assessments, Service Delivery and Training Plans, Knowledge Management and Information Systems (financial, ongoing monitoring and evaluation)

Immediate Outcome
Infrastructure to FNI Communities to Deliver the HCC Services - Capital and Program

Intermediate Outcome
Sustainable, Culturally Sensitive HCC Services

Ultimate Outcomes

• HCC Clients Maintaining Optimum Health
• HCC Clients Remaining in Their Homes and Communities

Evaluation Questions to Consider

• What facilities do we have for the administration and delivery of the FNI HCC program in the community?
• Are they sufficient and sustainable?
• If not, what are the gaps?