Literature Review

Evaluation Strategies in Aboriginal Substance Abuse Programs: A Discussion

II. Evaluation of Substance Abuse Programs

To be a worthwhile exercise, evaluation must be more than simply an objective look at processes and activities of a organization or program. It must explore the needs of the community and individuals affected by the organization or program being evaluated, and through outcome measurement, get a sense of how well these needs are being met. Most evaluations are constrained by cost and time, and the challenge therefore becomes selective choices of what to evaluate in the most cost-effective manner.

An organization that chooses to undergo an evaluation must consciously accept and welcome the process in order for it to embrace change and be transformed and strengthened and thereby more effectively and efficiently meet its mission. As evaluation has (similar to many other aspects of Aboriginal programs) its roots in institutions external to Aboriginal communities, it often has been adopted grudgingly. If the impetus for an evaluation comes from within the organization, the attitude and spirit of the program will be a central concern along with the operation and outcomes of the program's system.[Jorgenson, Ron. 1987. Trust the Process: Naturalistic Evaluation. Nechi Institute] At times evaluation is seen as a legitimate method for getting rid of a program, therefore the community will not welcome what is seen to be a threat to a needed service. Ultimately, evaluation should be integrated into a program, so that evaluation becomes the responsibility of the organization, and staff are participants not merely observers of the process.

The limitations to western evaluation in an Aboriginal context have been provided by the Four Winds Development Project. Although these limitations were directed at educational evaluations of students and learning, the relationship to substance abuse is two-fold. Firstly, education was seen as one of the central mechanisms to heal communities from the ravages of substance abuse. In this context, human development is a learning process from which native people will learn to respond creatively and positively to challenges in their environment which have been the root causes of alcohol and drug abuse. Secondly, the medical model of substance abuse treatment uses similar values to education, that is focussing efforts on persons judged to be deficient in certain areas (as determined through assessment) to bring those persons to a norm or standard of that society.

Although the Four Worlds holistic approach to evaluation was developed in the mid 1980s, the perspectives of this organization are still relevant when developing contemporary evaluation strategies. The following limitations of existing models of evaluation and screening procedures were described:

• the focus of evaluations on the individual in which a standard or normal person is defined and each person is then compared to this standard. In the Four Winds holistic view of humanity, screening and testing instruments must arise from a philosophical base that interrelates the physical, mental, emotional and spiritual aspects of the individual.
• evaluations usually focus on just one aspect of a system - that is the student or the treatment client. If this person does not meet the standard, the emphasis is on remedial action. Usually the system itself - staff, curriculum materials, program philosophy etc. are not tested or evaluated in any systematic way.
• remediation activities do not transform the context or environment in which certain behaviors arise, rather they isolate the behavior from its normal environment, and attempt to correct the inadequate behaviors. This is contrary to a community-focused holistic approach to treatment.

It is apparent through the preparation of this paper, that few Aboriginal substance abuse programs have been formally evaluated, although it is generally recognized that there are good programs in existence reporting successes in both drug and alcohol abuse treatment. Evaluation is an activity that requires a sizable investment of time and resources on the part of programs, that may already be burdened by large and demanding client populations and waiting lists. Evaluation is a learned skill, and the sporadic nature of it by most organizations ensures that outside expertise is required - again consuming scarce resources.

The process of evaluation is well established in health programs, and will not be summarized here. Health Canada has provided a framework for Aboriginal communities to follow when evaluating health programs.[Health and Welfare Canada. n.d. A Handbook for First Nations on Evaluating Health Programs. (Ottawa: Minister of Supply and Services)] The Health Canada publication includes how to prepare for and conduct an evaluation plan, and how to carry out the evaluation study. As well, scientific literature abounds with information on evaluation in general.

This paper will consider the indicators used to judge effectiveness of programs, the types of evaluation and sources of information used in evaluation. This will be followed by models of evaluation seen Aboriginal substance abuse programs.

Indicators of Effectiveness of Substance Abuse Programs

In treatment program evaluation, the customary indicator to judge effectiveness has been the rate of success in achieving abstinence from alcohol and drugs at some point past the completion of the program by individuals. This has classically been measured by the rate of post-treatment relapse among completed clients or the long-term "clean and sober" rate. The methods used to measure a relapse rate are often subject to error, as individuals questioned may have reasons for hiding a return to their addiction. As well, many previous clients may have moved or are otherwise unavailable for follow-up. In some evaluations, a large component of follow up clients simply will not answer questions about abstinence.

The emphasis on "clean and sober" as a criterion for successful outcome has been criticized as not recognizing the value of programs in reducing consumption rates, as some programs may place value on an individual's success in controlling the amount and frequency of drinking or drug abuse. The post-relapse rate is often not comparable among different programs, again due to the internal values of each program. For example, some programs have achieved success in reducing consumption levels among persons classed as non-completers who have either voluntarily left the program or have had their stay prematurely terminated by staff due to non-compliance with program policies. In this case, the rates that are measured take into account all program participants not just program completers.

There are a number of factors which have been shown in controlled studies in the literature to be correlated with treatment success (negatively or positively).[The scientific literature is full of contradictory evidence regarding the effect of various factors, such as age or mental health on the outcome of treatment. The reader is referred to the following for a thorough review of mainly American non-Aboriginal studies on this topic: Round Lake Treatment Centre. 1992. Research on Native Adolescents and Substance Abuse. The Next Generation Native Adolescent Substance Abuse Project] To understand and interpret correctly the results of a program in terms of success with achieving client and program goals, and to judge comparability among different program populations, it may be necessary to evaluate these factors. In the following (n) refers to results of research which have suggested a negative correlation with treatment success, and (p) is a positive correlation with treatment success. Those factors listed below without accompanying terms have shown variability among studies.

1. Factors outside of the program's control (usually client history and environment)

   • demographic variables: age, ethnicity

   • history of abuse: decreasing age of onset (n), severity (n), primary drug of choice

   • criminal history (n)

   • psychiatric history

   • education failure and dropout (n)

2. Factors related to the treatment program: It is now widely accepted that effective substance abuse treatment may rely on customizing therapies to individual clients. It is therefore difficult to evaluate the effectiveness of any one therapy, rather the model or direction of the program is used as a proxy for treatment strategies. However, more standardized indicators include:

   • residential treatment length (p)

   • outpatient treatment length (n)

   • client perceptions about being in treatment and attitudes toward treatment

   • number of years counselors have worked in program (p)

   • number of volunteer staff in direct client contact (p)

   • counselors' use of practical problem solving approach (p)

   • provision of special services (recreational, vocational, and contraceptive) (p)

3. Post-treatment factors:

   • drug cravings (n)

   • lack of involvement in productive activities (n)

   • lack of involvement in leisure activities (n)

This information on correlations between outcome and other factors was obtained from studies that were not directed at Aboriginal populations. A recent client outcome study completed by the Round Lake Treatment Centre investigated the relationship between outcomes of clients who reported being clean and sober at 3 months, 1 year and 2 years with a variety of demographic and life history factors.[Round Lake Treatment Centre. 1996. Client Outcome Study: Final Report] Demographic characteristics of persons who had completed the program (completer clients) considered in the study were: gender, age, marital status, educational status, and presence of children. Life history characteristics were types of life history trauma, types of substance abuse, and treatment history (prior to Round Lake). The follow up survey looked at the parameters of abstinence (at time of survey), family relations, quality of life, and self image. The following significant relationships were seen with the client data:

• The study found the following factors were significantly related to being clean and sober at one or more of the follow up periods (3 months, 1 year and two years): gender (female), increasing age (over 41 years), marital status (married or common-law), not raised in foster home, history of sexual abuse, history of physical abuse, spouse or family with a history of addiction, and no history of arrest.
• The following factors were significantly related to improved family relations at one or more of the follow up periods: living with someone (compared to living alone), post secondary education, never attended residential school, family history of alcoholism, and prior treatment for substance abuse.
• With respect to the same analysis for quality of life, the following factors were significantly related at one or more of the follow-up periods: attendance at residential school, and history of physical abuse.
• With respect to the same analysis for improved self-image, the following factors were significantly related at one or more of the follow up periods: employment at intake, presence of legal history, and spouse with a history of addiction.

The only factors predictive of program completion were related to life history factors: history of physical abuse (more likely to complete the program), history of intimate partner abuse (less likely to complete program) and client in treatment before (more likely to complete program). There were no client demographic factors which were related to program completion.

Interestingly, no association between client satisfaction and any of the outcome indicators were found as over 90% of the clients were satisfied with all treatment components. There was no satisfaction data on non-completer clients as the satisfaction survey was administered at discharge. The review attributes this high satisfaction rate to be related to a "positive halo" about the program and because clients may not have been totally honest about which parts of the program were most successful. Also respondents may have believed that this program was their only option so they had better make it work for them.

The Correctional Service of Canada has itemized the characteristics of effective substance abuse treatment programs in correctional facilities.[Correctional of Canada. 1992. Creating an Informed Eclecticism: Understanding and Implementing Effective Programs: A Focus on Substance Abuse. (Ottawa: Correctional Service of Canada)] These are:

Multifaceted treatment: The treatment is not based on a single method, but relies on a variety of different treatment modalities and uses a number of specific techniques.

Intensity: Services should be of sufficient duration and sufficient intensity to ensure that skills development and change can occur.

Integrity: The program delivery must conform to the program principles which have been established.

Quality staff: The quality of staff will largely determine the effectiveness of the program, as they must motivate clients to collaborate in exploring their skills and abilities and maintain their interest in training activities.

Well-trained staff: Staff must have the skills required to impart the knowledge or skills that the program aims to teach.

Supportive environment: This includes the staff and actual physical environment.

Good management: Programs must be well managed in terms of program intensity and quality control.

Cognitive-based: The program and its requirements are directed toward changing the attitudes of clients which thereby impact behavior.

Proper selection of program participants: Programs should define, in the methods used and approaches taken, for whom this program is best suited (i.e. target population), and under what conditions.

Proper evaluation: Programs should include the means by which the program and participants can be evaluated (pre and post-program) so as to determine what, if anything, is changed, modified or developed. The evaluation must be able to determine if the program has been successful and for whom.

Types of Evaluation

Evaluation should be an on-going function of an organization, with links to the planning and operational components of a program. However, evaluation is dependent on established objectives, from which progress in meeting those objectives can be measured. There are three broad types of evaluation:

Needs assessments: This is conducted at the planning stage of a program, and should address the needs of the community and its goals.

Process evaluation: Synonyms of this include operational review or formative evaluation. Although this is the most common type of evaluation as it looks at the operation of a program, it is often done poorly or superficially. It ignores a vital aspect of the success of a program - its outcome or results. The value of this type of evaluation lies in its attention to improving quality, efficiency and cost-effectiveness of operations.

Process evaluations describe the extent to which the program was implemented as planned, identify implementation problems for future improvements in program delivery, examine the logical sequencing and applicability of program components and how effectively they work in the field, and describe the subjective views and experiences of program staff and participants (e.g. client feedback).[ibid]

Outcome evaluation: This type of evaluation measures the impact of a program on the target population, and therefore is completed usually after the program has been in existence for some time. In order to effectively measure the impact of a program, it will be necessary to talk to the community and to recipients of services as well as to the program staff.

In an outcome evaluation, there are three types of effectiveness that can be measured:[ibid]

• Intermediate outcomes: Did the program change attitudes and behaviors that is crucial to reducing substance abuse?
• Ultimate outcomes: Does the program actually reduce substance abuse? What is the impact on the community?
• Differential effectiveness: Is the program more effective with some clients than with others?

Sources of Information for Evaluation

In substance abuse program evaluation, both quantitative and qualitative data are important. Quantitative data will provide hard proof on the impact of a program such as lowering substance abuse rates, and recording improvements in individual functioning and well-being. The evaluation of a program as a whole requires qualitative data in a dynamic approach that obtains information from a variety of sources, including clients, staff and community.

1 Client data

An information system is an essential tool to a comprehensive evaluation of a program. It is a systematic effort directed towards the accumulation of quantitative information to assist with the ongoing operation of a program as well as to provide the foundation for its evaluation. The information system can collect demographic, life history and substance abuse data on clients entering the program, which later can be used to interpret completion rates, outcome rates etc for trends. An example is the Treatment Activity Reporting System (TARS) of the National Native Alcohol and Drug Abuse Program (NNADAP) program which has provided valuable information on client demographics, including history, type and severity of substance abuse. The TARS data also provides information that will allow analysis on the relative cost-effectiveness and performance of in-patient programs among regions and different programs: bed utilization, non-completions, recidivism, and average cost per bed expended.

Standardized assessment tools provide a quantitative evaluation of client progress during treatment. These are often specific to different health professions, e.g. psychiatry and psychology. They can assist in the evaluation of a program if a discharge assessment is done in addition to the assessment at admission to the program.

To effectively evaluate treatment in the continuum of care, information systems must also include data on community services, outpatient services and aftercare. This will require an integrated or seamless information system that can be accessed by various health providers as the client moves through the treatment process.

2 Organizational Information

As the provider of the service, the program itself is a important repository of information for evaluation. An evaluation should include mission, goals and objectives of the program, systems management, standards of service delivery, qualifications of staff, staff morale, environment, program policies and procedures, client and staff satisfaction with the program, cost of program (per participant or by other measures) as well as observations about the actual functioning of the program.

3 Community Information

Although used more often in needs assessments prior to establishing a program, the focus on community involvement in Aboriginal substance abuse prevention and treatment suggests that community surveys and forums should form an integral component of a substance abuse program evaluation. Some evaluations of Aboriginal programs make reference to the views of community members, however the methodology is rarely included, and it is difficult to ascertain how comprehensive the consultation has been. In general, the community perspective is important to measure the value of the program, the concern of the community over the issue of substance abuse, the impact the program has had on the community, the level of personal involvement of community members, as well as suggestions for improvement to better meet the community needs. This information can be obtained through surveys or community forums. Also key informant interviews can be held with persons in the community who have first hand knowledge of the program and the community in general.

4 Focus Groups

Focus groups are broadly defined as a technique whereby 8 - 12 individuals discuss a particular topic of interest under the direction of a facilitator. The facilitator directs the conversation to ensure that the evaluation objectives of the activity have been met. The primary data produced by focus groups are the transcripts of the group discussion. Focus groups can aid an evaluation by (1) obtaining general background information about a program, (2) diagnosing program problem areas, and (3) gathering information about client's impressions of a program. Focus groups can be comprised of program recipients (including program clients, family, board members etc.). By targeting issues for focus groups that are important or relevant to clients, the process can empower clients provided the results of these focus groups are shared with the participants and the ideas are enacted. Research has supported the use of focus groups in physical rehabilitation,[Race, K.E., D.F. Hotch and T. Packer. "Rehabilitation program evaluation: Use of focus groups to empower clients." Evaluation Review. Vol 18, Nol 6, pp. 730-740] but the area of mental rehabilitation such as substance abuse has not been evaluated.

The Evaluation Team

The choosing of evaluators is often a difficult task as although the need for an external, outside observer is recognized to bring an objectivity to the process, this person should hold similar values to the community and program being reviewed, or at the very least, not impose foreign values on the evaluation process. External reviewers can be peers from a similar type of program or a professionals specializing in evaluation services. Formal evaluations conducted by external reviewers, however, do not preclude ongoing reviews of various activities in the program solely by staff and community. This method of continuously assessing the quality of a program by program staff should be complementary to the external review process, and form one of the review parameters.

Evaluation models

1 Naturalistic Model

The Nechi Institute conducted an evaluation in 1987 which was based on the naturalistic model and which used four stages: (1) awareness, (2) need identification, (3) knowledge and skills development and (4) integration.[Jorgenson, Ron. 1987. Trust the Process: Naturalistic Evaluation. Nechi Institute]

Awareness: Program participants and community members were included as both principal and associate evaluators. This facilitated the learning and teaching component of the evaluation, as well as the on-going mediation, negotiations and discussions. At the board level, awareness was fostered by the development of a paper on evaluation. As well, the team of evaluators were visible in the First Nations community. In the choosing of the evaluators, openness, value orientation, skills and credibility were essential.

Need identification: The evaluation process was internalized as a useful means of bringing about positive change, and was incorporated into the internal process (workshop and curriculum evaluation, personal evaluation). Through familiarity with the evaluation, the process was seen as facilitating expansion and refinement of the program.

Skills and knowledge about evaluation: Through the process, Nechi became more skilled with evaluation. The relationship between the evaluators and participants was one of mutual teaching and learning through the process of negotiation and discussion.

Integration of evaluation: The evaluation process is built into the institutional structure, as part of the strategic plan, the annual report, as well as operationally in staff and activity review. The need for an outsider perspective remained, and was included in the formal evaluation team. The evaluation team included a native evaluator with sensitivity to the relationship of the program to the wider First Nations community, an evaluator from a previous review who could visibly note change and progress, an evaluator new to the program to bring fresh objectivity, and the program administrator as the agent of change, who could motivate the organization to accept the results of the recommendation.

In the Nechi evaluation, a successful approach was to begin the evaluation without any preconceived systems of evaluation to govern perceptions. As behaviors, systems and patterns emerged from the observation process, they were discussed with staff and administration, and a mutual understanding about observations and recommendations was concluded. This practice empowered staff and increased their sense of ownership in the process.

The Nechi Institute conducted two naturalistic evaluations, and found that the first evaluation was important in building relationships within Nechi and developing an understanding of the values which underpinned people's behaviors. This allowed the second evaluation to be more valuable. The organization also found that by spreading the evaluation process over a period of months rather than relying on an intensive short-term visit, a relationship and trust developed which was crucial to the success of the process.

2 Best Advice Model

The type of evaluation used in the 1989 review of NNADAP was a "best advice model". This best advice model looked at what ideally should be done in the areas of treatment, prevention and training. The development of this model was based on previous work in the area, in this case the Addictions Research Foundation which undertook the review used their own work and related work in the field to come up with their model. Once the models (covering prevention, treatment and training) were developed, they were used as templates to evaluate how selected NNADAP projects were designing and delivering services. As well as this process-based evaluation by the best advice model, case studies were conducted of 37 selected NNADAP projects. This second approach relied on information gathering from staff, review of case records, and consultations with other stakeholders.

This evaluation approach was subject to cost constraint and therefore limited in that it did not undertake independent research to quantify the success nationally of programs directed to reducing alcohol and drug abuse. It did not evaluate all programs, rather 37 programs in three provinces were selected for the review. The reviewers felt that without a national perspective gained from a full evaluation of all programs, the ability of NNADAP to reduce the incidence and prevalence of substance abuse could not be properly addressed. The reviewers also acknowledged the following limitations in the study design that was chosen:

• it was difficult to make general recommendations on a national program based on a selective review in three provinces.
• the evaluation was a snapshot in time, and therefore could not capture well the effects of change and development over time on the program.
• respondents were concerned over possible funding repercussions of a review and therefore could be reluctant to relay negative information.
• the evaluation did not interview clients of the programs to gain an understanding of the positive benefits of the programs on individuals.
• the selection of programs to be evaluated was not random, and there was not an equal chance for workers to be interviewed. The evaluation team had no input into the selection process thus there were limitations as to the validity of their conclusions.
• the survey questionnaire was not designed appropriately, as many of the terms used were unfamiliar to the field workers questioned.

Overall, this evaluation was heavily process oriented, with little attention devoted to the actual outcomes of the program. Emphasis was on employee statistics (number of staff, time on job, time devoted to program activities), congruence of program with contribution agreement, and alignment with best advice model.

The criteria that were evaluated in the prevention program were:

• a community-based policy
• a comprehensive action plan
• training and orientation regarding programming for Band leaders, Band staff and other social services staff and community volunteers should be in place
• the presence of an integrated prevention effort
• a review process for prevention activities prior to implementation.

The criteria that were evaluated in the treatment program were:

• easy access to detoxification services
• easy and timely access, comprehensive assessment, and a sound, client involved referral system
• the option of a number of treatment methods so as to meet individual client needs
• existence of case management services where appropriate
• an aftercare component
• comprehensive record keeping

3 Outcome Analysis

Outcome analysis is a quantitative method which focuses on statistics concerning program outputs. This is often used as a method for comparing various programs that offer the same service. A recent example of an outcome focused analysis was conducted on solvent abuse centres in the First Nations and Inuit Health Program of Health Canada. In response to the need for solvent abuse prevention and treatment among First Nations and Inuit, Health Canada established funding in 1995 for six interim programs of residential treatment in existing solvent abuse programs across Canada. These programs were classed as interim, as funding has been announced for the creation of six permanent solvent abuse centres. The centres have various approaches to treatment, and not all are run by First Nations communities, although all receive a high number of clients who are First Nations or Inuit. The programs vary from a culturally-based program set in a bush camp to solvent programs in existing substance abuse treatment centres which may operate on a medical model, use a 12 step philosophy, incorporate medicine wheel teachings, or include behavioral model components. The program operating in the bush camp had a 28 day defined length of stay, the others were considerably longer or open ended.

An evaluation was done on the treatment outcomes of these centres in 1996[Glen Murray Ltd. 1996. Solvent Abuse Treatment Outcome Evaluation Study. For Medical Services Branch, Health Canada]. It was based on quantitative data from previous clients, and for many of the parameters, all five of the participating centres were evaluated together. This is despite considerable differences in the few centre-specific characteristics which were included in the evaluation report (e.g. average age of client varied from 16 to 25 years, average length of stay varied from 25 to 284 days). As well, the programs had differences in their policy for rule infractions (including the use of substances) and therefore had different policies on discharge.

A key weakness of the evaluation methodology was determined to be the method of collection of data from persons who had been clients of the centres. The clients were interviewed by program personnel who by virtue of their professional relationship with the clients could have influenced the truthfulness of their answers. As well, the program managers determined who would be the best person to contact - the client or significant caregiver. If an appropriate person could not be contacted, and file notes were used to complete the client data sheets.

Despite these limitations, the programs' data was presented together, presumably because of low client numbers in some of the programs. Data was provided however, on the percentage of solvent use abstainers after treatment by program. The rates varied from 40.9% to 60.0% - difference of almost 50% - although the reviewers did not value the 60% result as the sample size was low (15 clients). They stated that the type of treatment program attended appeared to have made little discernable difference in the number of post treatment abstainers, based on the other programs' rates which varied from 40.9% to 46.2%.

The study concluded that the persons who completed their entire program had a considerably higher treatment abstinence rate than those who did not complete. Interestingly, the programs with the longest (284 days) and shortest (25 days) length of stay had the best abstinence rates (60.0% and 46.2%) respectively, which would suggest that success of programs is largely independent of the physical time spent in treatment. [The length of stay (and percent abstinence) for the other programs were 193 days (40.9%), 181 days (42.9%) and 72 days (45.2%)]

4 Community Evaluation

The views of the community are important to determining the effectiveness and value of a program in meeting a community's needs. The 1993 survey of the Regional Advisory Board of for Saskatchewan NNADAP provides an example of community input into the evaluation process.[Socio-Tech Consulting Services. 1994. Addictions Intervention Needs of First Nations: 1994 and Beyond. Prepared for NNADAP, Saskatchewan Region] Community members were questioned about whom they would contact if they had a problem with drinking or drugs, as well as whom they actually did contact when they had an alcohol problem. Respondents who had attended treatment centres were asked about their opinions on the facility, its program, staff and physical condition. Regarding treatment success, the two indicators used were completion of the treatment program and changes in drinking patterns after treatment. In many of the Aboriginal substance abuse programs reviewed for this report, a community evaluation component is not included or the method to obtain community input has not been described.

5 Accreditation

Accreditation is widely used in Canadian health care facilities to measure the quality of service delivery. It is a client-focused process by which facilities are evaluated against standards of care and service delivery. Typically, an accreditation process utilize external reviewers who will interview all levels of management and staff, as well as clients and family members of clients. The review team will also evaluate the programs in the facility against national standards, and will review statistics and other relevant program data. The accreditation process is not a replacement for ongoing evaluation activities in the facilities (peer review, quality assurance, evaluation of utilization statistics etc.), and it provides a comprehensive review of all activities in relation to the mission of the organization and their relevance to community and client needs.

The NNADAP program developed national treatment program standards in 1992, along with an accompanying framework for an accreditation process of treatment centres. The accreditation process was defined as a system designed to improve the quality of NNADAP funded residential addiction treatment programs through a process of assessment of the services, resource management, organization and operations against national standards and criteria.[ARA Consulting Group Inc. National Accreditation Program for NNADAP Funded Addiction Treatment Centres. Prepared for the NNADAP National Accreditation Program, Health Canada] To date, this accreditation process has not been implemented.