Health Canada
www.hc-sc.gc.ca
Common menu bar links
First Nations, Inuit and Aboriginal Health
Literature Review
Evaluation Strategies in Aboriginal Substance Abuse Programs: A Discussion
VI. Observations
The purpose of this review was not to recommend one type of evaluation or treatment program in substance abuse among Aboriginal people, but rather to present the different approaches to prevention and treatment that are currently being used, and to identify the indicators of effectiveness these programs have used in evaluations. The following observations are pertinent to this review:
- Evaluations of Aboriginal substance abuse programs have concentrated on completion
rates, abstinence rates or decreases in substance consumption as outcome indicators of
effective ness. This quantitative approach suggests that comparisons should be possible
among programs, and therefore the success of different programs can be determined.
There are many differences among Aboriginal substance abuse programs that will preclude
this type of comparison. These include an absence of benchmarks in outcome statistics,
differing program approaches based on various models, differing policies regarding program
completion, varying severity of client populations, differing length of stays, differing views
on positive outcome (i.e. abstinence vs reduction in consumption levels), and differing
populations evaluated (inclusion of all participants in statistics or just those judged to have
completed the program).
There are no established benchmarks for programs to use when evaluating treatment or prevention of Aboriginal substance abuse. For a community embarking on a formal intervention program or mobile service, initially at least, any success in achieving sobriety appears to be welcomed. The issue of how successful the program should be becomes secondary, and in most cases, impossible to estimate due to the varying program approaches seen among communities and the lack of a common benchmark value. In the evaluations contained in this report, reported outcome measures included: eliminated or reduced use by participants (75%), significant long term change of completers (50%) and sobriety (20% - 87% depending on length of time of follow-up.
Some programs (e.g. Selkirk Healing Centre) are challenging the abstinence-focus to outcome measurement, saying that effectiveness of treatment should be evaluated by a reduction in the level of alcohol consumption or drug use. As well, the benefits of the program to non-completers is also recognized by this program, as all participants not just completers are followed to determine post-program success.
National evaluation data from the Treatment Activity Reporting System (TARS) used in the National Native Alcohol and Drug Abuse Program (NNADAP) program includes trends in abuse of different substances and cross addictions, bed utilization, numbers of admissions, number of completions, reasons for non-compliance, client recidivism and treatment costs per day. The TARS data has been judged to be limited by inconsistency of reporting and lack of accuracy. There is a need to ensure the reliability of the TARS data, and also to extend the data collection efforts to encompass the continuum of care.
- Contemporary Aboriginal substance abuse programming emphasizes the role of the
community in developing, delivering, and supporting programs. This would infer that the
community, as an integral part of the treatment program, should also be evaluated. There
are two aspects to community evaluation: (1) the effect of community involvement on the
program, and (2) effect of program outcomes on the well-being of the community. There
appear to be no formal evaluations that have focussed the first aspect. The most commonly
used indicator of community involvement is the presence of a supportive Band council
policy supporting the treatment goals of the program. The second aspect regarding effect
on the community has been commonly inferred from client outcome rates.
- Information on the quality of services provided through programs is also lacking in
evaluations. Undoubtedly some programs must include client satisfaction exit
questionnaires, however these are not obvious in the literature reviewed for this paper, with
the exception of the Round Lake Treatment Centre client outcome study.
There is no evidence of on-going continuous quality improvement or total quality management strategies in the program evaluations reviewed. In fact, quality of service is a rarely used indicator in these evaluations. The objective of these methods are to ensure quality and increase the effectiveness of services in organizations and programs. CQI and TQM concepts and models are well established in both the private and public non-profit sector. The quality circle approach of these methods uses team problem-solving that values contributions of all levels of staff, and would be complimentary to traditional Aboriginal consensus-building.
- Demographic factors, previous life history and prior treatment for substance abuse have all
been shown to have an influence on program outcome. Ideally, a program should capture
this data in an information system, so as to shed light on the reasons for the outcome rates
which have been achieved. These data requirements should be in place during program
implementation in order to obtain baseline data on clients.
- There were no evaluations among those reviewed for this report that looked at staff burnout,
overwhelmed case-workers or other staff-related factors that might prevent an optimal
program outcome. Other areas that could be measured include non-Aboriginal and
Aboriginal conflict (program design or direction), community rejection or lack of support.
- There was little differentiation by severity of alcohol or drug abuse when presenting outcome
results. Contemporary approaches to treatment strategies in substance abuse are becoming
increasingly sensitive to the importance of the levels of use as an indicator of severity.
Levels of use have been suggested to include experimentation, recreation (seeking drugs
out), habituation (psychological dependence), abuse (negative effects ignored) and addiction
(a compulsion to seek drugs). These levels may have benefit in determining early intervention strategies as well as appropriate treatment approaches.
- Follow up outcome data on clients who are re-integrated into the community are subject to
the truthfulness of respondents who may wish to not disappoint program staff. There can
be an appreciable number of clients who refuse to answer questions on sobriety (particularly
if the focus of the program was solely on abstinence, as in the Alcoholics Anonymous
approach). This can lead to two statistics being presented on client outcome which are
often greatly different: abstinence in respondents, and abstinence among persons who
answered that particular question.
- There is no formal mechanism to ensure that minimum standards of care are adhered to.
Although program standards have been developed in NNADAP, these have not been
formally used in an accreditation process nor is there a commonly accepted process by a
recognized body which conducts evaluation.
- Aboriginal approaches to substance abuse treatment appear to be moving away from the
Alcoholics Anonymous model, which sees abusers as not responsible for their actions, and
which uses confrontational techniques. Rather a combined medical and native cultural
model is favoured. In these programs, components of western approaches to treatment that
are complimentary and culturally-neutral are utilized. It is recognized that not all clients may
feel comfortable with a program that is based solely on traditional culture.
- The issue of cost is rarely broached in evaluations. This is not specific to Aboriginal
programs as there is a general lack of cost-effectiveness studies in substance abuse
treatment. Reasons for this include conceptual problems in designing the evaluation,
disagreement over treatment goals, disagreement about outcomes, uncertainty about the
appropriate length of stay in treatment, variability among treatment programs, high drop-out
rates and reliance on self-reporting by clients. [French, M.T.1995 "Economic evaluation of drug abuse treatment programs: Methodology and findings." American Journal of Drug and Alcohol Abuse. Vol 21, No. 1, pp. 111-135]
Areas that are important to evaluate with respect to cost include: What types of clients are most costly to treat? What client types benefit most from treatment? How do benefits compare to costs? In the fiscally constrained field of Aboriginal health care where each precious dollar must do double duty in meeting the great health needs of the population, this information becomes even more relevant than for the general population.
The most common units to evaluate costs are abstinence or reduced substance use, but equally important are the indirect benefits which are more difficult to measure and value. These include quality of life improvements, increases in employment, and reduced criminal activity.
- Effective evaluation requires a commitment by all persons associated with the delivery of a program, including staff, volunteers and administration. Evaluation must be seen as a beneficial aspect of the program design, not as a threat to either the individual staff or the continued existence of the program. Ideally, staff and volunteers should actively participate in the design of an evaluation and its implementation.
