Improving Nurses' Triage Skills through Web-based Learning

Health and the Information Highway Division, Health Canada - 2003

Table of Contents

• Acknowledgements
• Executive Summary
• 1. Introduction
  • 1.1 Background
  • 1.2 The Online CTAS Course
  • 1.3 The Pilot Study
  • 1.4 The Current Study: Research Questions
  • 1.5 Project Researchers
  • 1.6 Project Partners and Funders
  • 1.7 Project Objectives
• 2. Literature Review
  • 2.1 Online Tutorial
  • 2.2 Marks for Online Discussion
  • 2.3 Workplace Activity
  • 2.4 Impact on Clinical Practice
• 3. Research Method
  • 3.1 Design
  • 3.2 Sampling
  • 3.3 Control Group
  • 3.4 Experimental Group
  • 3.5 Criteria for Inclusion
  • 3.6 Measures
  • 3.7 Sample
• 4. Findings
  • 4.1 Course Attrition
  • 4.2 Learner Demographic Survey
  • 4.3 Computer Access and Skills
  • 4.4 Online Learner Support Instrument
  • 4.5 Chart Audits
  • 4.6 Interviews
  • 4.7 Nurses' Experiences
  • 4.8 Volume of Online Discussion
• 5. Discussion
  • 5.1 Overall
  • 5.2 Impact on Practice
  • 5.3 Attrition
• 6. Recommendations for Best Practices in Online Learning
  • 6.1 Overall Recommendations
  • 6.2 Technology
  • 6.3 Learners
  • 6.4 Course Design
  • 6.5 Facilitators
  • 6.6 Workplace
  • 6.7 Educational Institutions
• 7. Health Policy Implications
  • 7.1 Barriers to Implementing CTAS
• 8. Implications for Health Care, Educational, Professional, and Government Organizations
• 9. Study Limitations
• 10. Project Audience
• 11. Conclusion
• 12. Dissemination Plan
• References
• Appendix A: Learner Demographic Survey, Online Learner Support Instrument, and Interview Questions

The Research Team

Principal Investigator:

• Lynda Atack, RN, Ph.D., School of Health Sciences, Centennial College

Co-Investigators:

• James Rankin, RN, ACNP, Ph.D., Faculty of Nursing, University of Calgary
• Karen Then, RN, ACNP, Ph.D., Faculty of Nursing, University of Calgary
• Robert Luke, Ph.D. (c), School of Health Sciences, Centennial College

Acknowledgements

We would like to acknowledge the work of those individuals who shared their expertise and advice with the project team:

• Doris McLean, B.Sc.N., CTAS Program Coordinator, Centennial College
• Louise LeBlanc, B.Sc.N., Nurse Manager, Emergency Department, Scarborough Hospital
• Diane Duff, RN, Ph.D., Faculty, School of Health Sciences, Centennial College

Appreciation for their support and contribution goes to our partner organizations; their support was invaluable:

• The Change Foundation
• The Ontario Hospital Association

We would like to acknowledge the time and effort contributed by:

• The Emergency Room nurses enrolled in the CTAS course
• Managers and administrators from those hospitals that participated in the chart audit

We would also like to convey our sincere thanks to the Knowledge Development and Exchange (KDE) Applied Research Initiative, Office of Health and the Information Highway, Health Canada. This project would not have been possible without their support.

Executive Summary

Introduction

Patient situations faced by Emergency Room nurses have dramatically increased in type and complexity in the last decade. The earlier three-level system of triage (patient screening) does not sufficiently provide for these complex patient situations.¹ The Ontario Ministry of Health and Long-Term Care recommends the Canadian Triage and Acuity Scale (CTAS), a five-level system which has emerged as a more sensitive, accurate, and reliable tool for rapid patient assessment (Jelinek, 1996). The accuracy of triage assessment done by nurses affects patient health and safety, as well as hospital accreditation and funding. The CTAS training course, originally offered in a workshop format, required large numbers of trainers. In an effort to improve access and standardize CTAS training for nurses, the Change Foundation funded the development of a six-week web-based version of the course.

A research project using an experimental design was conducted to test certain online teaching and learning practices in the CTAS course. The study was funded through the Knowledge Development and Exchange Applied Research Initiative of the Office of Health and the Information Highway, Health Canada. Participants included 124 Emergency Department Registered Nurses from seven provinces and one territory who enrolled in the CTAS course over nine months. The investigators addressed the following policy question: What online teaching and learning practices enhance learner satisfaction and improve the accuracy of emergency nurses' triage skills?

¹ Triage refers to the rapid clinical assessment carried out when a patient arrives in the ER. The patient is assigned a triage code which relates to the urgency with which the patient should be seen by a physician and treatment initiated. Triage level also affects nursing re-assessment.

Project Objectives

1. Describe and compare nurses' experiences with different versions of the online triage course.
2. Measure the impact of web-based learning on nurses' knowledge of triage and on accuracy of triage assessment.
3. Measure gains made in nurses' computer skills.
4. Identify best practices for web-based teaching and learning for nurses.
5. Identify the online support structure required by busy professional learners.
6. Develop guidelines for web-based learning, in order to promote learner satisfaction and enhance nurses' transfer of learning to practice.

Investigators were particularly interested in examining the impact of three variables:

1. A mandatory, interactive, web-based tutorial on online learning and the e-learning platform.
2. Awarding twenty-five percent of the final grade for participation in the online discussion forum.
3. A workplace project, on the following outcomes:
   
   
   • nurses' satisfaction with web-based learning
   • volume of online discussion
   • accuracy in patient triage

Research Design

A two-stage sampling approach was used. Nurses were first identified by work site. All nurses from a particular site were then randomly assigned to either the experimental or control group. Data collection methods included surveys, interviews, chart audit, and online discussion review.

Control Group

Nurses assigned to the control group received the online triage course in its original form. It included exactly the same content and learning activities as the experimental group, with the following exceptions:

• Nurses were encouraged to begin by completing an online tutorial on the Internet and the course platform.
• Nurses were expected to post answers to learning activities in the online discussion forum every week; however, no marks were awarded specifically for the online discussion.
• All evaluation activities were web-based.

Experimental Group

Nurses who were assigned to the experimental group received the identical online triage course as the control group, with the following exceptions:

• Nurses were advised that the online Internet tutorial was mandatory. They were required to demonstrate basic computer skills (e.g., send the teacher an e-mail, post a message to discussion forum) before proceeding to the course content.
• Twenty-five percent of the final grade was awarded for participation in the online discussion forum.
• A further 15% of the final grade was obtained by completing a CTAS course-related project in the workplace.

Findings

One of the study's major findings was that the online CTAS course positively affects nurses' triage skills. Evidence from the chart audit showed a triage accuracy rate of 70% for all nurses. This figure compares favourably when measured against earlier studies and supports the further development of online courses for health care professionals (Travers et al., 2002). Overall agreement with the expert reviewer within one CTAS level was 99.7%, which suggests that nurses can learn and reliably use the CTAS (Murray, 2003). No significant difference was found between the scores of nurses in the experimental group and those in the control group on the end-of-course satisfaction survey, on the volume of discussion generated in the online forum or on the triage accuracy rate. There was, however, a highly significant difference in the types of triage errors made by the two groups. The majority (83%) of clinically significant errors (under-triaging) were made by nurses in the control group.

Another key finding was that the majority of nurses in both the control and experimental groups were highly satisfied with the course. Virtually every nurse expressed interest in further online learning. Numerous benefits of online learning were described by participants. These benefits included improved triage assessment skills and accuracy, connection to peers, improved computer skills, and professional networking. Most nurses reported that the course had improved triage practices in their workplace. Nurses reported that an increased understanding of the CTAS guidelines meant more accurate patient triage, which in turn meant better patient care.

Virtually every nurse completed the tutorial on online learning. Unlike in earlier studies, in this study lack of computer skills was not reported as a barrier to learning. This finding lends support for providing a mandatory course tutorial. Nurses in both groups made significant gains in computer skills during the six-week course. A significant number of nurses reported that marks should not be awarded for online discussion; instead, learning activities and discussion questions should be well designed and sufficiently interesting, so as to encourage participation. The workplace project, completed by nurses in the experimental group, facilitated the transfer of triage learning from the course to the individual and from the individual to other staff and administrators in the ER departments. This knowledge transfer greatly increases the value of the course for health care organizations and ultimately for patients.

Not all experiences were positive. Highly experienced nurses reported being disappointed with the introductory nature of the course content, and desired more complex case analyses. A number of barriers that interfered with the transfer of learning to the workplace were also identified. These barriers included: lack of time; insufficient computer access; no formal triage system in place; triage currently being carried out by ward clerks; insufficient staffing levels; other health care professionals using different triage assessment tools; and the physical layout of the ER department.

Guidelines for web-based learning were developed on the basis of the research findings. These guidelines include best practices for web-based learning for nurses, as well as recommendations for the support structure required to help busy professionals learn online. Implications for health care and for educational policy were also identified.

Conclusion

The current health care environment, with budget constraints and heavy patient loads, demands accurate triage assessment for the safety and well-being of the public. This in turn requires that health care professionals be skilled and accurate in triage assessment. The online CTAS course provided a standardized, effective, and enjoyable educational experience that developed nurses' triage accuracy. Two activities, i.e., an online Internet tutorial and a workplace project enhanced learners' preparation for online learning and the transfer of learning to the workplace. Technology will increasingly be used to help professionals maintain competency and to support professional practice communities. Learning is not a discrete activity. If health care organizations want to develop as viable learning communities, workplace policies to support online learning need to be in place. Findings from this study will be useful to professionals from a variety of health care disciplines, as well as to educators and administrators who are interested in developing quality online programs that improve professional practice.

1. Introduction

1.1 Background

Patient situations faced by Emergency Room nurses have dramatically increased in type and complexity in the last decade (Ontario Hospital Association, 1999). The earlier three-level system of triage (patient screening) does not sufficiently provide for these complex patient situations. The Canadian Triage and Acuity Scale (CTAS) is a five-level system that has emerged as a more sensitive, accurate, and reliable tool for rapid patient assessment (Jelinek, 1996). In 1999, the Ontario Ministry of Health and Long-Term Care made the CTAS mandatory in all Emergency Room departments in the province (Murray, 2003).

In many institutions there is no formal triage training in place (Cone & Murray, 2002). The training that is done is often hospital-based and tends to focus on documentation evaluation rather than on triage accuracy (Beveridge, Kelly, Richardson & Wuerz, 2000). There is an ongoing need for consistent quality improvement and for standardized training to develop nurses' skills with all triage systems, including the CTAS. The accuracy of triage assessment done by nurses affects patient health and safety, as well as hospital accreditation and funding.

The Ontario Hospital Association supports CTAS implementation and has provided nurses with on-site training. The course was developed by Dr. Michael Murray (Chair) and Louise LeBlanc, BScN, members of the National CTAS Working Group. Training was originally offered as an eight-hour, day-long workshop at various sites throughout the province; however, a number of barriers reduced access to the course.

One of these barriers was the challenge faced by hospital managers in releasing staff to attend an all-day workshop, at a time when nurses are in short supply and replacement costs place a further strain on staffing budgets (Hodson-Carlton, 1997). A further challenge was the provision of consistent, quality training across Canada's geographical expanse. In addition, traditional conferences and one-day workshops have been found wanting in terms of changing behaviour and thinking (Elmore, 1996). In an effort to overcome these barriers and improve access to CTAS training, The Change Foundation funded the development and delivery of a web-based version of the CTAS course.

1.2 The Online CTAS Course

The online CTAS course is offered by Centennial College over a six-week period and consists of five modules. Class size ranges from 11 to 31 students. The course platform is Prometheus, a standard e-learning courseware system. Upon successful completion of the course, the nurse receives a certificate from Centennial College.

Instructional design for the CTAS course is based on a constructivist approach and fosters knowledge application by using a case-based approach. Learning takes place in two ways:

• text, graphics, and case situations based on practice, presented in weekly modules
• an online discussion forum, moderated by course facilitator

1.3 The Pilot Study

In 2001, funds were allocated by The Change Foundation to evaluate a pilot version of the web course.

The goals of the pilot version were to:

• explore nurses' experiences with the web-based CTAS course
• explore nurses' perceptions of the impact of their learning on their Emergency Room clinical practice
• make recommendations to improve the course prior to offering the course nationally

Findings from the pilot indicated that nurses:

• were under-prepared, with regard to computer skills at the beginning of the course
• enjoy and require online interaction with the teacher and peers to motivate them and to facilitate learning
• applied their online learning to their nursing practice in the area of triage assessment (Atack, 2001a)

1.4 The Current Study: Research Questions

Based on a review of the literature and the pilot study findings, the decision was made to conduct an experimental study using surveys, interviews, online discussion message review, and chart audits of the Emergency Room nurses' triage scores. Investigators were particularly interested in examining the impact of three variables:

1. A mandatory, interactive, web-based tutorial on online learning and the Prometheus platform.
2. Awarding twenty-five percent of the final grade for participation in the online discussion forum.
3. A workplace project related to the course (i.e., making a triage poster, conducting a staff education seminar) on the following outcomes:
   
   
   • nurses' satisfaction with web-based learning
   • volume of online discussion
   • accuracy of patient triage

1.5 Project Researchers

The principal investigator was Lynda Atack, RN, Ph.D., Faculty of Health Sciences, Centennial College.

Co-investigators included:

• James Rankin, RN, ACNP, Ph.D., Faculty of Nursing, University of Calgary
• Karen Then, RN, ACNP, Ph.D., Faculty of Nursing, University of Calgary
• Robert Luke, Ph.D. (c), Online Learning Manager, Centennial College

An independent project evaluation was completed by Rona Achilles, Ph.D., Adjunct Professor, School of Public Health, University of Toronto.

1.6 Project Partners and Funders

• The School of Applied Arts and Health Sciences, Centennial College
• The Change Foundation and the Ontario Hospital Association
• Research funding was provided through the Knowledge Development and Exchange Applied Research Initiative, Office of Health and the Information Highway, Health Canada.

The policy question addressed by investigators was: What online teaching and learning practices enhance learner satisfaction and improve the accuracy of emergency nurses' triage skills?

1.7 Project Objectives

1. Describe and compare nurses' experiences with different versions of the online triage course.
2. Measure the impact of web-based learning on nurses' knowledge of triage and on the accuracy of triage assessment.
3. Measure gains made in nurses' computer skills.
4. Identify best practices for web-based teaching and learning for nurses.
5. Identify the online support structure required by busy professional learners.
6. Develop guidelines for nurses' web-based learning, so as to promote learner satisfaction and enhance the transfer of learning to practice.

2. Literature Review

In recent years, considerable research has been conducted in the field of online learning. A number of earlier studies have described learners' experiences with web-based learning (Andrusyszyn, Iwasiw & Goldenberg, 1999; Bullen, 1998; Daugherty & Funke, 1998; Novaczek & Gabriel, 2002). These researchers have identified a number of advantages associated with web-based learning, including: time convenience; decreased travel time and costs; potential for developing a community of practice; standardization and quality control; accessibility for rural learners; increased student participation; and development of writing skills. Another advantage identified is the use of technology to support relationships between individuals (John Seely Brown, 2000). The identified disadvantages with web-based learning included: hardware and Internet service costs; computer skills; technical problems; the need for self-direction; and the challenge of communicating with others in the virtual environment (Ludlow, 2002; Roschelle & Pea, 1999).

Findings from these earlier descriptive studies have been useful; however, there is a paucity of rigorous experimental research that explains or predicts phenomena in distance learning (Phipps & Merisotis, 1999; Ludlow et al., 2002). While the results of these studies have been invaluable, the investigators suggest that there is a gap in testing the impact of certain variables on the learners' experience. Based on the literature review and on the findings from the pilot study, a decision was made to examine the impact of a mandatory tutorial, and to award twenty-five percent of the grade allotted for online discussion and a workplace project on:

• nurses' satisfaction with web-based learning
• volume of online discussion
• accuracy in patient triage

A literature review was conducted for the period between 1997 - 2002, using CINAHL and MEDLINE, as well as search terms related to the three variables above.

2.1 Online Tutorial

The primary objective of the CTAS course is for nurses to be able to assess and triage patients more accurately upon completion of the course. A secondary goal is for nurses to develop skills and familiarity with computers and the Internet. A recent Canadian study (Atack & Rankin, 2002) indicated that many nurses' computer and Internet skills are at a beginner level. This finding was supported by findings of other authors, who have noted that many nurses are far from confident with the computer because they lack computer and Internet skills, and in-house help to support them is frequently not available (Alpay & Russell, 2002; Thede, 1999). Cartwright and Menkins (2002) found that 84% of nurses enrolled in a Master's program had no previous computer course experience. These nurses tended to over-estimate their computer skills and they encountered problems once the course was underway.

A more encouraging finding in the reviewed literature is that computer proficiency tends to improve when students are required to use technology (Atack & Rankin, 2002; Billings, Connors & Skiba, 2001). While there is much discussion in the theoretical and applied literature regarding the need for online learners to have a thorough orientation to the special instructional and technical features of the online courses they are taking (Weinstein et al., 2002), studies that specifically examined or tested this variable were not identified.

2.2 Marks for Online Discussion

Allotting marks to encourage online asynchronous discussions is commonly documented in the evaluation and applied literature (Bullen, 1998; Harasim et al., 1995; Higgison, 2001; Read & Gregory, 2001). Many university or college web sites that offer advice to online teachers recommend awarding marks to encourage discussion (University of Phoenix, University of Waterloo, University of Western Ontario). While awarding marks is a common practice in online courses, the search revealed a gap in the evidence-based literature as to whether it is the best practice for practicing professionals.

2.3 Workplace Activity

A number of nurses in the CTAS pilot course mentioned that they had applied their learning from the online course in the workplace. Nurses gave examples of creating posters for the Emergency Room and giving workshops for staff. Since the goal of the CTAS course was to introduce a new triage scale and thereby improve patient outcomes, investigators were interested in examining the impact of a workplace project on nurses' satisfaction with the course and on the accuracy of their triage skills.

The general education literature indicates that course content and the quality of continuing education (CE) programs influence learners' perceptions of the impact of the course on practice (Stahmer, 1995). Programs which are seen as relevant to learners' needs in the workplace have a greater impact on learning (Ferrell, 1988; Nolan et al., 1995; Yuen, 1991). Cervero and colleagues (1986) suggest that on-site courses are more likely to effect performance change, as these CE activities tend to be more congruent with practice-setting goals and values. The need for relevant content and assignments for workplace learners has also been identified (Billett, 1994; Hofstader & Munger, 1990; Schon, 1987). While the general education literature clearly supports linking theory with practice, no studies were identified in the online learning literature regarding the impact of a workplace project on practice.

2.4 Impact on Clinical Practice

Ehrmann (1995) noted that what matters most is not the students' use of technology, but rather how technology brings about personal growth and achievement of personal educational goals. An area of research that has been largely overlooked to date is the impact of online learning on clinical practice. Senior decision-makers from health care organizations are now interested in measuring the impact of online courses on clinical practice (Gale Murray, CEO, The Change Foundation, 2001).

In an earlier study investigating the impact of a web-based course on dermatological office procedures, dermatologists reported improved clinical skills after taking the course (Curran, 1999). A descriptive study of nurses' experiences with web-based learning and the impact of learning on clinical practice was recently completed by investigators for the current project (Atack, 2002). Findings from this study indicated that nurses applied their learning and improved their nursing practice. The Curran and Atack studies and the pilot study conducted for The Change Foundation (Atack, 2001a) provide a useful research foundation, but fall short of identifying cause-and-effect relationships in online education. Research of this nature is needed in order to identify best practices for web-based course development. More objective measures are needed to assess the short- and long-term impact of technology-assisted learning on practice (Ludlow et al., 2002).

Based on the literature review, investigators conducted a study using an experimental design to test the impact of certain existing and new practices in web-based learning on learner satisfaction, accuracy of nurses' triage skills, and volume of discussion in the online forum. This research direction is supported by a report from the Canadian Advisory Committee for Online Learning (2001) which calls for applied research that focuses on how people learn and how learning is effectively imparted, as well as on identifying the learning requirements of the different disciplines.

The theoretical framework used for the study was Lawton's (1997) model of distance education. Lawton emphasizes the interrelationships among students, teacher, course design, and environment in influencing learning outcomes. The subscales in the survey administered at the end of the course and the interview questions were developed on the basis of the variables from the Lawton model.

3. Research Method

3.1 Design

An experimental design was used to determine the effectiveness of certain online teaching and learning interventions. All nurses who enrolled in the online CTAS course from February 2002 to October 2002 were invited to participate. A package which included information regarding the nature and purpose of the study and identified study sponsors was mailed to the students. Nurses who were interested in participating in the study were also asked to give an informed consent by signing the consent form and returning it in the provided, self-addressed, stamped envelope.

3.2 Sampling

Using a power analysis, a sample of 123 nurses was deemed appropriate to test the research interventions. This sample size would give an 80% chance of detecting a difference in the experimental and control groups, based on the literature and the pilot study. A two-stage sampling approach was used. Firstly, all nurses were identified by work site. Following that, all nurses from a particular site were randomly assigned to either the experimental or control group. As it was recognized that nurses working at the same site would share and compare experiences regarding their online course -- which would pose a serious threat to the internal validity of the study -- the worksite was first identified. A total of five intakes were included in the study. The same teacher taught all the control and experimental group students, with the exception of one large spring intake where, in addition to this regular teacher, two other teachers taught the course; one taught a control group and the other an experimental group. All teachers were briefed on the study and followed the same content, learning activities, and evaluation methods.

3.3 Control Group

Nurses who were assigned to the control group received the online triage course in its original form, which included exactly the same content, learning activities, and final exam as the experimental group, with the following exceptions:

• Nurses were encouraged to begin by completing an online tutorial on the Internet and on the course platform.
• Nurses were expected to post answers to learning activities in the online discussion forum every week; however, no marks were awarded specifically for the online discussion.
• All evaluation activities were web-based.

3.4 Experimental Group

Nurses who were assigned to the experimental group received the identical online triage course as the control group, with the following exceptions:

• Nurses were advised that the online Internet tutorial was mandatory. They were required to demonstrate basic computer skills (e.g., send the teacher an e-mail, post a message to the discussion forum) before proceeding to the course content.
• Twenty-five percent of the final grade was awarded for participation in the online discussion forum.
• A further fifteen percent of the final grade was obtained by completing a workplace project related to the CTAS course.

3.5 Criteria for Inclusion

Participants were Emergency Room Registered Nurses who were taking the online CTAS course. Participants needed home or workplace access to the Internet. Ethical approval to conduct the study was obtained from the Centennial College Ethics Review Committee. Formal approval to conduct the chart audit was obtained from participating clinical sites.

3.6 Measures

Method triangulation was used in order to provide a comprehensive picture of the online learners' experiences and triage accuracy. By combining methods, the investigators were able to add depth and breadth to their understanding of the phenomenon under study (Begley, 1996; Shih, 1998). Data collection commenced in February 2002 and was completed in December 2002. Data collection methods included:

• surveys
• interviews
• chart audit
• online forum discussion review

Two surveys were administered: the Learner Demographic Survey (LDS) and the Online Learner Support Instrument (OSLI).

3.6.1 Attrition Records

Any nurse who withdrew from the course was queried about the reason for withdrawing, and the results were tabulated.

3.6.2 Learner Demographic Survey (Pre-Course)

All participants completed the Learner Demographic Survey (LDS) at the start of the six-week course. The LDS is a 21-item survey that provides a description of the participants and measures demographic information such as age, years of experience, hospital location, and education level. The LDS, which has been used previously (Atack & Rankin, 2002), also measures computer and Internet skills at the start of the course, computer course experience, computer access site, and learning style (see Appendix A). Descriptive statistics were used to compile a profile of the participants.

3.6.3 The Online Learner Support Instrument (Post-Course)

Nurses' experiences with the course were assessed using the Online Learner Support Instrument. The OLSI is a 52-item self-report instrument, developed and used previously by the investigators (see Appendix A). It measures learners' experiences in five sub-domains: 1) Interactions with Teacher and Peers; 2) Course Design and Resources; 3) Technology; 4) Environment; and 5) 'Overall Impressions' scale. A full description of the validity and reliability testing as well as the OLSI items is available (Atack, 2002). Alphas for the current study for all the subscales were over 0.80, with the exception of the work environment subscale, where the alpha was 0.76.

A section of the OLSI includes six self-report items that measure post-course computer skills and triage application. Descriptive statistics were used to summarize total and subscale scores on the OLSI within groups, for both the experimental and control groups. A Kruskal-Wallis test was used to compare scores on the OLSI between groups. T-tests were used to compare the two groups' responses to certain items on the OLSI, including pre-test and post-test scores for computer skills.

3.6.4 Chart Audits

A second stage of the study, which involved the final two intakes for the CTAS course, included a chart audit. Nurses enrolled in the CTAS course were working in hospitals across Canada -- from the Queen Charlotte Islands of British Columbia to the coast of Labrador. With a few exceptions, only two or three nurses from each site were enrolled at any one time. In order to facilitate data collection, investigators used the course database to identify clusters of nurses who were working at the same site. Fourteen sites were chosen by random selection; stratification was used in order to ensure that urban and rural sites from different provinces were represented. Emergency Room directors were contacted and an information package was sent to each site. Ethical approval was requested in order to conduct the chart audit, and directors were asked to identify personnel who would retrieve records.

A total of fourteen sites were approached to participate in the study; however, over half of the sites were ineligible or chose not to participate, for several unexpected reasons. Four sites reported that they did not use a formal triage process or that triage was done by ward clerk. One site did not have the resources to support data collection. One site was unable to conduct an ethical review within the study time period. Two sites had never participated in a research study and did not have an ethics committee in place to review the study. All six remaining sites were included in the chart audit.

In each of the participating sites with CTAS graduates, a shift was randomly selected for audit. ER charts for patients admitted on that shift were obtained. The ER record was photocopied, and any identifying information was removed in order to protect patient privacy. A total of 381 charts were returned. Three hundred and sixty-seven were audited; 14 charts were discarded, because the triage code was missing or illegible. The chart audit was conducted by an Emergency Room nurse - a CTAS expert - who acts as a provincial auditor for Emergency Room triage evaluation. The ER expert was blind to the nurses' participation in either the control or experimental group. The reviewer began by first looking at the presenting complaint and the triage nurse's documented assessment. The reviewer assigned her own triage code, and then compared that code to the code assigned by the CTAS nurse. The expert reviewer verified her triage designation by using current CTAS guidelines (Beveridge, Clarke, Janes & Savage, 1999). This methodology was based on the 'gold standard approach' described by Travers and colleagues (2002). The accuracy rate for both groups was calculated and, because nurses' triage scores were being compared to one expert's (the gold standard), a Fisher's exact test was used to test for significance. This test was also calculated to test for significance in the types of errors that were made.

3.6.5 Interviews

All participants were e-mailed at the end of the course and invited to participate in a telephone interview in order to discuss their experiences with the course. Twenty-three nurses from seven provinces and one territory were interviewed. Data saturation was observed around the 14th interview. However, the investigators decided to continue data collection, in order to record the experiences of nurses in the last course intake.

Focus groups were originally planned as part of the study design. The process of grouping nurses who were working shifts across four time zones proved unrealistic; consequently, nurses were interviewed individually or in pairs. Interviews were conducted two to four weeks following the completion of the course; they were tape recorded and typically lasted about 30 minutes. An interview guide was used in order to enhance a systematic approach, and all interviews were conducted by one investigator. The interviewer began by asking a general opening question: "Can you tell me about your experience with the online CTAS course?" Nurses were then asked about their experience with the tutorial and the online discussion forum, and what, if any, impact the course had on their practice. Nurses in the experimental group were also asked to describe their workplace project (see Appendix A). Throughout the interviews, the investigator periodically verified her understanding of nurses' statements with them, in order to enhance accuracy of interpretation.

The interview audio tapes were transcribed, and a content analysis was used to identify themes and classify responses to the interview questions. The data were coded independently by two investigators and compared. Inter-rater reliability for the coding was high, and in those instances where reviewers differed, the transcripts were re-examined and re-coded. Following coding, responses were grouped under the key concepts outlined in Lawton's model for clarity of presentation.

3.6.6 Online Discussion

The total number of participants and messages in the online discussion forum for each CTAS class was tracked automatically by the Prometheus course system software. An in-depth analysis of the level of discourse was beyond the scope of this project. Descriptive statistics and a t-test were used to compare the volume of messages in the control and experimental groups.

3.7 Sample

The target population for this project was Emergency Room Registered Nurses (RNs). Every nurse who enrolled in the online CTAS course was invited to join the study. Two hundred and three nurses enrolled, and a total of 180 nurses completed the course. Four nurses from outside Canada audited or participated; they were not included in the study. One hundred and thirty-five nurses (76.7%) returned the LDS and 124 (70.4%) returned the OLSI. Sixty-six nurses were assigned to the control group and 71 were assigned to the experimental group.

4. Findings

Findings from the course attrition review, chart audit, two surveys, interviews and online discussion postings are reported.

4.1 Course Attrition

Ten nurses (4.9%) who enrolled did not start the course. The main reason reported was that their managers were short of staff and had requested that the nurses enroll at another time. Thirteen nurses (6.4%) withdrew from the course. The nine nurses who provided rationale for withdrawing reported a variety of reasons. These included: no computer access at work (n = 2); lost Internet access (n =1); technical problems / did not like online learning (n = 3); and personal reasons (n = 3).

4.2 Learner Demographic Survey

The majority of respondents in the experimental and control groups were concentrated in age groups of 35-39; 40-44; 45-49; and 50-54. No statistically significant difference was found between the two groups on age group (p = 0.87). As expected, the majority of subjects in both groups were women (n = 68 and n = 64 for experimental and control groups, respectively) (see Table 1).

Table 1 - Demographic Profile of Participants

Variable		Experimental n	Control n	p Value
Gender:	Female	68	64	1.0
	Male	3	2
Work:	Full Time	47	37	0.15
	Part Time	21	28
Urban / Rural		29 / 38	24 / 34	0.85
Highest Level of Education:	RN	35	31	0.64
	Post-diploma certificate	17	13
	Baccalaureate	19	21
	Master's degree	0	1
# Years as ER Nurse:	Less than one	3	5	0.10
	1-5	20	20
	6-10	16	11
	11-15	13	23
	16-20	7	3
	21-25	6	3
	26-30	6	1

The majority of rural area respondents in both groups worked full time. No statistically significant difference was found between the experimental and control groups on highest level of education (p = 0.64), number of years in the ER (p = 0.10) or other key variables.

4.3 Computer Access and Skills

Fifty-five percent of respondents (n = 67) took the course from both home and work. Seven percent (n = 9) took the course only from work, and the remainder (38%, n = 47) only from home. The majority of RNs who accessed the course from work used a computer situated on the nursing unit, either while on break or before and after work hours. Most nurses (71.5%, n = 98) were enrolled in their first computer-related course. Forty-one percent (n = 56) ranked themselves as beginner or novice in their overall computer skills. Thirty-eight percent (n = 52) identified themselves as novice / beginner Internet users.

The experimental and control groups were combined and a comparison was made on computer skills at the beginning of the course (pre-test) and at the end of the course (post-test). The analysis showed that at the end of the course there had been improvement in basic computer skills (p < 0.001), keyboard skills (p = 0.02), Internet skills (p < 0.001), e-mail skills (p < 0.001), and overall computer abilities (p < 0.001). The experimental and control groups were then compared separately on the pre-test and post-test measures of computer skills (see Table 2).

Table 2 - Experiment and Control Groups' Pre-Test & Post-Test Computer Skills

Variable	Experimental Group		Control Group
	n	Pre-Test / Post-Test Difference? Yes / No and p Value	n	Pre-Test / Post-Test Difference? Yes / No and p Value
Word processing skills	59	Yes < 0.001	57	Yes < 0.001
Typing skills	59	Yes 0.002	58	Yes 0.02
Internet skills	59	Yes < 0.001	58	Yes < 0.001
E-mail skills	59	Yes < 0.001	58	Yes < 0.001
Overall rating computer ability	59	Yes < 0.001	58	Yes < 0.001

As may be seen from Table 2, the computer skills of nurses in both the experimental and the control groups improved significantly between the beginning and the end of the course.

4.4 Online Learner Support Instrument

The Online Learner Support Instrument (OLSI) measured nurses' experiences with online learning in five sub-domains. Respondents used a Likert-type scale ranging from 1 to 5 to rank their responses to survey items (1 = strongly disagree; and 5 = strongly agree). The more positive their experience, the higher their score. The maximum score on the OLSI is 165. Environment subscale items are not factored into the total score, as there is bridging in this part of the survey for home or work users. The overall mean score on the OLSI, for both groups, was 123.8 (SD = 17.4) or 75/100. Scores ranged from 71 to 165 (44 to 100/100). The mean for the total OLSI score for the experimental group was 124.41 (SD = 17.3) or 75.4/100, and the mean for the control group was 123.37 (SD = 17.8) or 74.7/100. There were no statistically significant differences in total OLSI scores (p = 0.74) or subscale scores between the experimental and control groups (see Table 3 for p values).

Table 3 - Total OLSI and Subscale Scores Normalized Out of 100

OLSI Scale	ExperimentalMean (sd)	ControlMean (sd)	p Value
Interaction with teacher and peers	73.7 (13.1)	72 (13.1)	0.48
Course design	73.2 (12.2)	72.4 (14.4)	0.74
Technical support	78.8 (13.3)	78.8(12.8)	0.99
Overall impressions	76.3 (12.4)	76.2 (13.6)	0.95
Total OLSI score	75.4 (10.4)	74.7(10.7)	0.74

4.4.1 OLSI Subscale Scores

Interactions with Teacher and Peers Subscale

Almost 20% of participants (n = 23) wanted more feedback from the teacher. Seventy percent (n = 86) reported that they felt part of a learning group. Sixty-three percent (n = 77) said that their online discussions with peers had been helpful to their learning.

Environment Subscale

The majority of work users (60%, n = 44) reported that they enjoyed taking the online course at work, 11% (n = 8) did not, and the remainder (29%, n = 21) were neutral. Seventy-four percent (n = 55) reported that they had convenient access to a computer at work. However, fifty-three percent (n = 39) reported that there was not enough time to learn CTAS at work. Twenty-seven percent (n = 20) said that there were not enough computers, and 41% (n= 30) noted that the computer was located in a place where it was difficult to concentrate. Ninety-two percent of work users (n = 68) thought that group enrollments were a good idea. Home users were more positive; ninety-five percent (n = 105) reported that they found it convenient to take a course from home.

Course Design Subscale

Seventy-six percent of participants (n = 94) reported that the use of patient simulations was helpful to their learning, and 78% (n = 97) found that the course content was presented in a way that was easily understood. Twenty-nine percent (n = 36) reported that they had difficulties recognizing when new items were added to the online discussions. Only 59% (n = 73) noted that it was easy to follow a topic in the online discussion forum.

Overall Impressions Subscale

Seventy-eight percent of participants (n = 97) noted that their knowledge of triage had improved. Seventy percent (n = 84) reported that they could think of two ways in which the course had improved their practice. Seventy-five percent (n = 92) felt that they were able to triage patients more accurately following the course.

4.5 Chart Audits

CTAS is a five-stage triage scale. When a patient is admitted to the ER, he / she is assigned to one of the following categories: resuscitation, emergent, urgent, semi-urgent, and non-urgent. The assigned CTAS level influences how quickly the patient should be seen by a physician and how frequently the nurse re-assesses the patient. Six sites from three provinces participated in the chart audit. Three sites were identified as urban and three were rural. In total, 367 charts were retrieved and evaluated; 183 from the experimental group and 184 from the control group (see Table 4). Of this number, 69.8% (n = 256) were identified as correctly triaged by the ER nurse expert. The accuracy rate for the experimental group was 72.1 % (n = 132) and 67.3 % (n = 124) for the control group. A Fisher's exact test showed that there was no significant difference in the accuracy rate between the experimental and the control group (p = 0.36). Overall agreement between the nurses and the expert reviewer within one level was 99.7%.

Table 4 - Chart Audit Accuracy Rates for Experimental and Control Groups

Group	Correct	Incorrect	Total	% Accurate
Experimental	132	51	183	72.1 %
Control	124	60	184	67.3 %

Of those charts in which the nurses' codes differed from the expert reviewer's code, 46.8% were under-triaged; i.e., the patient should have been assigned a higher, more urgent code. Fifty-three percent of the charts were over-triaged; i.e., the patient should have been assigned a less urgent code. Eighty-three percent of those patients who were under-triaged were triaged by nurses in the control group. Seventy-one percent of those who were over-triaged were triaged by nurses in the experimental group. There was a highly significant difference between nurses in the experimental group and those in the control group regarding over- and under-triage level assignment (see Table 5). The reviewer noted that all patients, including those who were under-triaged, were seen within the recommended CTAS timelines.

Table 5 - Analysis of Chart Errors According to Experimental and Control Groups

Group	Under-triaged	Over-triaged	p Value
Experimental	9 (17.3%)	42 (71.1%)	p < 0.001
Control	43 (82.6%)	17 (28.8%)

4.6 Interviews

Twenty-three nurses were interviewed. Eight nurses (35%) worked in rural settings, and 14 (61%) worked in urban hospitals; one nurse worked at outpost settings and onboard a ship. Fifteen nurses (65%) were in the experimental group and eight nurses (35%) were in the control group. The majority (75%) were computer course novices. Following the initial coding, findings were clustered, and variables from Lawton's (1997) distance education model were used as the organizing framework for reporting. Recall that Lawton identified student characteristics, student-teacher and student-student interactions, course design, and environment as key variables influencing the distance learner's experience.

4.7 Nurses' Experiences

4.7.1 General Impact of Online Course

The majority of nurses (78%, n = 18) from both the experimental and the control groups reported a positive learning experience, and one which also improved their practice. The following statement summarizes the impact of the online course for this nurse.

"I thought you might like to know how valuable the CTAS has been this past weekend at our 10 bed rural hospital. At 05:30 this morning, there were two of us on shift and alone as staff. We had a patient die on the ward at the same time we received five teen victims from a motor vehicle collision. All were wet and cold and had been drinking. They survived, thanks to seat belt use, a big rollover. The most seriously injured had several fractures and abdominal bleeding and needed evacuation to a tertiary care hospital for surgery. Our ceiling was low and it was touch and go getting him out by chopper. It was a challenging time for the teens, their families, and the family of the ward patient who died.... and for the two of us RNs. Being comfortable with a good triage system was a big asset for me."

Nurses enjoyed the convenience and flexibility of online learning. The online environment made access to the CTAS course possible for nurses who lived in areas where no formal CTAS training was available. The positive perspective is illustrated by one nurse who remarked:

"I really enjoyed it. It opened my eyes to a lot of things that we should have been doing differently. You see, we've been under-triaging. When CTAS came in a couple of years ago we wanted to get going with it. We are self- taught. There was no course for us to be taking. This was my first online course. I really enjoyed doing my own thing, calling my own hours to work on it, especially doing shift work."

Another nurse noted:

"In terms of working shift work, it was nice to do a course like this where when I had the time I would go and do some pieces. I didn't have to give up the three hour Tuesday evening off time."

The majority of nurses (75%) who participated in the interviews were novice computer and Internet users with no previous experience with web-based learning. Several nurses reported some frustration and difficulty in mastering the technology and learning to navigate around the online course platform. Experienced online learners learned to navigate the various components of the course quickly, and all participants reported feeling confident by the third week of the course.

"I quite enjoyed it. I've never done an online course before. There was a huge learning curve for me because I'm not computer literate. The huge learning curve was the program platform itself. Once you went though the tutorial you were ok; the practice sessions were a really good idea."

Participants described numerous benefits of online learning. These benefits included: improved triage assessment skills and accuracy; connection to peers; improved computer skills; and professional networking. Forty-three percent (n = 10) spontaneously commented that they had made gains in their computer skills. "As far as the computer, I am 100% improved from when I started." Nurses also described the course as having a positive impact on themselves as learners. All nurses reported that they were interested in further online learning, including baccalaureate and Master's level education. One nurse remarked:

"It has spurred an interest in doing more online learning. It makes me think I could do a Master's course via the Internet. ... being able to do a short course like this makes me think I can do more."

"I love learning and education. In Labrador there aren't a lot of resources. This is a benefit for us in the North that we are able to get computer accessed information."

Not all nurses were satisfied with the course. Four nurses (17%), all from the experimental group, reported that they were ambivalent about their course experience. All four were very familiar with CTAS. They reported being disappointed with what they considered the introductory nature of the course content, and desired more complex case analyses. One nurse reported that the course was unhelpful. This nurse was already very familiar with CTAS, and was disappointed that she did not learn anything new.

"For me personally, the course was not that helpful. Don't get me wrong, the course was well done. The course is fine for junior nurses but not for experienced staff."

4.7.2 Student-Teacher Interactions

While nurses were not asked specifically to comment on their teacher, several commented that the teachers had been prompt in replying to their questions, and were supportive and helpful in fostering the discussions. Many participants, however, reported that they would have liked more feedback from the teachers regarding the learning activities. A major component of learning in the CTAS course consists of case situations, in which nurses are required to review a patient scenario and assign a triage code. While nurses valued hearing whether the code they assigned was correct or incorrect, they wanted more discussion regarding the rationale for assigning a particular triage code. The same situation applied to their exam, which also consisted of patient scenarios. Nurses received a mark for their work on the final exam, but would have preferred the opportunity for an exam review, in order to delve into the rationale for correct / incorrect responses on their exam. Many nurses also wanted the teacher to have a more visible presence in the online forum discussions, as a moderator or chairperson. They wanted the teacher to play a stronger role as 'the expert'. While the nurses valued their peers' input, they expected the teacher, as the expert, to verify their comments regarding the 'correct' answer in the case study scenarios. One nurse remarked:

"It's not that I didn't enjoy what the others had to say, but the instructor, who is the expert, needs to validate what the others had to say. How valid are my classmates' remarks, unless they are backed up by an instructor?"

Two first-time online learner participants noted that they would have preferred some private e-mail exchanges with their teacher, regarding their progress in the course.

4.7.3 Student-Student Interactions

Participants' comments regarding their discussions with peers were generally very positive. Nurses reported that hearing about other nurses' experiences facilitated their learning, helped them to reflect on and evaluate their own work practices, and made them feel less isolated. Nurses enjoyed the feeling of being connected to nurses across Canada, and were encouraged that the difficulties experienced at their own institution were not confined to their region. One nurse from a small community in Newfoundland commented:

"It was helpful and nice to read other people's take on things. I don't know if it was the way our professor set it up or her encouragement, but I thought there were a lot of comparisons such as, 'This happened to us and this is how it applies to what we are doing this week.' There were a lot of incidents that we were able to share with people in Ontario and other places. When we threw these little ideas in, it was interesting to find that we were not the only ones who are isolated, we are living in the middle of nowhere, but some of these people were telling stories of living in big centers and we think, that doesn't happen in the real world. It really put your mind working and made it a whole lot more applicable to hear others take on things."

Some nurses suggested that the Student Introduction part of the course be expanded, so that nurses could better picture their fellow classmates. A small number of students also recommended making a more formal closure to the course, so that classmates would not simply drift away after the final exam.

4.7.4 Technology

Virtually every nurse who participated in the interviews reported that the e-learning platform (Prometheus) was time-consuming to learn and cumbersome in managing the online discussions. The major difficulty reported was identifying new messages or searching for previous messages. Nurses would have liked a feature that would have enabled them to hide previously read messages. Problems were compounded if nurses accessed the course from both home and work. For example:

"I found you couldn't automatically go from one posted discussion to the one below or search for old messages and move forward. I found out how to get the newest messages at the top, but you still had to understand if 0, 1 or 2 people had replied. You had to figure out when the last day was when you used a different computer. So if I accessed it at work, and then came home and accessed it, I had to figure out which ones I actually had seen and which ones I hadn't."

In addition, one group lost access to the server for four days, and the frustration arising from this glitch with the technology was conveyed in the interviews. While nurses felt that the college alerted them and responded quickly to the problem, this event contributed to their anxiety about work completion, given co-existing home and work demands.

While there were some exceptions, the majority of nurses who accessed the 24-hour technical help line reported that they had a prompt, helpful response to their call. Some nurses did not access the technical support because they were reluctant to reveal their lack of technical skills.

4.7.5 Course Design

Overall Layout

Once they became familiar with the platform, nurses found it easy to navigate the course. While many nurses found the course layout clear and easy to follow, others had difficulty.

"The Week One syllabus has a lot of things listed like a module, objectives, tests, and online questions and a self-quiz. You go through all this stuff and you don't know if you've done everything. You tended to go back over things to make sure you covered everything."

Nurses appreciated the course schedule of a new module being posted at the beginning of every week. They felt that this schedule kept the group moving through the course together, which enhanced group discussions. Several nurses noted that group progression through the course was essential to their learning. They did not want the entire course posted at once, with a group discussion at the end of six weeks. One suggestion was made to post two or three modules in advance, so that participants could work on them if they had extra time.

Tutorial

The first module in the course was a tutorial introducing the learner to the Prometheus platform and identifying the computer skills necessary to participate in the course. It will be recalled that one of the variables being tested in this study was the impact of the online course tutorial. Nurses in the experimental group were advised that the online Internet tutorial was mandatory. They were required to complete the tutorial learning activities, such as sending the teacher an e-mail or posting a message to the discussion forum, before proceeding to the course content. Nurses in the control group were strongly advised, but not required, to complete the tutorial and the accompanying learning activities.

All nurses in the experimental group and all except one nurse in the control group reported completing the tutorial. Computer novices found it helpful to demonstrate their skills to the teacher before the course started. The nurses reported that the tutorial was invaluable in orienting them to the course platform and to the skills needed for the course. Regardless of one's online course experience, the online tutorial was perceived as a helpful and necessary tool for orientation to an unfamiliar platform. One nurse remarked:

"I did do it [the tutorial] and I did find it helpful. It was very well set up and clear. I actually printed it off and referred back to it a few times, until I got the whole system figured out. I would have been lost without it."

Nurses made several recommendations regarding the tutorial. They suggested that the tutorial should be leveled. Level One should be a strictly 'needs to know' module, and should include only those tasks that would be needed during the course; for example, an introduction to the platform, cutting and pasting from MS Word into the discussion forum, and posting messages to the forum. Skills other than the basics could be included in Level Two, an optional component for those who wished to master more advanced skills. Nurses also recommended leaving the tutorial posted throughout the entire course as a reference tool. The tutorial should be made available as soon as students had registered for the course. One nurse suggested that complete Internet novices be provided a paper copy of the tutorial.

Online Discussion Forum

The majority of nurses from both groups noted that dialogue in the online discussion forum contributed to their learning and to their appreciation of practice in other provinces. One nurse in the control group commented:

"I think that's a huge part of the course, because technically you could have done the whole course in a week, not interacted with anyone, posted your little discussions, responded to others, ...but I probably learned more from the discussion part of it than from the content."

Another nurse noted:

"I thought the best part of the course was the discussions."

Some nurses reported that their online discussions had not been very active; nurses had simply completed the assigned learning activities. Some nurses reported that while they conscientiously read every message, they held back their contributions to the discussion, because they genuinely felt that they had nothing further to add. Others noted that they had not participated fully because of the many work, study, and family demands on their time. One nurse felt that the discussion groups should have been divided, in order to make the volume of discussion more manageable.

Some nurses felt that the online discussions wandered, and that they would have been more useful if they had been directed and summarized more closely by the teacher. Nurses were expected to complete their discussion contributions by Sunday evening, before the start of the new module on Monday. This often lead to a rush of Sunday night messages, which nurses who had completed their coursework earlier in the week found annoying. Two nurses mentioned that they received a negative response from one of their peers to one of their early online contributions. Following this incident, they decreased their discussion forum participation.

Marks for the Online Discussion

All nurses taking the CTAS course were required to post their work regarding the case study analysis to the online discussion forum, and part of their grade was awarded for completing these activities. A further twenty-five percent of the final grade for nurses in the experimental group was based on their participation in the online discussion forum.

To evaluate this process, all nurses were asked whether marks should be awarded specifically for the online discussion. Forty-eight percent (n = 10) replied yes to this question. Nurses felt that the online discussion was important, and marks acted as an incentive to read and contribute. Eight nurses (38%) from both the experimental and the control groups reported that marks should not be awarded for the discussion. Nurses felt that awarding marks forced participants to comment for the sake of obtaining a mark, and that this generated some irrelevant remarks. One nurse noted:

"My time is precious; I don't want to read something unless it's really pertinent."

Three nurses (14%) suggested that whether or not one contributes may be determined by personality or culture, and whether one is in a classroom or online; awarding marks for the discussion exerts an added, unwanted pressure on participants. Others felt that they would have participated, regardless of the mark.

"If I was interested in what the discussion was about, then I would respond. If I wasn't, then I would not."

Another nurse noted:

"You should just participate because the activities or questions are good. Just set the standard. If you do [award marks] ...well, it's not really adult learning oriented, is it?"

Three nurses (14%) commented that if marks were to be awarded, the percentage awarded should be small. They felt that the teacher should set the criteria for achieving the mark, and that this should mean a substantial contribution. They felt that the instructor could prompt those who were not contributing. If marks were to be awarded, the nurses wanted the marking criteria and the teacher's expectations to be made explicit.

Content

Content in the form of text was minimized in this course. The main learning activity was case situations, presented for analysis and triage coding practice. Nurses reported that they enjoyed this design. Nurses who were new to the ER or to CTAS reported that the content was sufficient and satisfactory. Experienced CTAS users, however, wanted a more in-depth content, more links to readings, and more complex case situations for analysis and discussion.

Workplace Project

Nurses in the experimental group were required to develop and implement a project regarding CTAS in their workplace. Nurses enjoyed carrying out the project, and reported that it facilitated their learning, improved their practice, and had a broad impact on staff in their ER departments and on triage policies. One nurse noted:

"We did our project presentation at our staff meeting, and as a result, we will be implementing the suggested CTAS documentation."

The nurses reported that the project improved their understanding of triage and increased their accuracy and confidence.

"I think it made me use the material better, it helped to take what you learned and put it to use, instead of shutting the books and putting them away."

Some projects were designed to improve communication with patients and families. Two nurses developed a triage poster showing physician and nurse assessment times.

"We are getting it made into a wall-mounted sign, to put in the waiting room and at the triage desk. And one of our First Nations artists has agreed to design a painting in relation to care given in emergency; he will illustrate the distressed patient and the comfort that they receive from the staff in the ER. .... As a manager of the department I think signage is very important. Too often we have high expectations from our patients, we do not keep them informed to what our expectations are. This is one of the reasons we went for a sign."

Nurses described a number of interesting, creative projects that they had developed. These included:

• Developing patient scenarios, based on charts from their own departments, for staff teaching purposes. Nurses presented the scenarios and asked nurses and physicians in their department to assign a triage code and provide rationale.
• Updating existing CTAS training packages in the department.
• Developing CTAS pocket cards with the most common problems seen in their ER.
• Developing a CTAS Powerpoint presentation for staff orientation or for refresher purposes.
• Conducting a staff meeting, in order to do a chart review and discuss triage codes.
• Conducting two five-hour workshops to train staff in CTAS.
• Working with a graphic designer to create a poster of CTAS guidelines, for MD assessment and RN re-assessment times.

One nurse shared with senior management what she had learned:

"I presented to the CEO and the Chief of Staff. It was good! I was pressured, I'm not the type to get up and make a presentation, but I did it. I talked about what I had learned, and made suggestions about what we could do at our site to change things. We had a long discussion about what we could do to make changes. Anyway, they agreed to further education for us in triage. That presentation definitely had a good impact on our site. They said there would be more money for education for others who want to do the course."

A suggestion was made to develop more options for the workplace assignment, in order to prevent project repetition at sites where several nurses were enrolled. Another suggestion was to develop a forum for showcasing nurses' projects to their peers, so that nurses could benefit from each others' work.

4.7.6 Environment

No specific comments were made about the home as the learning environment; however, comments were made about the workplace as the learning environment. Five nurses said that they had computer access and the time to work on the course while working their shift. Four nurses said that there was absolutely no time at work; the unpredictable nature of ER nursing and the hectic environment in the ER made it difficult to settle down to work on the course or to concentrate.

4.7.7 Impact on Practice

Seventeen of 20 nurses (85%) reported that the course had a positive impact on their practice. An increased understanding of the CTAS guidelines meant more accurate patient triage, which in turn meant better patient care.

Many nurses spoke of 'triaging up' more frequently after taking the course. This means that they assigned a triage code that would have the patient seen by the physician more quickly. One nurse noted:

"We then had great discussions around near-death asthma in our department, because it is now considered a level one (most urgent) triage code and everybody would have put that person down as a two before. I think there will be a lot of people who will be triaging that type of patient up."

Another nurse noted:

"It has definitely made me more aware and how to more accurately triage. I'm looking at things differently now, because you are looking at the potential diagnosis down the road of not just the obvious, it makes you think and do a plan of care."

"It's made me change my whole approach. I think I'm more efficient and thorough with my assessments."

Nurses reported a renewed attention to those patients who presented what had become, from the staff's perspective, a routine problem and assigning a higher level of urgency to those patients. For example:

"I know we have always been concerned about the headaches and abdominal pains that come in, but sometimes they are taken in a more lackadaisical manner and I think now I would look at that more objectively."

Nurses also reported increased confidence as congruency of triage reporting increased in their department.

"We were all varied in what we thought were emergencies. What I'll take away is that my co-workers beside me will all be triaging the same as me."

Nurses felt that their patients were benefiting from the implementation of a national triage standard. A nurse who worked in a rural hospital on the Prairies remarked:

"It is encouraging and professionally exciting to know we are using the same methods and that patients are hopefully being triaged the same as if they were in Toronto or Vancouver."

Several nurses who were serving a mainly adult population at their site commented that they found the pediatric component of the course helpful in developing their skills in practice. Nurses felt that they were more organized, that they had improved patient flow through the ER department and, as a result, improved patient care. They appreciated having a standard of practice to support their decisions. One nurse noted:

"Last night someone had something in their eye, a couple of babies came in, a respiratory problem came in. They all arrived at the same time. Having a rationale now for prioritizing the patient makes things easier."

Another nurse noted:

"It has improved the flow of things in the department. You could see it change over the six weeks of the course."

Nurses felt that they were being more attentive to families and patients, particularly to level 4 and 5 (less urgent) patients who typically had a lengthy wait in the waiting room. Several nurses commented that they were re-assessing all patients more frequently, particularly those who were in the waiting room. They also reported that they were more diplomatic and that they communicated better with patients and families, with regard to the patient's status and wait time.

"We are tending to the less urgent patients better, less aggressively. We're more diplomatic with them, more friendly and helpful. I'd say it had a good impact on our communication with them. The patients are less angry now."

Two nurses in the experimental group were ambivalent about the impact on their practice; however, they did cite examples of ways in which they used what they had learned. Both nurses were very experienced ER nurses and both were already familiar with CTAS. They reported that insufficient discussion regarding the rationale for triage codes assigned to case scenario and lack of in-depth content had reduced the impact of the course on their practice.

4.7.8 Barriers to Implementing CTAS

One surprising finding was the number of sites (17%, n = 4) where nurses reported that no formal triage system was in place or that triage was being done by the ward clerk.

"Before this [CTAS course], it was first- come, first-served."

Another nurse remarked:

"We don't have a formalized triage system here. Patients here are registered by a clerk, who often does not have any medical experience. Usually, the first person they see is the clerk who registers them and sends them to the waiting room. If the clerk thinks that the patient is sick enough, then she'll send them to see the nurse. We had kids coming in with high fever that were sent to the waiting room. Unless we sort through the files and go to the waiting room to enquire, we don't know that this child is there and we have had children seizure in our waiting room. ... I think it [the course] is going to change a lot. Now we will be looking and be more on our toes."

"In this course, you have to have the triage nurse out in the waiting room and so she [ER manager] is sort of coming around that these are the Canadian standards. We were the only one in the lower mainland that didn't see a nurse first, so I think this is great."

Nurses from these sites reported that what they had learned from the CTAS course had given them the ammunition to argue for more staff, for a physical change to the ER layout or for a change in policy, so that patients would see a nurse, rather than the unit clerk, first. One nurse remarked:

"So the triage nurse is the ward clerk plus the charge nurse who is also responsible for helping out on the floor, answering the phone and faxes, so there's no way you can concentrate on your interviews that are supposed to be three minutes but are more like 45 seconds, with phone calls in between. So it showed me that we really have to advocate for change."

Several challenges related to CTAS implementation were noted. Some nurses reported that because their ER department was small, there was insufficient staff available to triage according to CTAS guidelines. Others noted that while the nurses had been trained in CTAS, other professionals with whom they were working, such as physicians and paramedics, were not, which led to communication problems. Another barrier was the physical layout of the ER department. Some nurses reported that they had to assess patients while in view of others and that there was insufficient privacy for a thorough patient assessment.

The need for administrative support was also identified. Some nurses wondered whether their administration would follow through with the environmental and staffing requirements necessary to support CTAS, now that the staff was trained.

4.8 Volume of Online Discussion

Another source of data to be collected was the online discussion forum. The number of messages generated by nurses in each section of the CTAS course was automatically tracked by the Prometheus system software. The mean and standard deviation were calculated for the experimental and control groups, and a t-test was conducted to compare the groups. The mean for the control group was 18.6 (SD 6.6) and the mean for the experimental group was 16.5 (SD 4). The results of the t-test showed that there was no significant difference between the two groups in the number of messages posted to the forum (t = 0.622, degrees of freedom = 9; p = 0.549).

5. Discussion

5.1 Overall

A major finding from the study was that the online CTAS course positively affects nurses' triage skills. The triage accuracy rate for all nurses, as is evident from the chart audit, was 70%. This figure compares favorably when measured against the findings of previous reports of triage accuracy and supports the further development of online courses for health care professionals (Travers, et al., 2002). It is important to note, however, that the expert reviewer acknowledged that to triage a patient who is brought in by ambulance to a busy ER, and to assign a triage score based on documentation from the ER record are two different experiences. Another very positive finding was that within one triage level, the overall agreement between nurses and the expert reviewer was 99.7%. This finding implies that nurses can learn and reliably use the CTAS (Murray, 2003). The 70% agreement rate between the expert reviewer and the nurses provides further evidence for the work done by Beveridge et al. (1999) regarding the reliability of the CTAS scale.

While a 70% accuracy rate compares favorably to earlier studies, it is important to carefully examine the 30% of incorrect triage codes. A significant difference was found between the experimental and control groups with respect to the type of error made. Fifty-three percent of the incorrectly triaged patients were over-triaged; the majority of these were assigned a CTAS level 3 or 4. The majority of over-triaging (71%) was done by nurses in the experimental group. Over-triaging may have some cost implications and / or cause increased wait times for other patients, but these are issues of resources. Moreover, over-triaging is a practice that errs on the side of patient safety. Triage personnel are typically encouraged to 'triage up', even if the patient does not clearly present the signs and symptoms that match with the higher triage level (Beveridge et al., 1996).

Of greater concern are the 47% incorrectly assigned codes that resulted in patients being under-triaged. All patients but one were under-triaged by one level, and most of these were assigned a CTAS level 3 or 4. It is interesting to note that eighty-three percent of those patients who were under-triaged were triaged by nurses in the control group. The implications of under-triaging -- more serious than those of over-triaging -- are increased patient wait times and poor outcomes if necessary care is delayed. A positive finding was that all patients, including those who were under-triaged, were seen within or before the time designated by their triage level, according to CTAS guidelines. This differs from the present environment of emergency care, where wait times often exceed triage level guidelines (Louise Leblanc, BSN, National Triage Working Group, personal communication, January 12, 2003).

Although the hypothesis that a mandatory tutorial, marks for online discussion, and a workplace project would improve nurses' satisfaction and enhance triage accuracy was not supported, a great deal was learned about online learning and teaching practices. The overall satisfaction rate with the web-based course on the OLSI was 75%, which suggests that it may be considered a very satisfactory experience. This finding is supported in earlier studies with online learners (Andrusyszyn et al., 1999; Curran, 1998; Daugherty & Funke, 1998).

The finding of no significant difference between the two groups in nurses' satisfaction with the course (based on the surveys and interviews) and in the overall triage accuracy rate was surprising. It may be explained by the fact that the greater part of the online learning experience for both groups was the same with regard to content, patient case situations, and volume of online discussion. While the online tutorial was mandatory only for nurses in the experimental group, virtually every nurse did the tutorial. This means that the two groups' experiences ultimately differed only in respect to the workplace project. The finding of no significant difference may be cautiously interpreted as encouraging news for online course designers; if the core components of the course are sound from an instructional design perspective, learners will be largely satisfied. The finding that the nurses who completed the workplace project made significantly fewer errors of clinical importance (under-triaging) is important and warrants further study.

Table 6 - Levels of Satisfaction and OLSI Normalized Scores Out of 100

Levels of Satisfaction	Normalized OLSI Score out of 100	Control	Experimental
Highly satisfactory	85-100	10	10
Very satisfactory	75-84	26	28
Satisfactory	65-74	13	15
Unsatisfactory	Less than 65	10	10

The range of scores on the OLSI (post-course survey) is informative. The range (minimum score was 43/100 and maximum score was 100/100) means that some nurses were very satisfied, while others had an unsatisfactory learning experience (see Table 6). Information gathered from the interviews assists in interpreting the lower scores, which were typically generated by nurses who were already very experienced with CTAS. These nurses were seeking more in-depth content and resources, and greater discussion of more complex triage situations; they were over-qualified for this level of CTAS course. A given course brings together learners with varied backgrounds, in terms of prior knowledge and work experience. Online courses -- unlike those in the traditional classroom setting -- have tremendous potential to enable learners to customize their courses by adding or deleting modules and web links.

The OLSI subscales measured nurses' experiences with different aspects of web-based learning: interactions with teacher and peers; technology; course design and resources; the environment; and nurses' overall impressions of the web course. The classification scheme outlined in Table 6 suggests that nurses' experiences in the home environment (subscale mean of 85) were highly satisfactory. Experiences with technology (mean of 78.6) and nurses' overall impressions of the course (mean of 76.3) were very satisfactory. Experiences with teacher and peers (mean of 72.5) and course design (mean of 72.6) were satisfactory.

Interviews conducted with nurses assisted in the interpretation of the lower scores on these last two subscales. Nurses wanted more evidence of the expert perspective from the teacher and more interactive, focused discussions with peers. Thirty-four percent of nurses reported that they missed talking with their peers. The need for more feedback and interaction in online courses has been reported in numerous studies (Andrusyszyn, Iwasiw & Goldenberg, 1999; Bullen, 1998; Vrasidas & McIssac, 1999). Currently, computer conferencing only conveys written messages; non-verbal communication and verbal inflection are not transmitted. The absence of non-verbal cues from the teacher and peers may leave participants feeling as if they have been communicating with a machine, rather than with other human beings. Roschelle and Pea (1999) argue that regardless of the hype around web-based communication, the Internet is still not sufficiently advanced to facilitate a deep exchange of knowledge, beliefs, and values.

As identified in earlier studies, the discussion forum plays a major role in online learning (Cragg, 1994; Daugherty & Funke, 1998; MacPherson, 1997). The forum provides a valuable opportunity to connect course content to the real world of practice, to critically reflect on new content, to test new skills, and to learn from more experienced peers. The need to further discuss the rationale for assigning a particular triage level to a patient situation was identified by nurses in this study as a key part of their learning. Identifying this gap suggests that nurses recognize that access to information does not equate with knowledge construction (Salomon, 2000). The discussion forum also provides a valuable opportunity to network with colleagues across the country. Networking was perceived as morale-boosting and helped nurses put their own workplace practices in perspective.

However, not all comments about the online discussion forum were positive. Some nurses felt that they had not connected with their peers, and others thought that the discussions lacked focus or relevance. The study highlights the critical role of the teacher as discussion moderator. Nurses made it clear that while they value their peers' input, they want the teacher to moderate and to validate the discussion. The amount and quality of online student-teacher time greatly affect the student's experience (Beer, 2000; Hiltz, 1995; Powley, 1994). One surprising finding was that 38% (n = 8) of those nurses who were interviewed felt that marks should not be given for online forum participation. These nurses thought that this practice lends itself to simply adding to the volume, but not to the value, of online discussion.

The online platform also played a key role in the learning experience. Online learners want tools that make online discussion management straightforward. As busy professionals, nurses are interested in expedient ways of learning and they value systems that make message identification, retrieval, filing, and posting quick and efficient.

With respect to the course design, nurses liked the focus on patient scenarios. Seventy-six percent of nurses (n = 94) reported that the use of patient simulations helped to present subject matter effectively. They also emphasized the need to clearly post key learning activities and tests at the start of each module, so that these would not be missed.

Access from the work environment was considered satisfactory (mean of 67.6). In an earlier study conducted just two years ago, the mean for workplace access was 56 (Atack & Rankin, 2002). The improved score is encouraging and may be interpreted as a sign that workplace access to online learning is improving. While the environment may be improving, insufficient time to work on the course, inadequate numbers of computers, and the noisy, hectic ER environment presented major barriers to workplace learning.

One major theme identified in the interviews was the critical role that the tutorial plays in preparing nurses for online learning. In earlier studies, lack of computer skills to manage the technology was identified as a major barrier (Cragg, 1994; Daugherty & Funke, 1998). In this study, with 40.5% of participants self-reported as computer novice or beginner, insufficient computer skills to manage the course persist as a barrier to learning. Even experienced online learners noted that it was important to take time to complete the tutorial when adapting to a new online delivery platform. Nurses emphasized that the tutorial content and activities should closely match course requirements.

The pre- and post-test comparison of nurses' computer skills showed a statistically significant change. Both the experimental and control groups made statistically significant changes in all computer skill categories during the six-week course. There were, however, no significant differences between the two groups on computer skills. The finding that nurses gained computer skills is similar to findings in earlier studies, in which nurses who completed full-semester courses on the web made significant gains (Atack & Rankin, 2002; Panzarine, 1998). This finding is interesting, as it suggests that learners may benefit from taking even relatively short computer courses.

5.2 Impact on Practice

The majority of interviewed nurses noted that the CTAS course had positively affected their practice, and 75% noted on the post-course survey that they were now triaging more accurately. Nurses identified two factors that helped them to integrate knowledge and skills gained from the course into practice: the nature of the course, and the workplace project.

The CTAS course content is well suited to online learning in that the volume of delivered content is moderate, leaving time and resources for applying new assessment skills to patient simulations. The workplace project carried out by nurses in the experimental group had a major impact on their practice. Virtually all nurses in that group remarked that the project had improved triage practices in the workplace. The project facilitated the transfer of triage learning from the course to the individual and from the individual to other staff and administrators in the ER departments. For health care organizations, this knowledge transfer adds tremendous value to the course and, ultimately, to the patients themselves. Administrative support and barriers were also identified as key variables influencing nurses' ability to implement the CTAS. These supports have also been identified in numerous earlier studies (Armitage, 1990; Ferrell, 1988; Kiener & Hentschel, 1989; Leonard, 1994; Wildman, Weale, Rodney & Pritchard, 1999).

5.3 Attrition

Six percent of nurses who enrolled in the CTAS course withdrew. This figure compares very favorably with other web-based courses where attrition may be as high as 50% (Butler, 2003). The majority of nurses reported that they enjoyed the course and that it met their learning needs. Encouragement from managers and financial support are other factors which potentially kept attrition rates low.

6. Recommendations for best practices in online learning

The following recommendations are based on findings from the surveys, interviews, chart audits, and online discussion message analyses. Variables from Lawton's model (1997) have been used as an organizing framework.

6.1 Overall Recommendations

• Continue offering the CTAS course online in order to improve triage accuracy, standardization, and course access. Registered Nurses are interested in courses that focus on the social process of learning and on authentic problems faced in everyday professional practice.
• Develop more high quality online courses for health care professionals and enhance workplace support for e-learning. Nurses are motivated to learn, and they may be facing geographic and time constraints that provide a challenge to traditional learning; they are interested in expedient ways of learning to maintain competency and improve patient outcomes.

6.2 Technology

• Select a platform that has been well tested and specifically designed for web-based learning. The selected system should minimize learners' technical demands and enhance, rather than frustrate, online work.
• Select a platform that requires minimal training, makes message management easy to learn, and facilitates message posting, retrieval, and management. Ideally, the platform selected should provide an interface and navigational look that may already be familiar to students.
• Adopt a system that allows students to customize their courses and that manages and tracks students' experiences.

6.3 Learners

• Advise learners well in advance of the course of the hardware, software, and computer skills required for the course. Advise them to begin the tutorial as soon as they register.
• Alert learners to the need for self-direction in distance learning courses.
• Recommend that learners shop carefully for their online course rather than select a course simply because it is online / it is available.
• Urge learners to make use of the 24-hour available help. Advise learners that technicians are trained to take calls from novices.

6.4 Course Design

• Present vital information (assignments, exams) on the first page of each module and keep this page uncluttered, so that key information is not missed.
• Focus on learning activities that relate to the work world.
• Consider offering students the opportunity to customize their course by adding enriched activities, readings or optional modules (for students who want an extra challenge).
• Provide an online discussion forum to enhance learning and networking. Sub-divide large classes into groups of 10 to 12, in order to keep online discussion manageable.
• With mature professionals, reconsider the traditional practice of awarding marks for online discussion. Focus instead on developing interesting, practice-based learning activities that will stimulate discussion and that will require learners to share their work in the forum.
• Offer learners a choice of different participation activities to earn marks towards the final grade.
• Consider posting modules in advance, so that learners with extra time can work ahead; however, use a weekly schedule to keep the course moving along and to keep the class together.
• Request that for courses on a weekly schedule all postings to the forum be made by Friday, in order to allow course participants time to read and to respond to messages before a new module begins on Monday.
• Implement a workplace project that can be completed individually or in groups. Provide different options, in order to avoid repetition in the workplace. Provide an online opportunity and a place for learners to showcase their projects.
• Consider developing a short module that would provide an opportunity for closure and a 'where to from here' discussion.
• Provide a tutorial that orients students to the computer skills required for the course, 'netiquette', and the course platform. Make it clear that the tutorial is mandatory for anyone new to that particular platform or to online learning.
• Level the tutorial. Level I should include 'need to know' information. Level II could be listed as optional, and could include less urgent skills or 'nice to know' skills.
• Offer an interactive component with the tutorial. Provide novices with the opportunity to test their skills (i.e., send the teacher an attachment, cut and paste from MS Word into the forum).
• Leave the tutorial online throughout the course.
• Make the tutorial available as soon as a student registers (or make the tutorial open and free to anyone visiting your site).

6.5 Facilitators

• Practice pro-active tutoring. Assess learners' needs and adapt learning activities as necessary. Make judicious use of e-mail to connect with non-participating students. Assist learners in moving to a more advanced level of understanding by asking thought-provoking questions. Be visible, and take an active role that includes a weekly summary.
• Strive to create a balance between demonstrating your expertise and encouraging learners to share their skills and experiences.
• Provide the opportunity for an informal mid-term evaluation to check learners' experiences.
• Always conduct a test or an assignment review.
• At the start of the course, provide clear guidelines regarding the frequency and nature of online participation.

6.6 Workplace

• Examine the organizational strategic plan for staff development and determine whether sufficient resources and IT support for online learning are in place.
• If online learning is truly expected to take place at work, provide release time, sufficient numbers of computers, and a quiet learning environment.
• Agency education coordinators need to identify courses that mesh with the agency's strategic plans for nursing.
• On-site educators need to keep managers, administrators, and other members of the health care team informed of the course objectives, so that the administrative structures or policies necessary for the transfer of learning into practice are in place.
• Facilitate group enrollment in web courses.

6.7 Educational Institutions

• Market courses for your learners clearly, so that they can select wisely.
• Limit enrollment in an online class to a maximum of 30 students.
• For short, stand-alone courses in which participants are new to online learning, offer 24/7 technical support to online learners and facilitators.
• Support the online instructor and bolster the expert perspective in the online forum. Develop a bank of case scenarios, with accompanying rationale for correct answers which the instructor can post, as necessary, to the forum.
• Provide students the opportunity to complete formative and summative evaluations. Online courses need constant evaluation and will require sufficient funding for continual updating and revision.

7. Health Policy Implications

CTAS was reported as a useful, user-friendly, reliable triage instrument in this study. However, a number of barriers to CTAS implementation were identified. Some of the study findings have broad implications for the health of Canadians.

7.1 Barriers to Implementing CTAS

• A formal triage system is not in place at all Canadian hospitals.
• Contrary to Health Canada (1998) recommendations, in some hospitals across Canada the first point of contact for patients is not always a nurse or physician.
• Low staffing levels in rural areas (Canadian Association of Emergency Physicians, 2002).
• ER personnel within an organization are not always trained in the same triage system.
• Inadequate space in which to conduct a private, thorough triage assessment.

8. Implications for Health Care, Educational, Professional, and Government Organizations

Representatives from these organizations should collaborate to:

• Set minimal standards for online courses developed for staff training.
• Identify what evidence is required to document professional growth, and translate online course hours into Continuing Education hours.
• Secure the support needed for the development and maintenance of resource-rich online courses. Responsive evaluation implies that these courses will require continual updating and revision. This process, as well as course customization and technical support for users amount to an expensive undertaking. Organizations need to incorporate into their base operating costs the funds required to conduct this work, and will require funding support from professional, provincial, and federal organizations.

9. Study Limitations

A number of barriers restricted the number of charts available for chart audit. Few large clusters of nurses at any one site made the chart audit process difficult, as numerous individual sites had to be approached for an ethical review. The number of sites that have no formal triage system in place or that have non-professional staff conducting the initial triage was also unexpected. In addition, some rural sites did not have an ethical review process or the resources to collect data. Variables other than those identified in this study (e.g., previous experience with CTAS or onsite triage training) may have influenced the accuracy of the chart audits. Regardless of these challenges, the chart audit component of this study should be replicated, in order to gather further evidence of the impact of e-learning strategies on triage accuracy. Further studies regarding the impact of a workplace project on triage accuracy should be conducted. It must be acknowledged as well that the non-significant findings may have been due to a Type II error. While the number of participants matched the figure determined from the power analysis, investigators may have over-estimated the size of the effect.

The volume of messages in the online discussion forum was measured, but messages were not analyzed for evidence / depth of critical thinking. Further research should be conducted regarding the impact of various online learning activities on the nature and depth of the discussion in the online forum. The study should also be repeated with other groups such as physicians and paramedics, in order to determine how these professionals like to learn, and to measure the impact of CTAS on their triage skills.

10. Project Audience

Findings from this study will contribute to the body of literature on the impact of certain online teaching and learning practices, of the transfer of online learning to nursing practice, and of the impact on the emergency care received by Canadian patients. The target population for this project comprised Emergency Room Registered Nurses (RNs) working in rural and urban hospitals across Canada. A large number of nurses may potentially benefit from the study findings. There are approximately 12,000 Emergency Room nurses in Canada. The mean age of nurses in Canada is 44, the majority work shifts, have families, and live at a distance from colleges and universities (Canadian Nurses Association, 1997). Access to continuing education is a challenge for this group of learners. Traditional classroom learning models are less practical for busy professionals (Canada E-Health 2000: From Vision to Action, 2000).

Other health care professional groups, including the Canadian Paramedics Association and the Canadian Association of Emergency Physicians will potentially be interested in the study findings. In the broader sense, the findings may also be of interest to other health care providers such as physicians, occupational therapists, and physiotherapists.

11. Conclusion

The current health care environment -- with budget constraints and heavy patient loads -- demands accurate triage assessment for the safety and well-being of the public. In turn, accurate triage assessment requires that health care professionals be skilled and accurate in triage assessment. The online CTAS course provided a standardized, effective, and enjoyable educational experience that developed nurses' triage accuracy. Two activities -- an online Internet tutorial and a workplace project -- enhanced learners' preparedness for online learning and facilitated the transfer of learning to the workplace.

Technology will increasingly be used to help professionals maintain competency and to support professional practice communities. Learning is not a discrete activity. If health care organizations want to develop as viable learning communities, workplace policies to support online learning need to be in place. Findings from this study will be useful to health care professionals from a variety of disciplines, as well as to educators and administrators who are interested in developing quality online programs that improve professional practice.

12. Dissemination Plan

A research project is not complete until the findings have been disseminated. The following section includes a list of completed / planned activities.

Completed Activities

Conference Presentations

• Atack L. (November, 2002). A Partnership to Develop, Deliver and Study a Web-Based Course for Nurses. Registered Nurses Association of Ontario. Embracing the Future: Educating Tomorrow's Nurses. Toronto.
• Luke R, Kenny R, and McLean D. (November, 2002). Creating an International Classroom through Online Interactive Learning. Conference on Information Technology. League for Innovation in the Community College. Long Beach, CA.
• McLean D. (February, 2003). Delivering CTAS Online: Preliminary Findings from a Research Study. Meeting with executives from the Ontario Hospital Association and Ontario Medical Association. Toronto.
• Atack L. (January, 2002). Web-Based Learning for Emergency Room Nurses. McGill Faculty of Nursing Research Rounds. Montreal.

Papers will be presented at the following two conferences:

• Canadian Association of Distance Education, St. John's, June 2003. (Accepted).
• AMTEC 2003 E-convergence: Education, Media and Technology. Montreal, May 2003. (Accepted).

Scholarly Publications

We will be submitting papers to the following journals by September 2003:

• The Journal of Advanced Nursing
• The Journal of Distance Education
• Healthcare Management Forum

Research Alerts

By April 15th 2003, we will send out by e-mail a one-page Research Alert, with links to the final report, to the following groups:

• National Emergency Nurses Affiliation (9 provincial representatives)
• Canadian Nurses Association
• Canadian Nurse (journal)
• Canadian Medical Association
• Ontario Hospital Association
• Ontario Ministry of Health and Long Term Care
• The Change Foundation

A project web site will be created by April 30th 2003, which will house this report. Links to the final report will be made from the Centennial College web site and from The Change Foundation web site.

References

Alpay, L. & Russell, A. (2002). Information Technology Training in Primary Care: The Nurses' Voice. Computers in Nursing 20(4): 136-142.

Andrusyszyn, M., Iwasiw, C., & Goldenberg, D. (1999). Computer Conferencing in Graduate Nursing Education: Perceptions of Students and Faculty. The Journal of Continuing Education in Nursing 30(6): 272-278.

Armitage, S. (1990). Research Utilization in Practice. Nurse Education Today 10(1): 10-15.

Atack, L. (2001a). Report on the Web-Based Course Canadian Emergency Department Triage and Acuity Scale. Prepared for The Change Foundation, Ontario Hospital Association, May 7, 2001.

Atack, L. (2002). Online Training in the Workplace for Health Professionals: Developing a Framework for Efficient Workplace Training. A report for the Office of Learning Technologies, Human Resources Development Canada. Retrieved January 12, 2003

Atack, L. & Rankin, J. (2002). A Descriptive Study of Registered Nurses' Experiences with Web-Based Learning. Journal of Advanced Nursing 40(4): 457-465.

Bachman, J.A., & Panzarine, S. (1998). Enabling Student Nurses to Use the Information Superhighway. Journal of Nursing Education 37(4): 155-161.

Beveridge, B., Kelly, A., Richardson, D., Wuerz, R. (2000). The Science of Triage. SAEM, San Francisco. Retrieved March 16, 2003

Beer, V. (2000). The Web Learning Fieldbook. San Francisco: Jossey-Bass.

Begley, C.M. (1996). Using Triangulation in Nursing Research. Journal of Advanced Nursing 24(1): 122-128.

Billett, S. (1994). Situated Learning - a Workplace Experience. Australian Journal of Adult and Community Education 34(2): 112-130.

Billings, D.M., Connors, H.R., Skiba, D.J. (2001). Benchmarking Best Practices in Web-Based Nursing Courses. Advances in Nursing Science 23(3): 41-52.

Brown, A. (1997). Designing for Learning: What are the Essential Features of an Effective Online Course? Australian Journal of Educational Technology 13(2): 115-126.

Bullen, M. (1998). Participation and Critical Thinking in Online University Distance Education. Journal of Distance Education 13(2): 1-32.

Canada E-Health 2000: From Vision to Action. Office of Health and the Information Highway. Retrieved from the World Wide Web April 18, 2001

Canadian Nurses Association. (1997). The Future Supply of Nursing. Ottawa, Canada: Author.

Cartwright, J. & Menkens, R. (2002). Student Perspectives on Transitioning to New Technologies for Distance Learning. Computers in Nursing 20(4): 143-149.

Cervero, R. M., Rotett, S., & Dimmock, S. K. (1986). Analyzing the Effectiveness of Continuing Education at the Workplace. Adult Education Quarterly 36(2): 78-85.

Cone, K. & Murray, R. (2002). Characteristics, Insights, Decision-Making and Preparation. Journal of Emergency Nursing 28(5): 401-406.

Cragg, B. (1994). Nurse's Experiences of a Post-RN Course by Computer Mediated Conferencing: Friendly Users. Computers in Nursing 12(5): 221-226.

Curran, V.R. (1998). An Evaluation Report: Computer-Mediated Continuing Medical Education at a Distance. Newfoundland: Memorial University.

Daugherty, M., & Funke, B.L. (1998). University Faculty and Student Perceptions of Web-Based Instruction. Journal of Distance Education 13(1): 21-39.

Ehrmann, S.C. (2003). Asking the Right Question. Retrieved January 2, 2003

Elmore, R. (1996). Getting to Scale with Good Educational Practices. Harvard Educational Review 66(1): 1-26.

Ferrell, M.J. (1988). The Relationship of Continuing Education Offerings to Self-Reported Change in Behavior. Journal of Continuing Education 19(1): 21-24.

Harasim, L. Hiltz, S.R., Teles, L., Turoff, M. (1995). Learning Networks: A Field Guide to Teaching and Learning Online. Cambridge: MIT Press

Health Canada (1998). Information Technology Serving Health: Consultation Workshop with Emergency Room Staff in Quebec Region. Office of Health and the Information Highway. Retrieved from the web January 12, 2003

Higgison, C. (ed.). (2002). Online Tutoring E-book: The Tutor's Role. Retrieved from the web January 2, 2003 from: http://otis.scotcit.ac.uk/onlinebook/

Hiltz, S.R. (1995). Teaching in a Virtual Classroom. Paper presentation at the International conference on computer-assisted instruction. Hsinchu, Taiwan.

Hodson-Carlton, K. (1997). Redefining Continuing Education Delivery. Computers in Nursing 15(1), 17-22.

Hofstader, R.A., & Munger, P.D. (1990). Education in the Workplace: An integral Part of the Development of Professionals. In: R.M. Cervero & J.F. Azzaretto (eds.) Visions for the Future of Continuing Professional Education (pp. 79-105). University of Georgia.

Jelinek, G. (1996). Canadian Triage and Assessment Scale. Emergency Medicine 8: 229-230.

Kiener, M.E., & Hentschel, D. (1989). What Happens to Learning when the Workshop is Over? Journal of Continuing Education in Nursing 20(5): 206-211.

Lawton, S. (1997). Supportive Learning in Distance Education. Journal of Advanced Nursing 25(5): 1076-1083.

Leonard, D. J. (1994). Factors Perceived to Facilitate and Impede Learning in the Workplace. Journal of Nursing Staff Development 10(2): 81-86.

Ludlow, B.L., Foshay, J.D., Brannan, S.A., Duff, M.C., Dennison, K.E. (2002). Rural Special Education. Education Quarterly 2: 33-44.

MacPherson, K. (1997). Menopause on the Internet: Building Knowledge and Community On-Line: State of the Art. Advances in Nursing Science 1(20): 66-78.

Murray, G., CEO, The Change Foundation, personal communication, July 17, 2001.

Murray, M. (2003). The Canadian Triage and Acuity Scale: A Canadian Perspective on Emergency Department Triage. Emergency Medicine 15(1) (in press).

Nolan, M., Owens, R.G., & Nolan, J. (1995). Continuing Professional Education: Identifying the Characteristics of an Effective System. Journal of Advanced Nursing 21(3): 551-560.

Novaczek, I. & Gabriel, M. (2002). Learning from a Pilot Program to Put a College IT Curriculum Online. Educational Technology and Society 5(1).

Phipps, R., & Merisotis, J. (1999). What's the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education. Washington, D.C: The Institute for Higher Education Policy.

Powley, R. (1994). Development of Online Learning Systems. Victoria, B.C.: Innovative Training Solutions.

Read, J. & Gregory, P. (2002). Using Groupwise to Stimulate Academic Discussion with Computing Students. Retrieved October, 2002.

Roschelle, J. & Pea, R. (1999). Trajectories from Today's WWW to a Powerful Educational Structure. Educational Researcher 28 (5): 22-25.

Schon, D. (1987). Educating the Reflective Practitioner. San Francisco: Jossey-Bass Publishers.

Salomon, G. (2000). It's Not the Tool, but the Educational Rationale That Counts. Keynote address at: Ed-Media 2000. Retrieved from the web January 15, 2003

Shih, F.J. (1998). Triangulation in Nursing Research: Issues of Conceptual Clarity and Purpose. Journal of Advanced Nursing 28(3): 631-641.

Stahmer, A.E. (1995). Learners in the Workplace. In: J.M. Roberts & E.M. Keough (eds.) Why the Information Highway? Lessons from Open and Distance Learning (pp. 41-49). Toronto, Canada: Trifolium Books.

Thede, L.Q. (1999). Computers in Nursing. Philadelphia: Lippincott.

Travers, D., Waller, A.E., Bowling, J.M., Flowere, D. & Tintinalli, J. (2002). Five-Level Triage System More Effective than Three-Level in Tertiary Emergency Department. Journal of Emergency Nursing 28(5): 395-400.

Vrasidas, C., & McIssac, M.S. (1999). Factors Influencing Interaction in an Online Course. American Journal of Distance Education 13(1): 22-35.

Weinstein, C.E. Corliss, S.B., Beth, A.D., Cho, Y., & Bera, S.J. Learner Control: The Upside and the Downside of Online Learning. Innovation Abstracts 24 (25). University of Texas at Austin.

Wildman, S., Weale, A., Rodney, C., & Pritchard, J. (1999). The Impact of Higher Education for Post-Registration Nurses on their Subsequent Clinical Practice: An Exploration of Students' Views. Journal of Advanced Nursing 29(1): 246-253.

Yuen, F. (1991). Continuing Nursing Education: Some Issues. Journal of Advanced Nursing 16(10): 1233-1237.

Appendix A

Learner Demographic Survey, Online Learner Support Instrument and Interview Questions

Learner Demographic Questionnaire

In this questionnaire, there are 21 items that gather background information about online learners. Please check the responses that most closely reflect your answer.

Demographic Data

1. What is your age group?

1. <20
2. 20-24
3. 25-29
4. 30-34
5. 35-39
6. 40-44
7. 45-49
8. 50-54
9. 55-59
10. 60-64
11. 65-69

2. What is your gender?

1. Female
2. Male

3. What is your highest level of education?

1. RN diploma
2. Post-diploma certificate
3. Baccalaureate degree
4. Master's degree

4. When did you last take a post-diploma course in Nursing? (before this course?)

1. in the past year
2. two to five years ago
3. 6 to 10 years ago
4. 11-20 years ago
5. more than 20 years ago

5. How many years have you worked as an ER nurse?

1. Less than one year
2. 1-5 years
3. 6-10
4. 11-15
5. 16-20
6. 21-25
7. 26-30
8. 31-35
9. 36-40
10. more than 40

6. Are you employed full-time or part-time in nursing?

1. Full-time
2. part-time

7. Which shift(s) do you work?

1. Days only
2. Days plus shift work

8. Do you have a computer at home?

1. Yes
2. No

9. Where are you accessing your online course?

1. Home only
2. Work only
3. Home & work

10. Where do you work?

1. Rural ER
2. Urban ER
3. Other: please specify

11. If you are taking the course from work where is the computer located? (Select answer)

1. Nursing unit
2. Library
3. Laptop
4. Computer lab
5. Other: please specify

12. Have you ever taken a course that uses the computer in some way?

1. No
2. Yes

13. People take Web-based courses for a variety of reasons. Please indicate those which apply to you.
(You may select more than one answer)

1. Interest in the Internet
2. Course will help at work

14. Do you use the Internet for work purposes?

1. Yes
2. No

15. Do you know of plans at your workplace to help nurses incorporate skills gained from this course into nursing practice?

1. Yes
2. No

Computer Skills

For the following items, please rank your experience on a scale of 1 to 5 at the start of the course.
1=novice   2=beginner   3=intermediate   4=advanced   5=expert

16. How would you rate your basic computer skills (e.g. word processing)?

17. How would you rate your keyboard (typing) skills?

18. How would you rate your skills in using the Internet to find information?

19. How would you rate your skills with using Email?

20. Overall, how would you rate your level of computer abilities on this scale?

For the following item, please rank your experience on a scale of 1 to 5.
1= very group-directed   2=mostly group-directed   3= combination of group and self directed   4= mostly self-directed   5= very self-directed

Group directed: you prefer to learn in groups

Self-directed: you prefer to learn on your own

21. Overall, how would you describe yourself as a learner?

Online Learner Support Instrument

© Lynda Atack, 2001

Part 1: Background Information

1. Where did you access the online course you just completed?

1. Home Only
2. Work Only
3. Home & Work
4. Other

2. If you took the course FROM WORK where did you work most often?

1. Not applicable
2. Nursing Unit
3. Library
4. Laptop
5. Computer Lab
6. Other

3. If you accessed the course FROM WORK was it...

1. Not Applicable
2. During Break at Work
3. Before/After Work
4. From work, on a day off
5. A combination of these times

4. Do you know of plans at your workplace to help you incorporate skills gained from this course into your nursing practice?

1. Yes
2. No

For the following items, please rank yourself at the end of the course on a scale of 1 to 5
1=novice   2=beginner   3=intermediate   4=advanced   5=expert

5. How would you rate your basic computer skills (e.g., Word processing)?

6. How would you rate your keyboard (typing) skills?

7. How would you rate your skills in using the Internet to find information?

8. How would you rate your skills with using email?

9. Overall, how would you rate your level of computer abilities on this score?

Part 2: Online Experience

A. Interaction with Teachers and Peers

Please read each statement below, and then select a number from 1-5 that most closely represents your view on the following items.

1= strongly disagree   2=disagree   3=neutral   4= agree   5=strongly agree
Neutral refers to "neither agree, nor disagree".

10. I was able to participate easily in the online discussion part of the course with my teacher.

11. I felt I was part of a learning group.

12. I had enough feedback from my teacher during the course

13. I learned a lot from my online discussions with the teacher.

14. I was able to participate easily in online discussions with my peers.

15. The information I received during the course was helpful to me in completing my coursework.

16. Online discussions I had with my peers about course topics were helpful to my learning.

B. Course Design and Resources

17. Material that was presented online was well organized.

18. Course content was presented in a way that made it easy to understand.

19. The use of patient simulations presented the subject matter effectively.

20. Directions for online learning activities were clearly presented.

21. I was able to keep track of the section I was working on in the course.

22. Course web pages were easy to read.

23. It was easy to recognize when a new item was added to the online discussions.

24. It was easy to follow the topic during online discussions.

25. I found it easy to find my way around the different parts of the course.

1= strongly disagree   2=disagree   3=neutral   4= agree   5=strongly agree
Neutral refers to "neither agree, or disagree".

C. Technology

26. I was able to email the teacher easily.

27. It was easy to send email to my peers.

28. I was able to obtain any technical help I needed quickly.

29. I found it easy to send messages to the discussion part of the course.

30. The course was always accessible on the server when I was ready to study.

31. There was plenty of technical support in this course.

D. Environment

If you took ANY PART of the course from WORK please respond to items 32- 37.

If you took the course ONLY FROM HOME please go directly to items 38- 41.

32. Computers were conveniently situated for course-work in my workplace.

33. I had enough time to work on the course at work.

34. There were enough computers available for me to work on the course when I wanted.

35. The computer was located in an area that made it easy for me to concentrate.

36. I think it is a good idea to have several nurses taking the course at the same time.

37. I liked learning through an online course at work.

If you DID NOT access the course from home, please go to Items 42-52.

38. I had good cable/modem access from my home.

39. I found working on the course from home a convenient way to take a course.

40. I was able to use the computer whenever I wanted.

41. I like learning through an online course at home.

1= strongly disagree   2=disagree   3=neutral   4= agree   5=strongly agree
Neutral refers to "neither agree, or disagree".

E. Overall Impressions

42. I would participate in another continuing education course on the Web again.

43. Online learning was a convenient way to participate in continuing education.

44. Overall, the support I received during this course on the Web helped me to learn.

45. Patient care in ER would improve if other nurses took this course.

46. I missed talking to other students in the class.

47. Online contact with the teacher was important in this type of course.

48. I found online learning was helpful in meeting my learning needs.

49. I am able to triage patients more accurately now.

50. I would prefer classroom learning in the future.

51. My knowledge of triage has improved since taking this course.

52. I can think of at least two ways this course has improved my practice.

These Last Two Items were Added for the Final Intake

53. We assign a triage code for patients on our ER chart

1. yes
2. no

54. In the space provided, indicate who triages in your ER

1. RN
2. ward clerk
3. other
4. Please Specify (check all that apply).

Thank you so much for taking the time to complete this survey!
Please return the survey in the envelope provided.

Interviews Questions

1. Can you tell me about your experience with the online course?
2. The online tutorial at the start of the course: Did you do this? Did you find it helpful? Should it be mandatory?
3. Marks were given for the online discussion: what are your thoughts about that? Would you recommend this practice?*
4. You were asked to do a project in your workplace. What was your project and what are your thoughts on this assignment? **
5. Has the CTAS course influenced your nursing practice
6. Could you give me some specific examples from your practice?
7. Has the course affected you in any other professional ways that you could tell me about?
8. Is there anything else you would like to add to what we have discussed here today?

* This question was worded differently for nurses in the control group: Do you think marks should have been given for participating in the online discussion?

** This question was omitted for the nurses in the control group.