Fig. 1. Pre-/post-quiz scores by department on International Association of Fire Fighters (IAFF) Training Course. had affected their behaviors or that of their peers in responding to alarms. Responses to these questions gained from interviews in three departments (WVA1, WVA2, TX) are among those summarized in Table VII. Positive answers in two departments (WVA2, TX) appear to outnumber those shown for a third department (WVA1), especially with regards to linking personal actions in the field with course learning. Excerpts from all three include greater use of SCBAS in responding to common alarms, checking reference guides to identify potentially hazardous chemicals when responding to calls, limiting actions to defensive control measures while awaiting the HAZMAT team, and cleaning personal protective equipment more frequently Institutional Change In the post-course interviews described above, trainees were also asked: What effect, if any, has this first responders training had on your department's standard operating practices (SOPs)? What effect, if any, has this first responders training had on officers and command staff within your department? Responses to these questions for three departments are found in Table VII. Interview data of trainees in two departments (WVA2, TX) suggest a greater effect than that found in the third Fire Fighter Training for Hazardous Materials Response 337 (WVA1). There appears to be more attention to HAZMAT related procedures, and more thoughtful incident commander actions in responding to calls having the potential for harmful exposures. Regarding the latter, two of the most telling personal accounts of how the course changed command behaviors are noted below: Report: "Rail tank car left track and was spilling contents. I assessed situation from a distance with binoculars and determined that product in car was chlorine. A defensive perimeter was set up and a request made for the HAZMAT team. Before the course I would have just rushed up to the tank car to see the situation up close and probably exposed myself and others needlessly to this harmful chemical." Report: "Before course, our Captain had little use for reference sources such as the DOT ERG. (DOT ERG refers to the Department of Transportation Emergency Response Guidebook, which lists hazardous chemicals along with numbers to call for assistance are included along with evacuation distances). After the course, we responded to an alarm at a residence where our Captain spotted several containers on site. Viewing the placards and labels together with the ERG, a determination was made to pull back the company and call in the HAZMAT team. It was learned that one of the container contents would have reacted with water if we had proceeded and started hose lines: In the past, the Captain said he would have been the last person to pick up and use the ERG." Incidental information obtained during followback contacts with departments suggested other supportive administrative actions being taken since the course. Among those noted were: • Computerized information on chemical materials and properties as found at sites with people occupants; reference texts in vehicles now carry notations of whether they have equipment onboard to respond safely if called to the premises • Arrangements with local hospitals to pick up, sterilize, and return fire fighter clothing suspected of being contaminated with biologic material • More decontamination sinks installed in fire stations RELATIONSHIPS AMONG THE VARIOUS MEASURES Implicit in a popular model of training evaluation criteria is the assumption that positive attitudes about the quality of the instruction or course content, gains in skills or knowledge, behavior change and end goals of such actions are intercorrelated if not causally linked [Kirkpatrick, 1967; Alliger and Janak, 1989; Kirkpatrick, 1994]. Some suggest an ascending order of these measures; that is, favorable reactions to training lead to learning which, in turn, leads to changes in job behaviors, which then leads to related changes in an organization [Newstrom, 1978; Clement, 1982]. Data on the multiple measures from the course evaluations just described offered a means for verifying these assertions. Specifically, it was assumed that more positive reactions to the quality/utility of the course would be correlated with improved competencies and knowledge gain, and these measures, in turn, would be correlated with more behavior change. Pearson product-moment correlations computed between these different measures [Hatcher and Stepanski, 1994] for trainee data obtained in each department are shown in Table VIII. The highest coefficients are found between measures of course reaction and final competency level. All are in the expected direction, i.e., more positive reactions to the course correspond to better final ratings of competency, and are statistically significant. Other significant correlations, how Even measures of competency and knowledge show significant association in only a few instances and correlations between these measures and behavior are fewer still. DISCUSSION First results from evaluating the IAFF hazardous mate rials training course for first responders appear positive. Substantial percentages of trainees rate the instruction as high in quality and utility, and judge themselves more competent in handling tasks when alarms involve risks of exposure to hazardous materials. Significant gains are noted too in the level of knowledge post training, and trainees report more self-protective and preventive actions based on course learning. Lastly, and though still fragmented, there are indications of institutional changes which seem due in part to the trainees' learning experience. While gratifying overall, the aforementioned results also reveal variable effects in some cases which deserve comment or explanation. In doing so, issues are raised not only to this IAFF course but to others having similar objectives. For example, "Physical Properties of Chemicals," were ranked lowest in the quality and utility ratings given to various course topics, and also showed lesser shifts in related task competency. This subject is more technical than others, and course time devoted to this area may not be sufficient to ensure comprehension. In interchanging ideas for dealing with this topic, IAFF instructors have developed many novel laboratory-type demonstrations, enabling the trainees to witness examples of various chemical reactions. But the trainees may be viewing these effects as no more than exotic demonstrations. Further use of exercises to emphasize how such reactions could and have affected the outcomes of hazardous materials events would appear indicated. While competency shifts for all departments suggested improved capabilities, the shifts shown for some departments were greater than found in others. As already mentioned, these results may simply be a case of those with the poorer ratings at the outset of the course catching up with the rest of the group. The IL department whose trainees show the least gain in competency scores may be the result of a problematic training schedule. Indeed, a 6-week period was necessary for each trainee class in this department to receive 3 days of instruction, with some classes having 3- to 4-week intervals between the units of instruction, and taught by different instructors. By contrast, other department training schedules were more compact and used the same instructors thoughout. TABLE VIII. IAFF Course Results: Pearson Correlations (r) Between Different Evaluation Measures "Behavioral data are based on samples only; hence, n values are smaller than those shown for other measures. *Reaction index for each trainee defined as the number of high ratings given to both quality and relevance for the seven course topics noted in Table III. "Competency shift defined as pre/post-course differences in scaled ratings for 9 tasks in exercise. Final competency level defined as average of post-course scaled ratings for the 9 tasks. *Knowledge gain based on pre/post course differences in quiz scores. "Final knowledge quiz score was the post-course quiz score. Behavior index computed for follow through on action statements only. A value of 3 given to each statement for which trainees indicated regular followthrough efforts, 1 for sometimes efforts, -1 for no effort, and 0 for no opportunity. A bonus of 1⁄2 point was given to trainees indicating one-time use of course, and 1 point for using it more than once. The sum of these measures for each trainee defined his/her behavior index. The need for extended time schedules to cover the 3-day course requirements may be even more detrimental to scores on the knowledge quiz. One could argue that the quiz results for IAFF training could be better if the course were given on 3 successive days and with the same instructors. At the minimum, this would alleviate forgetting factors especially for material presented earlier in the course. The greatest gains in quiz scores (36%) for the IAFF first responder course have been posted by cadet classes at the fire academy where training conditions favor 3 successive days of instruction by the same instructors. Recommendations for resolving the time schedule problems just described are being considered by the IAFF. One possibility is to arrange for and pay backup personnel to cover the duties of those attending classes so that the trainees can complete the courses in minimal time. Another is to videotape or produce CD-ROM versions of the course so that those unable to meet class schedules, or called out on emergencies, could still keep up with the course work. The latter is also being considered for refresher training. On this point, extra training may be one explanation of why trainees in one department (WVA2) showed little loss in knowledge quiz scores when retested 6 months after the end of the IAFF course. Roughly two months after completing the IAFF course, the WVA2 trainees received an added day of hands-on instruction in techniques for managing hazardous materials incidents from another organization. Besides better retention, the original course learning reinforced by this added instruction may also be responsible for the other strong carryover effects found for trainees in this department (Table VII). Recommending added training days or refresher courses should have some merit in light of these findings. IAFF has given consideration to adding an extra day to their first responder course for staging mock drills and practical exercises. It bears mention that issues of refresher training are currently under review as part of the OSHA proposal for an occupational safety and health program standard (OSHA, 1996]. Positive actions from the course were most evident in the responses of the TX city department trainees. A greater number of TX city fire fighters followed through on more of their chosen action statements, made use of the course manual, and reported similar actions by their peers. Two reasons for this result come to mind. The first is that the TX city in question was a major petrochemical hub where hazardous materials calls are near daily occurrences. This increased risk presented numerous opportunities to put the course lessons to use. The second is that the IAFF instructors of the TX city course were themselves local fire fighters who used the local experiences and incidents to give the course and its objectives added cogency. Fire fighters in the other cities included in this report were not subject to the same risks, and thus could not be expected to have the same motivation. been mentioned. Refinements to the evaluation plan are also being addressed, especially easier ways to collect and process data on the various measures to gauge course benefits. However, the plan remains built on largely subjective or selfreport measures and, as we have seen, positive changes in some do not correspond to similar changes in others. Whether such training results in fewer occurrences of the sort reported at the outset of this paper will require more verifiable evidence. More objective study of first responder actions and their consequences before and after training is needed. ACKNOWLEDGMENTS The author thanks the staff of the Hazardous Materials Training Department of the IAFF for their support in developing the evaluation plan, overseeing its implementation, and the collection of data. Thanks are also due the IAFF training instructors and trainees who furnished the critical data for assessment. Materials for the IAFF first responder training course were developed via grants from the National Institute of Environmental Health Sciences. The National Institute for Occupational Safety and Health (NIOSH) is currently funding IAFF direct training of this course and its evaluation. The opinions, findings, and conclusions expressed in this document are those of the author and not necessarily those of the LAFF or NIOSH. APPENDIX: ACTION STATEMENTS Correlations found between the various evaluation measures give only limited support to the assumption that they would be interdependent. The evidence for association was most obvious for reaction index and competency measures and not for the others. One possible explanation is that the data on these two measures were collected from the same form which was administered at the same time (i.e., at the end of the course). Measures of knowledge gain reflected two different times of administration, and data on behavior Unit 1: Common Alarms change were obtained well after the course ended. Hence, temporal factors may enhance associations in one instance and obscure them in another. [Lead sentence to each unit]: “I intend to do things that I have not done up to now or will do differently. Specifically, I will. Admittedly, restrictions in the variance for the different measures used in this analysis probably hampered prospects for obtaining larger correlations. For example, competency ratings could vary only across a 4-point scale, and reaction and behavior indices were likewise limited in their range of numerical scores. On the other hand, and as reported by Alliger and Janak [1989], attempts to show interdependency among different levels of evaluation for other training courses have similarly yielded uncertain results. It is believed that uncontrolled factors such as differences in trainee motivation and attitudes, and situation-specific conditions affecting transfer of training may be responsible for the failure to show greater linkages. Ways to improve the content and delivery of the IAFF first responder training course and its evaluation continue to be the subject of internal review. Adding extra days to the course for practical hands-on exercises, and computerized versions for refresher training and other uses have already 1. Review department's standard operating procedures (SOPs) and guidelines to see whether they cover risks of hazardous materials in responding to common alarms. Will suggest ideas for improving SOPs or guidelines. Routinely discuss with others on my shift our responses to calls involving hazardous materials, reviewing conditions and any particular risks to department personnel, civilians, or the environment. Discuss lessons learned from these incidents. 2. 3. 4. 5. 6. Request to attend other hazardous materials training courses. Drive or walk through my first due area to note places likely to be occupied by people, transportation corridors, and other sites where hazardous materials may be stored, carried, or disposed of. Devise my own ways for thinking about or reinforcing defensive actions in responding to alarms that involve strange odors, unknown spilled materials, other potential chemical releases. [Use this space to write your own statement.] |