% with positive

86

showing upward shifts in their competency ratings as a result of the training.

Two departments (WVA1, WVA2) display near-identical results in trainee competency ratings when compared to the other three departments. Both show relatively poor competency levels before the course and then greater amounts of positive shift afterward. A higher percentage of trainees in these two departments also display positive shifts than in the other three. In effect, these changes bring the two departments up to the same post-course competency levels as the other three. Statistical treatments of the competency data (mixed analysis of variance (ANOVA] as described by Hatcher and Stepanski [1994]) yielded significant differences between departments for both the percent showing shift and amount of shift values. Post hoc analysis showed the WVA1 and WVA2 departments to be superior to the other three in these comparisons; the IL department was found to be significantly poorer than all others in terms of the percentage showing shift.

Rank-ordering the competency data by the nine tasks within departments revealed no significant agreement across the five departments. However, like the quality and utility results reported above, competency shifts for the task, “Applying knowledge of chemical properties in assessing exposure risk" tended to have the lowest ranks in most departments.

Knowledge Gain

Plotted in Figure 1 are the mean percentages of correct answers on the 75-item test taken just before and at the end of the IAFF course for five departments. Differences between the pre- and post-course scores reflecting the knowledge gained from the instruction were greatest for the TX

335

department (11.3%). Nevertheless, statistical tests (ie., mixed ANOVA treatments with post hoc multiple comparisons as described by Hatcher and Stepanski [1994]) found all such differences to be significant (p <0.01).

Follow-up visits in three departments (WVA1, WVA2, and TX), conducted from 6 to 12 months after the course ended, included retests on the original quiz for samples of trainees randomly drawn from the class rosters. Table V shows the mean quiz scores for the pre-, post-, and follow-up course test times for these trainees and differences between these measures. Most notable is the slight drop (less than 3%) in the follow-up test scores for one department (WVA2) when compared to its post-course results. Similar comparisons for the two other departments (WVA1, TX) show losses that are 3-4 times greater. A mixed analysis of variance and post hoc multiple comparisons of the data [Hatcher and Stepanski, 1994] confirmed significant follow-up losses only for the latter two departments. One probable reason for these differential findings in retention is offered in the discussion.

Behavior Change

The aforementioned follow-up sessions with samples of trainees, also included one-on-one interviews with IAFF staff to determine carryover impacts of the course. Interviews to determine the extent of followthrough on actions chosen by the trainees have been conducted in four departments as of this reporting. During such interviews, each action statement originally chosen by the fire fighter was read, and questions asked about followthrough. An add-on question during each interview was whether the course manual had been used by the trainee after the course, and if so, how frequently. The responses of trainees in four departments are shown in Table VI.

The percentage of trainees in all samples reporting regular adherence to at least one action is 85% or better and to two selected actions is greater than 60%. Followthrough for three or more actions shows a significant drop for all but the TX City trainees. Almost twice as many TX City trainces also report at least a one-time use of the course manual in comparison to the other groups.

The selected actions with the highest followthrough counts in the four department samples revealed both similarities and differences. Three statements, each with high followthrough rates, were common in all four departments:

• Routinely observe placards and other information systems at fixed sites and on transportation cargo carriers to test my accuracy in recognizing hazard markings.

• Keep a record of my responses to alarms where hazardous materials were detected. Learn about the materials and their harmful effects.

• Devise my own ways for thinking about or reinforcing

had affected their behaviors or that of their peers in responding to alarms. Responses to these questions gained from interviews in three departments (WVA1, WVA2, TX) are among those summarized in Table VII. Positive answers in two departments (WVA2, TX) appear to outnumber those shown for a third department (WVA1), especially with regards to linking personal actions in the field with course learning. Excerpts from all three include greater use of SCBAs in responding to common alarms, checking reference guides to identify potentially hazardous chemicals when responding to calls, limiting actions to defensive control measures while awaiting the HAZMAT team, and cleaning personal protective equipment more frequently Institutional Change

In the post-course interviews described above, trainees were also asked:

What effect, if any, has this first responders training had on your department's standard operating practices (SOPs)?

What effect, if any, has this first responders training had on officers and command staff within your department?

Responses to these questions for three departments are found in Table VII.

Interview data of trainees in two departments (WVA2, TX) suggest a greater effect than that found in the third

(WVA1). There appears to be more attention to HAZMAT related procedures, and more thoughtful incident commander actions in responding to calls having the potential for harmful exposures. Regarding the latter, two of the most telling personal accounts of how the course changed command behaviors are noted below:

Report: "Rail tank car left track and was spilling contents. I assessed situation from a distance with binoculars and determined that product in car was chlorine. A defensive perimeter was set up and a request made for the HAZMAT team. Before the course I would have just rushed up to the tank car to see the situation up close and probably exposed myself and others needlessly to this harmful chemical."

Report: "Before course, our Captain had little use for reference sources such as the DOT ERG. (DOT ERG refers to the Department of Transportation Emergency Response Guidebook, which lists hazardous chemicals along with

numbers to call for assistance are included along with evacuation distances). After the course, we responded to an alarm at a residence where our Captain spotted several containers on site. Viewing the placards and labels together with the ERG, a determination was made to pull back the company and call in the HAZMAT team. It was learned that one of the container contents would have reacted with water if we had proceeded and started hose lines: In the past, the Captain said he would have been the last person to pick up and use the ERG."

Incidental information obtained during followback contacts with departments suggested other supportive administrative actions being taken since the course. Among those noted were:

• Computerized information on chemical materials and properties as found at sites with people occupants; reference texts in vehicles now carry notations of whether they have equipment onboard to respond safely if called to the premises

• Arrangements with local hospitals to pick up, sterilize, and return fire fighter clothing suspected of being contaminated with biologic material

• More decontamination sinks installed in fire stations

RELATIONSHIPS AMONG THE VARIOUS MEASURES

Implicit in a popular model of training evaluation criteria is the assumption that positive attitudes about the quality of the instruction or course content, gains in skills or knowledge, behavior change and end goals of such actions are intercorrelated if not causally linked (Kirkpatrick, 1967; Alliger and Janak, 1989; Kirkpatrick, 1994]. Some suggest an ascending order of these measures; that is, favorable reactions to training lead to learning which, in turn, leads to changes in job behaviors, which then leads to related changes in an organization [Newstrom, 1978; Clement, 1982]. Data on the multiple measures from the course evaluations just described offered a means for verifying these assertions. Specifically, it was assumed that more positive reactions to the quality/utility of the course would be correlated with improved competencies and knowledge gain, and these measures, in turn, would be correlated with more behavior change. Pearson product-moment correlations computed between these different measures [Hatcher and Stepanski, 1994] for trainee data obtained in each department are shown in Table VIII.

The highest coefficients are found between measures of course reaction and final competency level. All are in the expected direction, i.e., more positive reactions to the course correspond to better final ratings of competency, and are statistically significant. Other significant correlations, how

Even measures of competency and knowledge show significant association in only a few instances and correlations between these measures and behavior are fewer still.

DISCUSSION

First results from evaluating the IAFF hazardous materials training course for first responders appear positive. Substantial percentages of trainees rate the instruction as high in quality and utility, and judge themselves more competent in handling tasks when alarms involve risks of exposure to hazardous materials. Significant gains are noted too in the level of knowledge post training, and trainees report more self-protective and preventive actions based on course learning. Lastly, and though still fragmented, there are indications of institutional changes which seem due in part to the trainees' learning experience.

While gratifying overall, the aforementioned results also reveal variable effects in some cases which deserve comment or explanation. In doing so, issues are raised not only to this IAFF course but to others having similar objectives. For example, "Physical Properties of Chemicals," were ranked lowest in the quality and utility ratings given to various course topics, and also showed lesser shifts in related task competency. This subject is more technical

than others, and course time devoted to this area may not be sufficient to ensure comprehension. In interchanging ideas for dealing with this topic, IAFF instructors have developed many novel laboratory-type demonstrations, enabling the trainees to witness examples of various chemical reactions. But the trainees may be viewing these effects as no more than exotic demonstrations. Further use of exercises to

emphasize how such reactions could and have affected the outcomes of hazardous materials events would appear indicated.

While competency shifts for all departments suggested improved capabilities, the shifts shown for some departments were greater than found in others. As already mentioned, these results may simply be a case of those with the poorer ratings at the outset of the course catching up with the rest of the group. The IL department whose trainees show the least gain in competency scores may be the result of a problematic training schedule. Indeed, a 6-week period was necessary for each trainee class in this department to receive 3 days of instruction, with some classes having 3- to 4-week intervals between the units of instruction, and taught by different instructors. By contrast, other department training schedules were more compact and used the same instructors thoughout.