Page images
PDF
EPUB

ONE

CHAPTER 10

Appraisal of Institute Effects

NE HEALTHY FEATURE of the 1959 summer Counseling and Guidance Training Institutes was their concern about selfappraisal. In every case there was a planned effort to find out how successfully objectives were being achieved. Arrangements were made for continuous week-by-week evaluation of the ongoing program and for final evaluation at the conclusion of the session. Appraisals were made by staff, by enrollees, and sometimes by outside. experts invited in to look the program over. The data collected included reactions to the total program, to separate aspects of it, such as planning and organization, physical facilities and the like, and to separate features, such as courses, seminars, practicum, and field trips.

Many ingenious methods were worked out for obtaining this evidence. The standard quantitative technique is a rating, and many varieties of rating forms appear in the reports. Most of them were variations of graphic scales. As an example, enrollees were asked to make checkmarks corresponding to their opinions on items like the following:

[blocks in formation]

In order to get more differentiated reactions to various aspects or features of the institute, ranking methods were often used. Even persons highly favorable toward the whole experience were thus forced to distinguish between things they liked best about it and things they liked less. In many instances both ratings and rankings were used-ratings to evaluate the overall level of response, rankings to pick up distinctions between different features.

Besides these numerical measures of attitudes, procedures designed to elicit a freer sort of response were tried. In a few instances, enrollees were asked to keep diaries from week to week. These often reflected feelings about what was going on, such as anxieties and misgivings, new insights and convictions. A more commonly used method was a variation of the incomplete sentences technique that has proved so useful in the projective assessment of personality. A blank would be prepared using items of the following type:

1. The institute experience I valued most highly was

2. The most disturbing thing to me was

3. I still feel a need for

Responses to such items can be rated by institute staff for general positive attitude toward the experience, or they can be read primarily for qualitative content, to facilitate an understanding of what each enrollee's experience was like.

In one institute, enrollees discussed the institute in a session led by an outsider. These discussions were tape recorded and analyzed. Most of the appraisal plans included some method of analyzing changes that occurred during the institute session. The most common technique was a comprehensive examination based on the material presented in the courses, given at the beginning and again at the end. An increase in score constituted a straightforward indication that students had learned something from their experience.

The measurement of changes in attitudes and feelings about counseling is a more difficult task, but one considered to be highly important by a number of institute directors. No one method of such measurement was used in a majority of the programs, but several ingenious plans were tried out in individual situations. One obvious possibility is to use a standardized test such as the Minnesota Teacher Attitude Inventory on two occasions. Another is to repeat some of the rating and ranking procedures outlined above.

In addition to these techniques, some new ones were adapted from other areas of personality research. One of these is the Q-sort procedure. Through an analysis of the way in which a subject sorts into separate piles personality items that have been written on separate cards, a picture of his self-concept is obtained. He can also be asked to sort the items in such a way as to describe his ideal self. When the sorting procedure is repeated at a later date, changes can be analyzed, both in the subject's self-concept and in the degree of resemblance between actual and ideal selves. This technique would seem to have some promise for evaluating the kinds of changes an institute hopes to facilitate.

Another interesting special technique is generally called a situational test, because the subject is asked to make his typical response to a standard situation with which he is presented. In a common paper-and-pencil form, each item consists of a few sentences spoken by a counselee during a counseling interview. Space is left for the counselor to write in what he would have said at this point. These responses can be classified according to some predetermined system. In one place, a test of this type was used in a research study of the effects of institute experience. It indicated that there were statistically significant increases in the frequency of understanding responses, accompanied by decreases in the frequency of evaluating, interpreting, supporting, and probing responses. In this particular institute, the first half of the time was devoted to course work, the last half to practicum. It is interesting to note that the major shift in the nature of the responses enrollees made came during the course work even before the practicum had started.1

In several places, attempts were made to assess changes in the way counselors understood the counseling role. One technique was to ask enrollees to make up time distribution sheets for counselor positions, and then to repeat the procedure when the institute was over. Decreases occurred in the amounts of time assigned to peripheral activities of a clerical and administrative nature; increases, in actual counseling activities. One institute made use of a questionnaire about who should handle problems of various sorts. Another used a special questionnaire to reveal the respondent's perception of the counselor's

task.

In several places, attempts were made to evaluate changes in actual counseling skills as well as in knowledge and attitudes. Ratings by the staff and sometimes peer evaluations were used here. In one institute a standardized interview situation was devised to measure what enrollees actually did.

This descriptive account of appraisal procedures in the 50 institutes can do little more than to give the reader some conception of how diverse and interesting they were. Institute staffs were interested in evaluating their work, and they tried out all sorts of ideas. The fact that no two of them used exactly the same procedures makes it impossible to summarize in any but the most general terms what they found. Two conclusions can be stated with certainty: The first is that appraisals of institutes by enrollees were extremely favorable. The modal response was one showing enthusiastic approval of programs as a whole and of most parts and aspects taken separately. The

1 This might indicate that counseling students pick up fairly easily the attitudes of their teachers. They learn what they are expected to say.

second general conclusion is that changes in knowledge, attitude, and skill did occur. Wherever and however attempts were made to measure them, significant differences between initial and final measures were found.

These conclusions should not make one lose sight of the variability of the appraisal data. Some enrollees registered very unfavorable attitudes toward the experience they had undergone, even in situations where almost everybody else described it in glowing terms. Some institutes evoked less enthusiasm than others in the participants. Some parts of the programs were rated considerably lower than others. As one reads the reports, he begins to suspect that these unfavorable ratings do not reflect discredit on the institute programs where they occurred, but they do pose problems for concern. One particular institute, for example, where a good many of the enrollee ratings were somewhat negative, had selected counselors from States that constitute an underdeveloped area, so far as guidance is concerned, and then attempted to present course work that would show them what professional training for counselors is really like. Many of these enrollees did not meet the standards such courses set and were graded accordingly. The experience was undoubtedly painful for some individuals, but perhaps it was essential if improvements are to come. In several other cases, differential evaluations of separate parts of the program resulted in a more negative evaluation of the measurement and statistics courses than of other parts of the program. Yet it seems clear that in present-day high schools, with their extensive testing programs, this is essential knowledge for counselors, whether they like it or not.

Furthermore, the feelings should not be ignored of those few individuals who were unhappy and extremely critical in situations that were congenial and stimulating to the majority. We are tempted to conclude that they do not belong in the counseling profession, but we do not really know this. In the long run, either selection methods should screen out such persons at the beginning, or training methods allow them to progress in their own ways.

The amount and kind of appraisal that could be carried out while institutes were in progress was necessarily limited. The real test of their value will be the accomplishments of the participants after they return to their own schools. Fortunately, followup procedures have been planned for most if not all of the 50 institutes. In many cases the enrollees themselves made these arrangements before they separated. It reflects their strong feeling that this was such an important. experience in their lives that they wished to make permanent the friendships and the kinds of mutual stimulation they had learned to

prize. In addition to this social and inspirational function, followup plans in several places specifically provide for the collection of research data. Tests and attitude measures used during or at the close of the institute are to be repeated at various intervals to measure longterm changes. Information of various sorts about counseling duties is to be collected. These individual followup studies will be of great interest to all those who are concerned with counselor training.

Here, as in the other areas, some novel ideas about followup plans have emerged. In one State, guidance programs in individual schools are to be evaluated, according to a predetermined set of standards, by visiting committees made up of institute enrollees from other schools. In some States, administrators in the school where enrollees serve will be asked to make comments. Since title V (B) of the National Defense Education Act does not provide for the use of any funds for research purposes, such investigations of institute effects must necessarily be the responsibility of the training institutions rather than of the Office of Education. It is gratifying to find that research of this kind has been considered in so many places.

There is one difficulty that arises in trying to draw firm conclusions about the value of the short-term institutes. We cannot assess with certainty the value of the outcomes without one or more control groups for comparison purposes. Two kinds of questions, then, are apparent to any skeptic. First, are changes in institute enrollees significantly greater than changes that occur in similar persons undergoing no special kind of educational experience during the same interval? Second, are changes in institute enrollees significantly greater than changes that occur in similar persons registered in regular summer session courses in counselor education?

Though we must maintain some skepticism until long-range and control group studies have been done, we need not hesitate to say that the qualitative character of the institute experience was memorable. The spontaneous statements of both enrollees and faculty members testify to its unusual vividness. Many instructors remarked that while they had never worked so hard at summer teaching jobs, they had never enjoyed their work so much. The enrollee's eagerness to learn met the instructor's eagerness to teach, and an exciting interchange occurred.

In addition to the high ratings of institute programs and the changes in institute participants, many directors took note of other results. Enrollees, besides adding to their knowledge and skill, and changing their attitudes about many aspects of counseling and human relations, became more professional in their outlook. This was indicated in a number of ways. Many of them worked out plans for fur

« PreviousContinue »