SIMILAR FINDINGS BY EXPERT EVALUATORS OF VOCATIONAL EDUCATION PROGRAMS The Center for Vocational and Technical Education at Ohio State University has compiled and summarized the findings of numerous expert evaluators of Vocational education. A consensus of these findings, prepared by the Center, showed that these evaluators have encountered problems with information on program results similar to those we found. One of the Center's 1970 publications commented: "Follow-up studies continue to be plagued by some inherent hazards-reliance upon questionnaires and interviews and the attendant problems of accounting for non-respondents ***. Research workers agree that decisions are being made about occupational education and training programs without adequate informa tion about their current or potential effectiveness, Some school officers have neither research budgets nor skilled research workers for mounting *** evaluative efforts. As education's share of tax expenditures increase, however, it is fairly certain that educational administrators will be asked to provide better information than they now possess about the cost-effectiveness of their programs, vocational or non-vocational." HEW regional officials generally concurred with our findings and said thatparticularly in the area of inaccurate data-they could have done a better monitoring job with additional staff. The Program Officer for Reports and Statistical Data at HEW headquarters agreed that HEW and the States were having difficulty in obtaining the information necessary to adequately evaluate programs, He said that attempting to develop a management information system to adequately evaluate program effectiveness has been a slow process. One of the problems has been convincing persons at all levels –Federal, State, and local- of the importance of such a system. He said that most educators---State and local-did not consider this information important and that they were more concerned with the techniques of teaching and curriculum de velopment than with evaluating program effectiveness, He said that adequate systems would be implemented only when officials at all levels were convinced of the value of good management information and that efforts were continuing in this direction. CONCLUSIONS Improved management information systems which provide adequate and accurate information, particularly on program results, are needed if program Lanagers are to adequately evaluate programs, identify weaknesses, and make necessary improvements. The need for such systems has been recognized by the States and HEW. of concern to us is that the independent development of management information systems by HEW and the various States-which seems to be the current trend--could easily result in duplication of effort without the valuable benefit gained from the comparability of data and the interchange of ideas. HEW effcials should more closely coordinate their efforts with those of State and local governments in defining the information needed for an adequate managment information system and should assist the States in establishing such systems. HEW officials should also explore the possibility of using techniques, such as statistical sampling and analyses of Social Security data, to assess the effectiveness of vocational education. They should also consider (1) gathering followup information on nonvocational graduates to better assess the impact of vocational education and (2) compiling followup information over a longer period after graduation. Combining several or all of these techniques might result in better information, whal holding costs to a reasonable level. Any system, however, will be of limited usefulness if the information gathered is inaccurate and incomplete. RECOMMENDATIONS TO THE SECRETARY OF HEALTH, EDUCATION, AND WELFARE HEW should coordinate its efforts and those of the States in defining the information needed to adequately evaluate program results and should assist 99 98 -73-rol 2 --17 the States in the design and implementation of management information systems. The Secretary should take action to insure that HEW and the States-in monitoring programs-verify the accuracy and completeness of reported information. AGENCY COMMENTS AND ACTIONS The Assistant Secretary, Comptroller, agreed with our recommendations, stating that: "Regional staff through personal consultation with State personnel and by conducting leadership workshops for State and local administrators are taking positive steps to achieve the objectives included in the recommendations. Suen action consists of conducting State program reviews and reemphasizing proper accountability throughout the entire system to improve report information." Officials of the four States involved also agreed. The actions promised or taken by HEW and the States should result in needed program improvements. CHAPTER 5.-SCOPE OF REVIEW We reviewed selected aspects of the vocational education programs in California, Michigan, Ohio, and Pennsylvania for fiscal years 1970 and 1971. These States have been consistently among the top 10 in amounts of Federal assistance received. They received $104 million, or 22 percent of the total allotted to all States, in fiscal year 1972. Our review was directed toward determining whether legislative objectives were being achieved and toward identifying major problems. Our review concentrated on high school vocational education, because the majority of funds in all four States are spent on this level of education. California spends also a significant portion of its vocational education funds on post-secondary vocational education in 2-year community colleges, and the other three States have also undertaken programs to increase vocational training in community colleges. We included community college vocational education programs in the preliminary phase of our review, but expenditures in all four States had not reached a sufficient level to warrant including these programs in our detailed review work. We reviewed the Vocational Education Act of 1963 and its 1968 amendments, the legislative history of these acts, and the administrative regulations and instructions for their implementation. We also reviewed the plans of the four States to determine if the objectives of the act were included. We interviewed Federal, State, and local education officials and reviewed available records and reports to determine if these objectives were understood and to obtain information on the progress of State programs, the extent of State and local funding, and the direction of effort. Our work was performed primarily at State departments of education and at local education agencies in three selected cities in each State-one large (over 500,000 population), one medium (100,000 to 400,000 population), and one small city (under 100,00 population). In each of the large cities, we visited three high schools, in the medium cities two high schools each, and in the small cities one high school each. Our work also included reviews and discussions at HEW headquarters and regional offices responsible for programs in the four States. We considered recent findings and conclusions by expert evaluators of vocational education, as summarized by the Center for Vocational and Technical Education. The Center, an independent unit of Ohio State University, receives funds from HEW to be used, in part, for summarizing reports of vocational education studies by experts under contract with HEW, State and Local governments, and other public and private groups. We also reviewed the annual reports prepared by or for the National Advisory Council and the four State advisory councils. APPENDIX I LEVELS OF STATE AND LOCAL EXPENDITURES FOR VOCATIONAL EDUCATION, FISCAL YEARS 1963 AND 1970 AND STATE ESTIMATES FOR 1975 Associate Director, Manpower and Welfare Division, Washington, D.C. DEAR MR. HY NIG: The Secretary has asked that I reply to your letter dated April 25, which forwarded your draft report entitled "Training America's Labor Force: The Unfilled Promise of Vocational Education.” Detailed comments on the recommendations, together with the statement of actions to be taken to implement them, are set forth in the enclosure. They are the product of a review of the report by the eognizant Departmental and Office of Education regional and headquarters staff. We appreciate the opportunity to review and comment on the report. Enclosure. JAMES B. CARDWELL, Assistant Decretary, Comptroller, DEPARTMENT OF HEALTH, EDUCATION AND WELFARE COMMENTS PERTINENT TO THE DRAFT Report to the Congress of the United STATES BY THE UNITED STATES GENERAL ACCOUNTING OFFICE ENTITLED "TRAINING AMERICA'S LABOR FORCE: THE UNEHED PROMISE OF VOCATIONAL EDUCATION" A general reaction is that this is a fair and impartial report. However, the tone and accuracy of the report could be improved by changing the title of the report. The present title implies a total study of vocational education in contrast to the actual scope of the study which relates primarily to the disadvantaged. Replacing the word "Promise" with "Goal" would be much more acceptable. Also, Regional Adult, Vocational and Technical Education Director should be substituted for HEW Director (page 10). In addition, the nature of the report militates against the inclusion of positive achievements. Better balance could be achieved in the report if some positive accomplishments in Vocational education during the period covered were included. For example, the report might identify the expansion of programs with reference to the number of participants as well as occupations. GAO recommended that: The Secretary of HEW should undertake research into the exact nature and extent of the funding and image problems attached to vocational education with toward determining what actions may be necessary to more fully achieve the objectives of the Vocational Education Act, Department Comment We concur with the recommendation. The Office of Education, through its central and regional offices, will undertake research into what appears to be primarily an attitudinal matter. In addition, a research study of a positive and definite type will be mounted as soon as practicable to produce a report on vocational education graduates who have been employed for five or more years. The Secretary of HEW should instruct regional offices to more closely monitor the use of Federal funds for special programs and services for the disad vantaged to insure that these funds are being used as intended by the Act and HEW implementing guidelines. HEW should also require the States to describe, in their State plans, the procedures they intend to employ to ensure that funds for the disadvantaged are properly used. Department Comment We concur with the recommendation. One of the States studied has established stricter controls for LEA's for use of disadvantaged funds. Regional staff are emphasizing to the other States the concerns reflected in the GAO study and are expanding monitoring activities. HEW should coordinate its efforts and those of the States in defining the information needed to adequately evaluate program results and assist the States in the design and implementation of management information systems. The Secretary should take action to ensure that HEW and the States-in monitoring programs-verify the accuracy and completeness of reported information. Department Comment We concur with the recommendation. Regional staff through personal consultation with State personnel and by conducting leadership workshops for State and local administrators are taking positive steps to achieve the objectives included in the recommendation. Such action consists of conducting State program reviews and reemphasizing proper accountability throughout the entire system to improve report information. APPENDIX III.-PRINCIPAL OFFICIALS OF THE DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE, RESPON. SIBLE FOR THE ADMINISTRATION OF ACTIVITIES DISCUSSED IN THIS REPORT 1 On March 31, 1971, an Assistant Secretary for Public Affairs was appointed to the position formerly held by the Assistant Secretary for Education PART III STATE ADVISORY COUNCILS ON VOCATIONAL EDUCATION: Analysis of the Third Annual Reports INTRODUCTION We are pleased to report that the approximately $500 million. provided vocational education by the Federal Government to the States last year has helped in achieving improvement in quality of programs and expansion of enrollments at all school levels and for all target groups, according to the reports of the State advisory councils on vocational education. However, many problems remain unresolved and many needs are still unmet in terms of our Nation's expectations from vocational education. The source of the substantive information in this part of the National Advisory Council on Vocational Education's report is the findings, deliberations, activities, and recommendations of the State advisory councils on vocational education as contained in their third annual reports. Each State council report was analyzed by National Council staff, including 14 specific topics considered of major significance to vocational education. Sections A, B, and C of this part contain, respectively, the current recommendations of the councils, the previous year's recommendations of the councils and notes on action taken on them by the State boards, and summary listings of the activities in which the councils engaged during the past year. The information for each of the these sections, as well as section D described below, was taken almost verbatim from the State council reports. Each section is organized alphabetically by State. Section D contains selected analyses of findings of the State councils organized by topic and alphabetically. Persons interested in learning what any particular State council has to say about the problems of vocational education should review sections A and B before going on to the topical chapters in section E. Although there was a recommended format for the reports, many States adapted their report to meet their own requirements. It should also be noted that much more information is contained in the State council reports than was extracted for this report. For example, some reports include studies and statistical data of prime importance to researchers interested in specific topies. Following is a brief summary of the findings reported in section D. Recommendations for resolving many of the identified problems and issues are contained in section A and B. Topie. The acceptance and implementation of the career education concept by the State departments of education, the State advisory councils and the local school systems. |