Introduction

requirements but that most were not fully adhering to the act's reporting requirements concerning the reporting of small business participation goals.

Our March 21, 1986, report entitled Research and Development: A Profile of Selected Firms Awarded Small Business Innovation Research Funds (GAO/RCED-86-113FS) provided information on 19 small firms participating in the SBIR program and discussed the availability of venture capital funds for commercializing results developed with SBIR awards in response to a congressional request for information.

Our report, Federal Research: Effectiveness of Small Business Innovation Research Program Procedures (GAO/RCED-87-63, June 2, 1987), evaluated federal agencies' procedures for making SBIR selections and awards. We found that federal agencies with SBIR activities had established evaluation and selection procedures that reasonably ensured that awards were based on technical merit. However, less than one half of the participating agencies had awarded their SBIR Phase I contracts and grants within 6 months of receiving the proposal, a goal established by SBA guidelines. In addition, we could not determine the length of time needed to make Phase II awards at many agencies because of limitations in agency data.

Federal Research: Small Business Innovation Research Participants Give Program High Marks (GAO/RCED-87-161BR, July 27, 1987) contains information on the characteristics of SBIR recipient firms, the reported effects of the program on firms' operations and products, and the firms' perceptions of the administration of the program.

On March 15, 1988, we issued a legal opinion (B-230594.2), at the request of the Chairman of the House Committee on Small Business, on whether the NRC could maintain an SBIR program if its extramural R&D budget dropped below $100 million. We concluded that federal agencies are not precluded from voluntary participation in SBIR, even when their external R&D budget is below $100 million. NRC subsequently decided to continue its SBIR program on a voluntary basis during fiscal year 1988.

This report was prepared in response to Public Law 99-443, which reauthorized SBIR programs until 1993. The law directs GAO to report on the effectiveness of Phase I and Phase II of the SBIR program, including

the extent to which the goals of the SBIR program are being met,

Introduction

the quality of the research supported by the SBIR program compared with that traditionally supported by the affected agencies, and

the judgments of the heads of departments and agencies as to the effect of SBIR legislation on research programs.

Public Law 99-443 requires GAO to report on SBIR Phase III activities by December 31, 1991. Accordingly, this report includes only preliminary information on this aspect of SBIR activities.

To obtain information on the SBIR program goals of stimulating technological innovation and increasing private sector commercialization and to obtain information on current project status, we selected 1,406 SBIR projects that had been conducted in fiscal years 1983 through 1985, according to a stratified sampling plan described in appendix V. We mailed the firms that conducted these projects a questionnaire asking for information about the firms' experiences with the SBIR program and the characteristics of the firm at which the project took place. We adjusted the analysis of responses to reflect the stratification of the project sample, as described in appendix V. The questionnaire, summary of responses, response rate, and selected sampling errors are included in appendix II.

To obtain information on the goals of stimulating technological innovation and meeting federal R&D needs, as well as the quality of SBIR research projects in comparison with other research supported by R&D agencies, we mailed two types of questionnaires to 530 project officers who had administered SBIR projects in DOD, DOE, HHS, NASA, and NSF— agencies that together administer 96 percent of all SBIR funds. All project officers received one questionnaire asking for resporses concerning the SBIR program in general, as well as one or more questionnaires concerning individual SBIR projects that they had been responsible for. The questionnaire concerning individual SBIR projects asked the project officers to compare the SBIR project with non-SBIR research for which they were responsible. To measure research quality, we asked project officers to compare specific SBIR projects with other research that they were responsible for according to factors that we identified as potentially relevant to research quality by consulting science policy experts, reviewing published material, and pretesting questionnaires. To obtain information concerning incomplete or unclear responses, we followed up with telephone calls to selected respondents to all three questionnaires. The questionnaire concerning the SBIR program in general, together with a summary of responses and response rate, is included in appendix III. The questionnaire about individual SBIR projects, with responses and

Introduction

response rate, is included in appendix IV. Appendix V contains information on the selection approach and the techniques we used for all questionnaires.

We interviewed SBIR program managers and other officials and reviewed records in DOD, DOE, DOED, HHS, NASA, NSF, EPA, NRC, and SBA to obtain information about efforts to foster and encourage participation by minority and disadvantaged persons in technological innovation and about the extent to which program goals are being met and the quality of SBIR research. We also consulted with experts in research evaluation, technological innovation, and government policies to encourage the commercialization of R&D. These experts were located in government agencies, academic institutions, and private practice. We also solicited, and received, judgments concerning the effect of SBIR legislation on research programs in 11 agencies: USDA, Commerce, DOD, DOED, DOE, HHS, DOT, EPA, NASA, NSF, and NRC. Their responses are included in appendixes VI through XVI.

We performed this review in accordance with generally accepted government auditing standards. This review was conducted from September 1987 to September 1988, primarily at the agencies' headquarters offices in the Washington, D.C., area.

Are SBIR Programs Meeting Their Goals?

Stimulating
Technological
Innovation

Three of the four SBIR program goals-to stimulate technological innovation, use small business to meet federal R&D needs, and increase private sector commercialization of innovations from federal R&D are complex, interrelated, and hard to measure. For example, the development of new technological innovations may be critical to meeting federal R&D needs. Private sector commercialization, which depends on the development of new technological innovations, may contribute to meeting federal R&D needs in areas such as health or aeronautics. Although all agencies seek to stimulate technological innovation, agencies differ in the emphases they place on meeting federal R&D needs and on increasing private sector commercialization of federal R&D.

SBA and agencies with SBIR programs seek to achieve the fourth SBIR program goal-to foster and encourage participation by minority and disadvantaged persons-through outreach programs to inform them about SBIR activities. According to data compiled by SBA, the percentage of money awarded to minority and disadvantaged firms was lower in fiscal years 1986 and 1987 than in the 2 previous fiscal years, but SBA officials believe that the data may contain some inaccuracies because of inconsistent reporting by participating firms.

DOD and NASA have SBIR programs that strongly emphasize the goal of meeting federal R&D needs by soliciting and funding projects that are closely coordinated with agency applied R&D programs to meet agency mission objectives. In contrast, programs at NSF and HHS emphasize the selection of projects with high potential for private sector commercialization within broad technological categories of interest to these agencies, and SBIR projects are less closely coordinated with other agency programs, which focus mainly on basic research at academic institutions. SBIR programs at other agencies, such as DOE, seek-like DOD and NASA-to meet specific agency R&D objectives with some projects but also try to support private sector commercialization with other projects.

Technological innovation is a complex, hard to measure process, and federal agencies seek to stimulate technological innovation in many different areas. Although difficult problems in assessing technological innovations exist, and only limited comparisons are possible across the wide range of federal efforts to stimulate innovations, several factors indicate that SBIR programs have been supporting projects that contribute to technological innovation.

Are SBIR Programs Meeting Their Goals?

SBIR programs have adopted highly competitive selection procedures to identify those proposals of highest technical quality and innovative potential, and only about 5 percent of the proposals obtain funding through Phase II.

According to their questionnaire responses, agency project officers rated
many Phase II projects as technologically innovative and in general
ranked many SBIR projects more likely than other research for which
they were responsible to lead to inventing and commercializing new
products, processes, and services.

Responding to our questionnaire, firms reported that a high proportion
of projects would not have been undertaken without SBIR funding. In
analyzing the questionnaire responses, we found that projects that prob-
ably or definitely would not have been undertaken without SBIR funding
were about as likely as other projects to produce patent applications, or
lead to market testing, and somewhat less likely to result in follow-on
R&D or commercial products, indicating that SBIR programs are contribut-
ing to technological innovations that might not have occurred otherwise.
Firms also indicated that they are continuing R&D on some projects after
SBIR funding is completed.

Although definitions vary, there is widespread agreement that technological innovation is a complex process, particularly in the development of sophisticated modern technologies. Technological innovation can involve many steps, including research, engineering, prototype testing, and product development. The steps necessary for technological innovation can differ, depending on the specific situation. Technological innovation is closely related to the process of commercialization, which includes the development and marketing of new goods and services. It is important to recognize that technological innovation is an uncertain process so that, even in an ideal world, the results of the SBIR projects would not be all positive: supporting truly innovative, ground-breaking research implies that failed or unsuccessful projects will be a regular, and even frequent, occurrence.

Measuring technological innovation is difficult, for several reasons. Because technological innovation occurs in many different ways, no one indicator can accurately assess innovativeness. For example, patents may serve as a good indicator of technological innovation in the development of some products but be less useful in measuring other innovations, such as new computer software, where patents are less relevant. In addition, differences among firms can create measurement problems.