Are SBIR Programs Meeting Their Goals? Some innovative firms will file many patent applications, while others will prefer to retain trade secrets. Because of the wide diversity in the R&D responsibilities of federal agencies, the agencies seek to encourage innovation in many different technological areas, making comparisons difficult. NASA, for example, seeks innovation in areas related to aeronautics and astronautics, such as new aircraft designs, power systems for spacecraft, and lightweight construction methods. Similarly, DOD, DOE, HHS, and other agencies try to develop new technologies that can help them meet mission responsibilities in areas such as defense, energy, and health. In addition to supporting technological innovation to meet a wide range of mission responsibilities, agencies also support research to improve fundamental scientific knowledge that can ultimately lead to technological innovations. NSF funds basic research at universities in a wide range of disciplines, while HHS provides almost all federal support for basic research in biological areas related to health needs, and DOE is responsible for basic research concerning high energy and nuclear physics. Other agencies also fund lesser amounts of basic research. SBIR programs seek to promote technological innovation primarily through the identification and funding of project proposals with high scientific and technical merit. SBA has established the following criteria, which must be considered in the evaluation of Phase I and Phase II SBIR proposals: the technical approach and the anticipated benefits to be derived from the research, the adequacy of the proposed effort and its relationship to fulfilling the requirements of the research topic or subtopics, the soundness and technical merit of the proposed approach and its incremental progress toward topic and subtopic solution, and qualifications of the proposed principal investigators. When Phase II proposals are of equal technical and scientific merit, special consideration is to be given to proposals that demonstrate commitments from nonfederal sources to support further development after completion of Phase II (Phase III follow-on funding commitments). An SBA official said that a main purpose of these criteria is to identify proposals of high technical merit that are likely to lead to innovations. In addition to directing use of these criteria, SBA encourages SBIR programs Are SBIR Programs Meeting Their Goals? to adopt proposal selection procedures used in funding other agency research. At all agencies, the selection procedure starts with a widely distributed proposal solicitation, usually issued annually. In some agencies, awards decisions are made by the central SBIR office after the awards are reviewed and rated by technical officers, while at others the decisions are made in a decentralized manner. In a June 1987 report,' we reviewed the selection procedures for SBIR awards at 11 agencies. All agencies used four procedures to ensure selection of proposals of high technical quality: (1) evaluations by technical experts, (2) use of SBA's selection criteria, (3) utilization of a system to rate or rank proposals, and (4) selection based on a ranking system. While we found some differences in emphasis among agencies, we concluded that agencies are making a good faith effort to maintain a system that is fair and provides for final selection based on technical merit. Although innovation is not addressed specifically by SBA's selection criteria, all agencies have revised SBA's criterion concerning technical merit to include consideration of a proposal's innovativeness and originality in making Phase I awards. In addition, the following factors indicated SBIR programs were funding proposals of high technical quality: the SBIR proposal selection process was highly competitive, because a large "pool" of proposals was available for agencies to consider in selecting proposals that meet standards of technical quality; the high average scores received by successful proposals indicated that quality research was being funded under agencies' SBIR programs; and SBIR program managers judged the quality of funded proposals as good to excellent. 1Federal Research: Effectiveness of Small Business Innovation Research Program Procedures (GAO/ RCED-87-63, June 2, 1987). SBIR Project Officer Table 2.2: Project Officer Responses Only a small fraction of all SBIR proposals obtain substantial SBIR funding. As table 2.1 shows, since 1984, about 15 percent of the proposals have received the relatively small Phase I awards. In fiscal year 1987, only 35 percent of the projects completing Phase I were selected for the larger Phase II awards. Thus, only about 5 percent of all proposals received Phase II funding in 1987. Our mail questionnaires asked SBIR project officers to assess (1) how well SBIR programs stimulate technological innovation, (2) whether individual SBIR projects were innovative, and (3) whether individual SBIR projects were more likely than other research for which the officer was responsible to lead to innovation and commercialization. As table 2.2 shows, a large majority of project officers responded that the SBIR program definitely or probably supports technological innovation. The percentage of project officers that thought that the SBIR program certainly or probably helped stimulate technological innovation was highest at NASA (89 percent), followed by DOD (88 percent), DOE (78 percent), HHS and NSF (73 percent each). When we asked about specific Phase II SBIR projects that the officers had managed, 23 percent of the project officers rated the project as very innovative, while 38 percent believed their project was moderately innovative. Only 5 percent reported that the project that they managed was not innovative at all. Project officers at different agencies again varied in their assessments of individual projects. NASA project officers rated Are SBIR Programs Meeting Their Goals? SBIR Firm Responses the highest percentage of projects in our survey as moderately or very innovative (73 percent), followed by DOD (64 percent), DOE (63 percent), HHS (48 percent), and NSF (48 percent). Project officers believed that over half (53 percent) of the SBIR projects were more likely than non-SBIR research under their responsibility to produce inventions or products. Another 29 percent of the SBIR projects were assessed as having the same likelihood of invention or commercialization as non-SBIR projects. (Ch. 3 includes more information on these responses as part of our analysis of research quality.) To obtain information on whether SBIR projects were funding research that would not be done otherwise, we asked firms whether they would have undertaken the research without this support and then analyzed the reported results of these projects. We also asked firms whether they were continuing R&D on projects that were no longer receiving SBIR funding. Firms reported that much of the research would not have been undertaken without SBIR. Only 16 percent said they would have definitely or probably done the research without the SBIR program, 20 percent were uncertain, and 64 percent said they definitely or probably would not have proceeded. We analyzed the questionnaire responses to see whether completed As table 2.3 shows, firms reported that projects that probably or definitely would not have been undertaken without SBIR funding were about as likely as other SBIR projects to produce patent applications and lead to market testing. These projects were, however, somewhat less likely to result in continuing R&D or have results that were being sold commercially. To determine whether SBIR programs encouraged firms to invest additional resources in R&D after completion of SBIR funding, we asked firms about the current status of SBIR projects. Firms responding to our questionnaire indicated that SBIR programs encouraged them to continue R&D using their own funds. Firms reported that they are continuing R&D on 49 percent of all SBIR projects that have completed Phase II. In addition, some firms have decided to continue R&D when projects did not receive a Phase II award. Firms reported continuing R&D on 34 percent of the projects that did not receive Phase II funding. In comments added to their questionnaire responses, several SBIR awardees told us that especially risky efforts would not have been undertaken by their firms without SBIR support. For example, one firm said that SBIR funding from DOE had helped it develop a new medical device to the stage at which it could be demonstrated to the private sector. A second company with an SBIR project investigating the use of Xrays noted that the program's support had allowed it to develop projects that investors were often unwilling to back. |