In the 1980s, there was a growing realization by many business analysts and consumers that the quality of U.S. products and services was falling behind that of other leading nations, especially Japan. Some industry and government leaders saw the need for an emphasis on quality as a necessity for doing business in an ever expanding, and more demanding, global market. Many American businesses either did not understand the quality problem or did not know where to begin. Some U.S. business leaders, noting the success of the Deming Prize in improving the quality of Japan's products, suggested that a U.S. quality award might create an incentive for quality improvement and a basis for sharing information on successful quality strategies.

A difficult choice for a U.S. award was whether it should be managed by a private or government entity. Some argued that only those directly experienced in business could provide the needed understanding. Others believed that only a Presidential Award could bring the needed national visibility and recognition. The final decision was to utilize the strengths of both sectors. The pending legislation called for a publicprivate partnership in which each sector plays a crucial role. In August 1987 Congress enacted legislation establishing the Malcolm Baldrige National Quality Award and assigned overall management of the Award to the National Bureau of Standards (soon to become the National Institute of Standards and Technology). Congress selected NBS/NIST to design and manage the Award program because of its role in helping U.S. companies compete, its world renowned expertise in measurement quality and quality assurance practices, and its reputation as an impartial third party. NIST worked with key private sector leaders to create a private organization, the Foundation for the Malcolm Baldrige National Quality Award. In 1988, the Foundation raised $10M to support the purposes of the Award.

Congress named the award in honor of Malcolm Baldrige, Secretary of Commerce during the period when the legislation was developed. Just prior to the enactment of the legislation, Mr. Baldrige died after a rodeo accident. This naming was in recognition of the efforts of Secretary Baldrige to strengthen U.S. competitiveness. The Award is the country's highest level of recognition for performance excellence and has been presented annually by the President since 1988.

The intent of the Award is to recognize U.S. companies for their achievements in quality and business performance and to raise awareness about the importance of quality and performance excellence as a competitive edge. It was envisioned that the Award would serve as a standard of excellence to aid U.S. companies in achieving world-class quality. Rather than recognize specific products or services, the Award Program would recognize role-model business practices and emphasize the sharing of best practices. Three Award categories were initially permitted: manufacturing, service, and small business.

A key step in implementing the new Law was developing the criteria for the Award. In late 1987, NBS/ NIST created the basic criteria concept and category structure as well as the basis for a multi-stage evaluation. In parallel with this Award design, a network of private sector experts was created to implement and expand the design and to serve as evaluators. A design feature of the Award Program was the commitment to annual revision of the criteria to reflect key learnings and changing business needs. The first Award Criteria were published in the Application Guidelines for the Malcolm Baldrige National Quality Award in1988. [1]

Although the criteria were needed to serve as a basis for making Awards, their use was intended mainly for organizational self-assessment and improvement. The 1988 Application Guidelines included Award Criteria based upon seven evaluation Categories. Each Category consisted of two or more Examination Items, which serve as the basis for Awards and for self-assessment. The Award Criteria represented a major new vehicle for the United States to define best quality practices and to disseminate practices more quickly using an approach that permits self-diagnosis. Thus the Criteria serve as the focal point in education and cooperation, going beyond the requirements of the Award contest, via a kind of excellence standard.

The Criteria take a systems approach to organizational excellence. They incorporate a set of core values and concepts that reflect beliefs and behaviors found in high performing organizations. The Criteria are expressed as operational requirements, including key processes and measures that connect processes to comprehensive business results. The business results include not only financial and market results, but also product

and service quality, public responsibility, employee development, and other key requirements. Accordingly, the Criteria provide a common language for businesses-large and small, manufacturing and service. A key characteristic of the Criteria is that the requirements are nonprescriptive-they do not require the use of specific tools and techniques. Users of the Criteria are asked to demonstrate success via well-defined processes and outstanding results. In this way, successful tools, techniques, and processes are identified and used as a basis for sharing.

Starting in 1998, the Baldrige Award was opened to health care and education organizations. In order to accommodate to these communities, while supporting cross-sector cooperation, the business Criteria were adapted and published for health care and education organizations [2,3]. The adaptation of the Criteria to health care and education is largely a translation of the requirements language and basic concepts of business excellence to parallel requirements in health care and education excellence. The common framework for all sectors of the economy has resulted in cross-sector cooperation and sharing of best practice information.

Although the Baldrige Award and the Award recipients are the very visible centerpiece of U.S. quality improvement, a broader national quality program has evolved around the Award and the Criteria. A 1995 report, Building on Baldrige: American Quality for the 21st Century, by the private Council on Competitiveness, said, “More than any other program, the Baldrige National Quality Award is responsible for making quality a national priority and disseminating best practices." [4].

"The Baldrige public/private partnership has accomplished more than any other program in revitalizing the American economy," said Barry Rogstad, president of the American Business Conference and past chairman of the Baldrige Program's Board of Overseers.

Following are some highlights of the program's first decade:

Called the "single most influential document in the modern history of American business," almost 2 million copies of the Baldrige Criteria for Performance Excellence [1] have been distributed. This number does not include many hundreds of thousands of copies available in books, from state and local award programs, or downloaded from the Award's World Wide Web site.

For the past six years, a hypothetical stock index made up of publicly-traded U.S. whole companies that have received the Baldrige Award has significantly outperformed the Standard & Poor's (S&P) 500 Index.

State and local quality programs-most modeled after the Baldrige program-have grown to over 50 in 1999. The national network of awards and sharing of practices involves several thousand volunteers each year.

• Internationally, 45 quality programs are in operation. Most are modeled after the Baldrige Award Criteria and evaluation approach, including a new award established in Japan in 1996.

Baldrige criteria and case studies are used extensively in U.S. business schools.

Prepared by Curt Reimann and Debbie Smyth.

Bibliography

[1] Application Guidelines, Malcolm Baldrige National Quality Award, 1988, Baldrige National Quality Program, National Institute of Standards and Technology, Gaithersburg, MD. [2] Health Care Criteria for Performance Excellence, 1998, Baldrige National Quality Program, National Institute of Standards and Technology, Gaithersburg, MD; most recent edition 2000. [3] Education Criteria for Performance Excellence, 1998, Baldrige National Quality Program, National Institute of Standards and Technology, Gaithersburg, MD; most recent edition 2000. [4] Building on Baldrige: American Quality for the 21st Century, Council on Competitiveness, Washington, DC (1995).

The Advanced Technology Program (ATP) is a competitive cost-sharing program in which the Federal Government works in partnership with industry to foster the development and broad dissemination of challenging, high-risk technologies that offer the potential for significant, broad-based economic benefits for the nation. This unique government-industry research partnership fosters the acceleration not only of dramatic gains in existing industries, but also of the development of emerging technologies leading to revolutionary new products, industrial processes and services for the world's markets while working to spawn industries of the 21st century. ATP's focus on civilian technologies offers the potential for substantial increases in productivity and competitiveness of firms; provides consumers with new, better, and lower-cost products and services; and increases high-wage employment in the United States.

From 1990 through 1999, the ATP made multi-year awards for a total of 468 projects, including over 157 joint ventures, involving more than 1,000 participants (not including the many subcontractors and informal partners and collaborators that participate in many of the projects). These projects entail approximately $3.0 billion of research, of which industry committed slightly more than half ($1.53 billion).

Since its inception, the Program's direct and indirect impacts have been substantial. In its brief existence, the ATP has helped shape the face of long-term enabling technology development undertaken by industry. It has also developed, by necessity, a wide variety of new tools for evaluating the impact of technology investment on the economy. Furthermore, it has often provided expertise at an important juncture-staff members, who are often leaders in their field, provide the critical insights to their peers necessary to assess how a certain technology has evolved and describe its potential future trajectories.

The ATP Proposal Preparation Kit [1] was the means by which the program has been introduced to the public. Its publication in 1990, as well as subsequent editions, spelled out the ATP's selection criteria and provided a manual for submitting a proposal. It was developed by the ATP specifically to help the public understand the program and addresses ATP's selection criteria. In the first year, nearly 1,000 copies of these guidelines were distributed to potential proposers; by the time the competition began for the year 2000, requests topped

50,000 copies. With such wide distribution, the kit serves many purposes: it informs the public of the nature of the program, answers questions about the program, and supplies all of the necessary information to complete an application, offering "one-stop shopping" for the potential proposal writer. From the outset, the Proposal Preparation Kit has undergone annual revisions to reflect both the needs of the public and the changes to the program.

The Proposal Preparation Kit has had widespread impact; to date, the rules contained within it have influenced the formation of almost 4,500 project teams. As such, it has changed the face of long-term technology development considered by industry. While the ATP has funded approximately 10% of these proposals, anecdotal evidence suggests that the kit was influential in fostering collaboration even when proposals were not funded.

A critical part of the ATP program management has been the development of methodologies and tools for assessment of impact and broad-based economic benefits. The ATP relies on the presence of expected private returns to induce companies to plan, propose, and cost-share research with the government. If the research is successful in overcoming its technical hurdles, the ATP relies on awardees to pursue commercial development of the new technology with private capital. The ATP applies its criteria to the proposals it receives to identify those projects expected to accomplish ATP's public-interest mission and objectives. Selected projects must demonstrate that they have the potential to generate social rates of return (the return to the nation) far in excess of the private rates of return on investment. In addition, selected projects must demonstrate that, without ATP funding, the private sector would be unwilling to do the research in either a timely fashion or a scale needed to realize the social benefit potential.

A study conducted by economist William F. Long [2], an ATP contractor, provides the most detailed examination to date of the outcomes of the earliest ATP projects. The study covers all 38 ATP projects completed by the end of March 1997. It documents research accomplishments and highlights subsequent work by the participants to commercialize the results. It also provides nearterm outlooks for the technologies. In the high-risk environment in which the ATP-funded projects operate, failure is to be expected. Thus the report also highlights the

reasons for failure of 12 terminated projects that had been selected between 1991 and March 1997.

The 38 projects surveyed by Long cover a broad range of ATP investments. The technologies were distributed over seven broad areas-chemicals and chemical processing; materials; discrete manufacturing; energy and environment; biotechnology; information, computers and communication; and electronics-with the majority in electronics. Industry participants provided $65.7 million, a little more than half the funding, while the ATP contributed $64.6 million to the 38 projects.

Notable new technologies highlighted by Long include a merger of tissue-engineering and textileweaving to help regenerate lost or damaged tissue in the body, an application of high-temperature superconductors to improve cellular phone service, and a suite of process-monitoring and control technologies that are cutting costs and improving quality throughout much of

the U.S. auto industry. Other new technologies developed under the projects ran the gamut from a desktop bioreactor capable of growing large amounts of human stem cells isolated from bone marrow for cell replacement therapy, a device now in clinical trials; to a computer programming tool to simplify the task of writing software for parallel-processing computers, which already is in commercial use; to a new navigation system for mobile robots which is being used to guide delivery robots in hospitals.

This analysis shows that technical success has not always led to commercial success. In a few cases, financial reverses or corporate takeovers left technically successful projects in abeyance. In other cases, commercialization activities are expected to take longer. For example, projects with medical applications generally had yet to be widely applied because they must go through a long process of clinical trials.

Other projects, however, have shown strong early returns. The process-monitoring and control technologies for the auto body industry, developed by a consortium of large and small companies, have been adopted in more than half of the Chrysler and General Motors plants in the United States and Canada. When the technologies are fully implemented, annual savings in production costs are expected to range from $65 million to $160 million for one of the most important U.S. industrial sectors.

A project pursued by one small start-up company developed a novel technology for processing very large semiconductor wafers, allowing the United States to be the first in the market with processing equipment for the next generation of 300 mm semiconductor wafers. A small New Jersey firm developed a new laser light source, which is the most powerful tunable source of laser light over much of the ultraviolet spectrum, and already has incorporated it in three new products for laser surgery and other applications.

The Long study is influential because it is among the first to demonstrate through a systematic assessment of the ATP portfolio the breath of the impact of the public investment undertaken by the ATP. In his introductory comments for the report, former Commerce Secretary William Daley characterized the report's influence by stating, "this new study fills in the details behind the statistical analyses of the ATP and demonstrates the overall success of the program. These pages say that the ATP is working. Industries as diverse as biotechnology, electronics, manufacturing and software have new technologies in place, today, that they wouldn't have had here and now without the ATP. The projected benefits to the nation's economy from just three of these early ATP projects would pay for every ATP project funded to date." Daley concluded by stating, "this report is a portrait of a program that works."

Since the ATP funds the research underlying the development of enabling technologies that are expected to have benefits extending substantially beyond the direct ATP award recipient, impact evaluation remains a challenge. Even under the best of circumstances, technology diffusion takes time, and tracking and measuring externalities, or spillover effects, is often complicated and difficult. As such, program and project assessment requires that the ATP go well beyond such traditional measures as return-on-investment, firm profitability, or increased tax revenues. Available evaluation tools often are insufficient to meet the task.

A special issue of the Journal of Technology Transfer [3] highlights a collection of ATP evaluation studies and illustrates the variety of evaluation issues ATP faces. The works in the publication are among the first to demonstrate the conceptual challenges ATP is faced

with and the wide variety of tools developed by the ATP. That publication features articles on methodology, the evaluation program, results from the path-breaking ATP "Business Reporting System" database, survey-based research, and a macroeconomic impact study. Other papers in the volume examine the effects of collaboration, special concepts that need to be considered in examining social benefits, and counterpart programs in the industrialized world.

In the Journal of Technology Transfer issue [3], Adam Jaffe, of Brandeis University, reflects on the importance to the ATP of generating and measuring "economic spillovers." It illustrates with simple models how the benefits of ATP projects may extend beyond the direct ATP award recipients through market and knowledge spillover effects. The paper also discusses network spillovers. Jaffe recommends that the ATP fund projects for which expected social benefits are large and substantially exceed expected benefits to the awardees. He recommends that evaluation efforts include measures of spillover effects. This paper is influential in that it affected the way in which the Advanced Technology Program, as well as others, undertakes economic analysis. Jaffe's paper codified the economic concepts used by the ATP and has served as a springboard for many other research projects underway.

Rosalie Ruegg the Director of the Economic Assessment Office of the ATP, provides an overview of the ATP, explains how the program operates, how it seeks to accomplish its mission, and what it has funded to date. She identifies the major components of ATP's evaluation program and identifies research areas of particular interest. The program is internationally recognized and has made a substantial impact in the evaluation community. In the United States, Congress has directed other agencies to model their evaluation programs on the ATP. Furthermore, state governments seek assistance from ATP in establishing their evaluation programs.

Jeanne Powell from the ATP presents and analyzes data from ATP's "Business Reporting System" database to evaluate short and medium term project effects. She describes ATP's principal data collection tool and identifies a number of pathways through which ATP-funded technologies are generating impact. The enabling nature of the funded technologies is suggested by the many potential applications that have thus far been identified. The survey-based database employed in this study is unique because it is among the first in the world that allows a researcher to explore the inner-workings of firms and technology projects at such a magnitude. As such, the tools used in this paper serve as the basis for future evaluation efforts.