Appendix ! Participants, Funding, and Other Details "The State and Local Outreach Program was primarily intended to help lay a foundation for greenhouse gas emission reductions beyond 2000, not to achieve greenhouse gas reductions by 2000 However, according to EPA. the program did achieve substantial reductions through 1996 and is expected to achieve even greater reductions in 2000. "Represents number of projects. "Data for the Source Reduction and Recycling Program are for fiscal year 1995. *Represents "grass" reductions. "Net" reductions are estimated to be about 40 percent of the "gross reductions-1.1 MMTCE in 1996 and 2.6 MMTCE in 2000. See comment 1 See comment 2. Mr. Peter F. Guerrero Director Environmental Protection Issues U.S. General Accounting Office Dear Mr. Guerrero: OFFICE OF AIR AND RADIATION I appreciate the opportunity to review and comment upon the draft GAO Report, Information on Results of Four EPA Yoluntary Climate Change Programs. My first comment is that the initial scope of your review, as presented in a Memorandum from GAO to EPA (August 27, 1996), was a much broader review of the Climate Change Action Plan (CCAP) programs than what you refer to in the Report. This original scope included first a determination of "the types of performance measures EPA has developed for CCAP programs." It is therefore disappointing that the draft Report fails to mention EPA's significant accomplishments in measuring, evaluating, and reporting on the progress of CCAP programs. EPA has developed a successful and extensive system of performance measures and program evaluation. EPA devotes considerable effort to obtaining the best possible information upon which to evaluate the programs. For example, EPA reports the results of the Green Lights program based exclusively on detailed reports submitted by the program's partners on over 14.000 completed projects around the country. These efforts and the efforts of other programs have provided maximum accountability and valuable information for program development. EPA's performance measures have been reviewed in detail by your staff and are largely the basis for GAO's Report. I would like to draw your attention to a recent report by the EPA Office of the Inspector General (DIG). The OIG recently completed a review of some of EPA's important CCAP programs (Risk Reduction Through Voluntary Programs, Audit Report No. EIKAF6-05-00807100130, 3/19/97). The OIG found that the programs "effectively estimated the impact their activities had on reducing risks to health and the environment,” and that the programs "used good management practices," including good planning, progress evaluation, and program adjustment. The report concluded that "future voluntary programs could benefit from using similar measurement techniques." The revised, narrow focus presented in GAO's draft Report does not sufficiently recognize the high standard of accountability that EPA uses in evaluating and reporting on its CCAP programs. Appendix II Comments From the Environmental See comment 3 While the GAO draft Report does include important issues regarding measuring program success, they are ones that are difficult for all market transformation efforts. EPA has alway! recognized that there are many difficult analytical issues in evaluating the success of voluntary, market-based programs, and has therefore conservatively estimated the impacts of the CCAP programs. EPA has openly discussed these issues with your staff. There is some uncertainty, for example, in isolating the effects of a program such as Green Lights from other factors within the market. This uncertainty can work in either direction - leading programs to overestimate or underestimate results -- depending on the measurement techniques used. In order to address this uncertainty, EPA has either adjusted a program's numbers or chosen methodologies that would likely underestimate the net impact of a program. For two of the four programs examined, GAO points to the absence of specific “adjustments” as, in itself, a significant conclusion. For some programs, however, EPA has instead decided to use a generally conservative approach rather than make arbitrary "adjustments" where sufficient data is not available. GAO should recognize in the final Report that there are different means of handling uncertainty, that EPA has addressed these issues in a reasonable manner, and that EPA does not overstate its program accomplishments. For example, the Green Lights program's reported accomplishments likely significantly underestimates the actual accomplishments for a number of reasons. The Green Lights program is an informational program that generates broad awareness and provides technical informational to everyone who is willing to use it, regardless of whether or not they join the program. EPA monitors the program's performance based exclusively on completed projects reported by those who join the program and fill out annual reports. EPA believes that this methodology is highly conservative. Although a majority of lighting technologies purchased today for buildings remain the least efficient technologies, there has nevertheless been substantial improvement in the market share of the more efficient technologies promoted by Green Lights since the program began in 1991 (based on U.S. Census manufacturing and sales data that we have shared with your staff). EPA is reporting only a fraction of this larger market improvement as being attributable to the accomplishments of the Green Lights program. The true program impact of the Green Lights program is likely much larger than what EPA has been reporting to date, and EPA intends to study improved means of measuring this impact. We have provided your staff with evidence to support the many reasons that the Green Lights program estimates are conservative. The main reasons are summarized as follows: (1) (2) The impacts of Green Lights' efforts to generate awareness of cost-effective investment opportunities for energy efficiency are widely dispersed, with only a portion of those who make such investments joining the program. EPA widely distributes its important technical information on lighting. A large number of people who attend the Green Lights' lighting upgrade workshops, for example, do not belong to the program. Appendix II Comments From the Environmental See comment 4 See comment 5. (3) (4) Because partner reports are submitted once a year, there is up to a one-year time lag in measuring program performance. This is especially significant because the program's accomplishments are now rapidly accelerating; 40 percent of the program's current accomplishments have been generated by upgrades that were reported within the last year, despite the fact that the program is 6 years old. This alone suggests that truc program impact is underestimated by 20% or more because of the reporting delay. Not all partners complete and submit reports once they've completed lighting upgrades, resulting in under-reporting of partners' true accomplishments. EPA is studying alternative methods for information collection. The GAO Report raises some of the many important issues regarding measuring program results that EPA attempts to address in evaluating its CCAP programs. As the programs' market impact increases and better information becomes available, we intend to better isolate the broader market impact of the programs, rather than relying exclusively on techniques such as measuring direct program participation. EPA intends to study the issue further this coming year. EPA does not believe that asking a few partners retrospectively whether or not they would have completed the upgrade is an appropriate means of completing a study. The intent of the Green Lights and other CCAP programs is to generate awareness and provide the support and technical information needed to allow partners to invest in profitable energy efficiency. After realizing extremely high returns on their investments, while improving the quality of their lighting, it is not surprising that partners' hindsight includes "20/20" vision. We view this as a major accomplishment -- making energy efficiency investments part of the normal business practices is the ultimate measure of program success. However, we know from experience, and from the continued inefficient practices of a majority of businesses today, that getting partners to devote their capital to non-traditional investments, such as facility energy, is anything but normal business practice. This is also widely documented outside of our own program experience (including the Office of Technology Assessment study referenced in the GAO Report). Although GAO has not shared with us the names of the companies that it finally interviewed, I encourage you to discuss with my staff that supports these partners the considerable efforts it took to turn each of those partners into a success story. With regard to estimating the future impact of the programs, EPA does assume that, for some programs, current and future partners will do better than initial partners in the program. For example, the Green Lights program expects that partners who have recently joined the program will do better in meeting their full commitments than the partners that joined in the first year (i.e., the partners that have been in the program for the full five years of the commitment). As you acknowledge in the Report, EPA has demonstrated that this improvement is already occurring. After two years in the program, for example, partners that joined ion 1995 have done considerably better than the first year's partners, achieving four times the energy and pollution reductions (despite smaller commitments). This success has improved steadily since the beginning of the program, and current partners are well ahead of the program's targets. This is in part due to changes made in the program to improve partner support. EPA is pleased that the 3 |