14.

C.R. Deeter and A.A.J. Hoffman, (eds), "Validation of
Mathematical Models of Energy Related Research and Development,"
Proceedings of an NSF Sponsored Conference held in Dallas, Texas,
June, 1978.

15. General Accounting Office, "Review of the 1974 Project Independence Evaluation System," report to the Congress by the Comptroller General of The United States, 1975.

16. General Accounting Office, "Guidelines for Model Evaluation," U.S. General Accounting Office, Washington, D.C. (PAD-79-17), January, 1979.

17. Energy Information Administration, "Annual Report to Congress, 1978," (DOE/EIA-0173) in 3 volumes, 1979.

18. Saul Gass (editor), "Utility and Use of Large-scale Mathematical Models," proceedings of a workshop held at the National Bureau (NBS Special Publication 534), May, 1979.

20. Lester D. Taylor, "The Demand for Electricity: A Survey," The Bell Journal of Economics. Volume 6, number 1, (Spring), 1975, pp. 74-110.

21.

Robert S. Pindyck, "The Regulatory Implication of Three Alternative
Econometric Supply Models of Natural Gas," The Bell Journal of
Economics and Management Science. Volume 5, number 2 (Autumn),
1974, EP 633-45.

22. Brian L. Crissey, "A Rational Framework for the Use of Computer Simulation Models in a Policy Context," unpublished PhD. dissertation, The Johns-Hopkins University, 1975.

23. David Kresge, "Third Party Model Assessment," (this proceedings).

24. MIT Model Assessment Group, "Independent Assessment of Energy Policy Models," Electric Power Research Institute, Palo Alto, California, (EA-1071), May, 1979.

25. Raymond Hartman, "Frontiers in Energy Demand Modeling," Annual Review of Energy, Volume 4, 1979.

26. Kenneth C. Hoffman and David O. Wood, "Energy System Modelling and Forecasting," Annual Review of Energy, Volume 1, 1976.

27. Jerry Hausman, "Project Independence Report: An Appraisal of US Energy Needs up to 1985," The Bell Journal of Economics and Management, August, 1975, BP.517-51.

28. David Wood and Jerry Hausman, "Energy Demand in the ERDA National R&D Plans, in An Analysis of the ERDA Plan & Program, Office of Technology Assessment, U.S. Congress, 1975.

29. John A. Neri, "An Evaluation of Two Alternative Supply Models of Natural Gas," The Bell Journal of Economics, Vol. 6, No. 2 (Autumn) 1975.

30. Systems Control, Inc., "Applicability of Brookhaven National Laboratories Energy Models to Electric Utility R & D Planning, Electric Power Research Institute, Palo Alto, California, (EÖ807), June, 1978.

31. Charles River Associates, Inc., "Long-range Forecasting Properties of State-of-the-art Models of Demand for Electric Energy," Electric Power Research Institute, Palo Alto, California, (EA-221), December, 1976.

Richard L. Gordon, "Economic Analysis of Coal Supply: An Assessment of Existing Studies," Electric Power Research Institute, Palo Alto, California, (EA-496), July, 1977.

33. Milton Holloway, "The Texas National Energy Modelling Project: An Evaluation of EIA's Energy Midrange Forecasting Model," (this proceedings).

34. Saul Gass, "Evaluation of Complex Models," Comput and Ops Res, Volume 4, 1977, pp. 27-35.

35. Federal Energy Administration, National Energy Outlook,

(FEA-N-75-713) February 1976.
Washington, D.C., 1976.

Government Printing Office,

36. Peter House and John McLeod, Large-scale Models for Policy Evaluation, John Wiley & Sons, 1977.

37.

Energy Research & Development Administration, "A National Plan for
Energy Research Development and Demonstration: Creating Energy
Choices for the Future," (ERDA 76-1) U.S. Government Printing
Office, Washington, D.C., 1976.

38. Charles River Associates, Inc., "Review of Large-scale Energy Models," Electric Power Research Institute, Palo Alto, California, (EA-968), January, 1979.

39. Mathematica, Inc., "The Comparative State-of-the-Art Assessment of Gas Supply Modelling," Electric Power Research Institute, Palo Alto, California, (EA-201), February, 1977.

40. Optimal Analysis Company Co., "A Critique of the Federal Energy Administration's Energy Model," report prepared for the House Subcommittee on Energy and Policy, July, 1976.

41. William W. Hogan, "Energy Modelling: Building Understanding for Better Use," presented at The Second Berkeley Symposium on the Systems and Decisions Sciences, Berkeley, California, October, 1978.

42. Saul Gass, "Computer Model Documentation: A Review and an Approach," (NBS special publication 500-39), National Bureau of Standards, Washington, D.C., February, 1979.

43. National Bureau of Standards, "Guidelines for Documentation of Computer Programs and Automated Data Systems," FIPS PUB 38, Washington, D.C., February, 1976.

44. Stanford Institute for Energy Studies, "Stanford-EPRI Workshop for Considering a Forum for the Analysis of Energy Options Through the Use of Models," Electric Power Research Institute, Palo Alto, California (EA-414-SR) May, 1977.

45. James Sweeney, "The Energy Modelling Forum," (this proceedings). 46. Neil L. Goldman, and James Gruhl, "Assessing the ICF Coal and Electric Utilities Model," (this proceedings).

47. James Gruhl and David O. Wood, "Independent Assessment of Complex Models," proceedings of a workshop Validation of Energy Related Mathematical Models, C.R. Deeter and A.A.J. Hoffman (eds.) Forthcoming.

48. Walter Short, John Rambo, and Robert Fuller, "Historical Simulation with the Livermore Economic Modeling System," Laurence Livermore Laboratory, Livermore, CA.

Forthcoming.

49. Andrew Ford, Glenn H. Moore, Michael D. McKay, "Sensitivity Analysis of Large Computer Models: A Case Study of the COAL2 National Energy Model," Los Alamos Scientific Laboratory, Los Alamos, New Mexico (LA-7772-MS), April, 1979.

50. Albert J. Eckstein and Dale M. Heien, "A Review of Energy Models with Particular Reference to Employment in Manpower Analysis," Report for the Employment and Training Administration, U.S. Department of Labor, Washington, D.C., March, 1978.

51. Data Resources, Inc., "The Residential Demand for Electricity," Electric Power Research Institute, Palo Alto, California, (EL-235, Volumes 1 and 2), January, 1977.

52. Paul Joskow and Martin Baughman, "The Future of The U.S. Nuclear Energy Industry," Bell Journal, Vol. 7, No. 1, 1976.

53. Professional Audit Review Team, "Activities of the Office of Energy Information and Analysis," Report to the President and the Congress, December 5, 1977.

54.

Michael Shaw, John Farquhar, and S. Michael Lutz, "Management and
Implementation of PIES Access," and "Recommendations for PIES
Access," a report prepared for the Energy Information Administration
by Logistics Management Institute, Washington, D.C., March 1978.

55. Michael S. Mac Pakis (ed), Energy: Demand, Conservation and Institutional Problems, The M.I.T. Press, Cambridge, Massachusetts,

1974.

56. Paul L. Joskow and Martin L. Baughman, "The Future of the U.S. Nuclear Energy Industry," Bell Journal of Economics. Volume 7, number 1, 1976.

57. Edward A. Hudson and Dale W. Jorgensen, "U.S. Energy Policy and Economic Growth, 1975-2000," The Bell Journal of Economics and Management Science, Volume 5, No. 2, (Autumn), 1974.

58.

MIT Policy Study Group, Energy Self-Sufficiency: An Economic
Evaluation, American Enterprise Institute for Public Policy
Research, Washington, D.C., 1974.

59. Nuclear Energy Policy Study Group, Nuclear Power: Issues and
Choices, Ballinger Publishing Company, Cambridge, MA, 1977.

60. Energy Policy Project of the Ford Foundation, A Time to Choose: America's Energy Future, Ballinger Publishing Company, Cambridge, MA, 1974.

DISCUSSANT COMMENTS

William W. Hogan

John F. Kennedy School of Government

Harvard University

Cambridge, Massachusetts 02138

I endorse the purposes of this workshop, and I am impressed with the
breadth and quality of the speakers that Saul Gass has assembled.
a discussant, I find myself in the happy position of agreeing with the
general tenor and substance of the remarks of the speakers who have
preceded me. My comments, therefore, are primarily in the form of a
few observations and suggestions as to the implications of the chief
points of the previous speakers.

The common theme so far is the development of taxonomies of the problems and approaches to the assessment of models. George Lady described the distinctions among the concepts of model verification, validation, and ventilation, emphasizing the critical importance of understanding the objectives of any examination of a model and its uses. Dave Wood expanded this taxonomy to distinguish between the types of models: policy research versus policy analysis models. It is clear that models and model uses are heterogeneous, and this diversity must be recognized in the design and use of the model assessments. At one end of the spectrum, we have models constructed to examine the state of nature and to test hypotheses, and the parallel model assessments are designed for the benefit of scientists and model builders. At the other extreme we have models constructed to deal with the messy policy problems immediately at hand, and the evaluation structure and standards must be adapted to the model user and the decision maker.

This mapping of the terrain is a primitive but crucial step for the development of a model assessment process. And it points to a number of immediate implications:

1. The process is expensive. Assessing a model and its uses can be an activity on the same scale or larger than the effort of constructing the model in the first place. If this essential component of an analytical process is to develop and mature, then it must be funded on a scale commensurate with its benefits and its difficulty. The sponsors to date have been generous in their support of the early phases of model assessment processes, but it is clear that the actual costs of model assessments have greatly exceeded our earlier expectations. It is essential that future model assessment efforts be put on a stable and professional basis.

2.

We should not rush too far ahead, to the setting of standards, too fast. It is clear that we do not know much about the model assessment process, and a great deal more experimentation and research will be necessary before we can approach the panacea of validated models that can be certified for third-party use. The National Bureau of Standards is to be complimented for organizing this workshop, but they should not be misled to thinking they are near the establishment of modeling standards.