3. Upper Bound Scrubber Capacity: 106 GW (Base Cases, Unbounded), 100 GW 4. Lower Bound Appalachia Coal Production: 400 Million Tons (1990) with Scrubbers not Mandatory and with Stricter Sulfur Standard than NSPS 5. Mine Lifetime (Years): 30 (Base Cases), 20, 40 6. Electricity Demand Growth Rate (1985-1995): 4.0% (Base Cases), 5.0%, 3.0% 7. Nuclear and Hydro Capacity Additions: 25% Decrease from a Base Case 8. Non-Utility Coal Demand: 10% Increase over a Base Case 9. Oil/Gas Prices: 25% Increase over a Base Case IX. Acknowledgments The authors performed this research at the M.I.T. Energy Laboratory as members of the Model Assessment Group. The principal investigator is Mr. David O. Wood, Associate Director of the Laboratory. Prof. Michael Manove contributed to several portions of this paper. Other members of the Model Assessment Group include Prof. Fred Schweppe, Dr. Ingo Vogelsang, and Mr. Vijaya Chandru. The principal sponsor of this project has been the Electric Power Research Institute, Dr. Richard Richels, program manager. Some additional support has been contributed by the U.S. Department of Energy, Dr. George Lady, program manager. X. References 1. Coal Supply Analysis, Prepared for the Federal Energy Administration by ICF, Inc., May 1976. 2. 3. 4. 5. 6. 7. 8. 9. 10. Review of Federal Energy Administration National Energy Outlook, 1976, Economic Analysis of Coal Supply: An Assessment of Existing Studies, The National Coal Model: Description and Documentation, Prepared for the Coal and Electric Utilities Model Documentation, ICF, Inc., July 1977. Coal in Transition: 1980-2000, Energy Modeling Forum, EMF Report 2, Effects of Alternative New Source Performance Standards for Coal-Fired The Demand of Western Coal and its Sensitivity to Key Uncertainties, Further Analysis of Alternative New Source Performance Standards for New The ICF Coal and Electric Utilities Model: An Overview Assessment, DEVELOPING, IMPROVING AND ASSESSING C. Hoff Stauffer, Jr. ICF Incorporated When I sat down to sketch out some notes for this talk, I wrote at the top of the paper, "Model Validation and Assessment." I didn't notice until today that my talk was supposed to deal with the ICF Coal and Electric Utilities model. Fortunately, one of my conclusions was that the models ought to be validated and assessed using the same process that we actually used to develop the ICF model, and that we use continually to validate it and assess it for our own purposes. In other words, I believe the model development and model assessment should employ the same analytic steps. I should make it clear that I view myself as a model user, not a model developer or a modeler. Similarly, I have a limited experience with models. The only models with which I have dealt have been structural, not econometric. I think many of my comments would not apply to econometric models. Definition of Assessment and Validation When I began to organize my thoughts on this topic, I realized I didn't really know what was meant by "assessment and validation." After listening to the talks so far today, I still don't know what others mean when they use these and related terms. However, I developed an operating definition for myself so that at least I know what I mean. Put most simply, I decided "assessment and validation" must have to do with whether the model gets the right answer. Then I note there are two parts to that. One part is whether the model measures relative changes correctly. The federal government is generally concerned with relative changes. The second part is whether the model measures absolute levels correctly. The private sector is generally most concerned with absolute levels. Finally, I note there is another important dimension as well. One must determine for what kinds of questions the model gets the right answer, and for what kinds of questions it does not get the right answer. Use of Historical Data for Validation Now, if our definition of model validation is whether or not it gets the right answer, how can we determine whether it is capable of this? First of all, we cannot use historical data. Most structural models involve investment decisions, and these involve lead times and expectations. The expectations being modeled were never written down. As an example, if we want to predict a decision someone would have made in 1965 regarding a powerplant which would come on in 1975, the last thing I would want to use would be actual historical oil prices. I would want to use the oil prices which the decision-maker thought were going to exist. But there is no way to know that precisely. So we discover that "backcasting" is no easier than forecasting. In backcasting and forecasting, you need to assume expectations. There is no comprehensive data source for expectations. So I think it is clear that backcasting is not a useful approach to model validation. Model Comparisons Another way to determine whether a model is capable of getting the "right" answer is to compare its answers to those of other models. But this seems to me to have limited utility. If we wanted to decide things by consensus, we could ask people to vote. More importantly, if the answers are different, what does that prove? Unless model comparisons proceed to the seven steps I outline below, I think it is clear that such superficial comparison exercises--where only outputs are compared--are effete endeavors. Develop Confidence In Model These two deadends--backcasting and model comparisons--lead me to a third approach. This approach is not simple nor quick, nor is it mindless as the first two approaches. This approach requires intellectual capital, time, and hard work. This approach is to do the analysis required to develop confidence in the model. This in turn requires an in-depth understanding first of the phenomenon being modeled, the issues the model is designed to address (or the question it is designed to answer), and the dynamics of how the issue areas affect or are affected by the phenomenon being modeled. It also requires an in-depth understanding of the structure of the model, its data, and its assumptions. Finally, it requires experience with the model and careful analysis of its fore casts. The process we used to develop our model and refine it, and the process which, in my opinion, people should use to assess and validate other models, has seven steps. Step 1: Understand Phenomenon to Be Modeled The first step would start off by understanding, in great detail, the phenomenon that the model is trying to deal with. In the case of the ICF Coal and Electric Utilities model, these phenomena are the coal and electric utility industries. The richness of detail necessary for a complete understanding of the subject in this case begins with a knowledge of coal reserves: the quantity, quality, and physical characteristics thereof, all by geographic region. And, by the way, each of those dimensions has several sub-dimensions. Similarly, you have to understand mining; you have to understand technology and cost and environmental regulations. You have to understand the economics of mining and how a producer views an investment to open a mine. You have to, and I'm going to come back to this, understand what is meant by price. There are many definitions of what seems to be a simple number price. Only one definition is correct for any one use of it. Then you have to know about coal preparation, the effect it has on coal quality, the cost thereof, and the tradeoffs. You have to understand that coal transportation, its various modes and their costs vary by geographic region. You You have to understand differences between consuming sectors: utility, industrial, metallurgical, export, and so forth. Within a sector, you have to understand the combustion trade-offs, particularly in the utility sector. have to understand how electric utilities dispatch their capacity to the daily load curve. You have to understand how they plan capacity expansion. You have, therefore to, understand power plant costs: both capital and operating. Very importantly, you have to understand environmental regulations, and the cost of complying with them. You have to understand transmission. You have to understand the difference between the long-term dispatch kind and the short-term dispatch kind of models, and the effects each would have on the kind of decisions you'll get. You have to understand finance very well. Financial considerations have overwhelming influences. You have to understand the nature of the coal markets. By that I mean, in this case, there is a long term contract market and there is a spot market; there is very little in between. The market varies by geographic region. It varies by type of coal and by sector. Even the kind of firms that produce coal for those different markets are different. |