DISCUSSION Mr. McKay (Los Alamos): I have two questions. It appeared that you were looking at one at a time perturbations, which means we are looking effectively in directions parallel to coordinate axis, is that correct? Dr. Schweppe: The numerical results that we had were one at a time, the periods very definitely not one at a time. It falls apart if it is not one at a time. Mr. McKay: So that in your fit, you didn't really worry about cross-product terms and in fact you probably did not include them in your response surface polynomial? Dr. Schweppe: What we did actually on this project is, we had a limited amount of data, when we finally thought of doing all of this stuff, and began to understand what it was all about and that stage was about over, we only had a one at a time perturbation at one block of matrices; we only had that in the computer. We were not able to fit cross terms. The quotation that I was giving on how much computer time it would cost, that was our best guess, involving many, many more runs that would include cross terms. You are right, the results we actually computed did not have the cross terms in th Mr. McKay: When you used your surrogate model, this polynomial, how did you study it? You said "pictures" and what not, but did you look just at plots and curves, or did you look actually at coefficients? Dr. Schweppe: We did not look at plots and curves, that is something I just added as something we should have done. It is always more probably what one shoul have done, not what one did. What we actually looked at is outputs of the mean and covariance of the output. You can compute from those coefficients the mean and the covariance of the output. So we had a covariance of the output. We had a 15 by 15 covariance matrix of the uncertainty in the output. That is the only thing we really looked at. Also the mean vector. That is the only thing we really looked at and to be honest with you towards the end of it we found more and more problems with what we were doing. We finally discovered the right way to do it and the only thing we ever looked at that was really precise was runs where we took some of the earlier runs which we didn't like and fixed them up by hand. it was valid, but we never looked at the night's computer printout that had all the correct covariance matrices in it. I am confident now that the technique does work but it was done by hand. And Mr. McKay: Did you just happen to look at the mean vector that you calculated from your fit and compare it to the mean of the data and the covariance matrix of your output variables to that that you derived from your fits? Dr. Schweppe: For this particular REM case, the mean of the deviation was very small compared to the covariances, i.e. the mean was small. It is hard to put a value judgment on what is small, but it looked small. 1. 2. 3. 7. REFERENCES Joskow P.L. and M.L. Baughman, "The Future of the U.S. Nuclear Energy Industry," Bell Journal of Economics, Vol.7, No. 1, 1976. MIT Model Assessment Group, "Independent Assessment of Energy Policy Models: Two Case Studies," MIT Energy Lab, Report MIT-EL 78-011, May 1978. Cecilia Sau Yen Wong, "New Approach in Parameter Sensitivity for Model Assessment," MIT Energy Laboratory Working Paper, June 21, 1978. A NEW APPROACH TO ANALYZE INFORMATION CONTAINED IN A MODEL Harvey J. Greenberg Office of Analysis Oversight and Access Energy Information Administration INTRODUCTION The purpose of this paper is to summarize the development of a new approach to address the general question: What information is contained in a model? For example, the equation, E=mc2, is a model that relates to two variables, energy (E) and mass (m), with a numerical constant (c2). The Energy Information Administration (EIA) is required to provide not only numerical data, but relations among data; not only historical measurements, but forecast estimates; not only basic projections, but impacts of proposed policies. Since energy information is complex, analysis is imperfect, and decisions are difficult, instructive use of energy information depends upon the accuracy, reliability, and credibility by which the information is recorded and interpreted. Furthermore, since the scope of energy analysis affects every person, industry, and environment, it is vital to apply engineering and economic skills not only artfully, but scientifically. The new approach proposes to account for relational and numerical information with a unified structure to record and analyze the information contained in a model. Questions of information contents may pertain not only to the explicit data that was recorded, but to implied relations. For example, suppose a model relates three processes: production, transportation, and consumption of oil. Their amounts may be related, for example, to associated prices at points of supply and demand. Figure 1 illustrates such a structure, where the constants, 1 and -1, and the parameters, C1, C2, C3, U1, U2, and U3, comprise the numerical data.1/ One may think of the Physical Flows Model as a linear program. The Supply and Demand rows then represent "material balances," and the columns represent three activities. The Cost and Capacity rows contain objective and bound values, respectively. The goal of the new approach is to be able to answer questions pertaining to a model's implicit, as well as explicit, information contents, for three forms of analysis: validation, verification, and assessment. A validation exercise may be concerned with comparing the accuracy of the model's information contents with evidence obtained from other sources, such as judgments from experts or indications provided by historical trends. Verification, on the other hand, deals with whether the model's information contents agree with the documentation. Assessment may be relative to other models that are designed to represent the same numerical information but with different relations. All three forms of analysis--validation, verification, and assessment--require answers to questions pertaining not only to the explicit information, but to the model's implied relations--that is, the implicit information contents. The new approach, which is described in a series of technical memoranda (see references), proposes a unified structure in two dimensions: the modeling framework and the form of analysis. To indicate the extent of the unification, the next section outlines the scope of the proposed approach. Then, an overview of the constructs that comprise the new approach is presented. Focus is on three related questions: How are relations defined?; How are they determined?; and, How are they measured? The conceptual approach, however, is only one of the prerequisites for success. A second issue is whether the proposal possesses sufficient rigor that it can be automated--that is, "Is it feasible to implement the approach?" We are especially interested in large, complex models, where the information is not readily apparent. The concluding section summarizes the proposed approach |