Page images
PDF
EPUB

regional impact analyses. The methodology does not capture the
regional shifts in economic activity driven by changing investment
patterns resulting from changing relative prices of energy or govern-
ment policy affecting the relative costs of doing business. Model
results also indicate this weakness. For this reason the model is
not usable for some of the most important policy questions facing
policymakers--diverse and uneven regional impacts.

Model Assessment Criteria

--There is a need to further clarify a taxonomy of evaluation criteria.
This exercise is a prerequisite to a common understanding of what
measures should be used to test the reliability of information from
large-scale models. This project has added to the understanding of
what is involved in third party model assessments and transfer.
Without DOE/EIA cooperation and many hours of personal communications
at the staff level, transfer of MREFS, though still incomplete,
could not have been accomplished. This fact reinforces our conclusion
that transferability must be more than a mechanical operation.

Continued Texas Work

--Texas should further develop its own modeling capability for assess-
ing the regional impacts of national energy policy proposals for input
into the decision making process; no other institutional entity can
be expected to adequately complete such analyses. Such a modeling
effort should be done taking full advantage of the results of TNEMP
experience.

DISCUSSION

Dr. Nissen (Chase Manhattan Bank): Before last December I was the proprietor of the PIE's Model and I built the demand model in PIES. I thought I would make two specific responses to what Milt Holloway said. I generally agree with the assessment that he has presented. I think the regional weaknesses are very much more important for Milt's purposes or PIES purposes now than they were for the model in its original guise. In fact, for the project Independence Blueprint version, which was the first version completed in 1974, the demand model operated at the national level. The totals were shared out to regions according to a set of fixed historical shares so that the demand model did not even depend on regional transient income and population, let alone have any feedbacks between the energy developments and the regional developments. So there has been an evolution towards regionality. It is generally unresponsive and incapable of dealing with regional energy economy interactions as it is currently configured. Also, generally it is incapable of handling, in any systematic way, national energy economy interactions. That is a very hard problem and I agree that it is an unsatisfactory state of affairs. There interactions have turned out to be very much more important, both at the regional, national, and world level-much more than anyone expected in 1974. If you look back and look at the sort of impact assessments about what was going to happen to the world economy, I agree with that.

I would like to make a technical response which I think has some functional importance to this group as to the way the add factors were used in comparing the use of add factors in the DRI Macro Model which drives the macro forecast for PIES. Add factors as they are used in PIES, and I think--let me check this--what you especially mean is in the way the demand model is reconfigured to handle various specifications. Is that what we are talking about?

Mr. Holloway: Yes.

Then

Dr. Nissen: Okay, add factoring is a precise term and it is a technical term in macro modeling. What you do is you have a set of equations that in general don't predict current history. So what you do is you change the intercepts so that they start at what we know is true now in the monthly exercise, if you are Otto Eckstein and you are running the DRI model. you either leave those in or you let them decay according to some things so that the representation of the model decays back to the data base that it was estimated on. That is an agnostic policy--a modeling process that simply recognizes that a good estimator of today's state of the world is today's state of the world, but there is no further analysis to that.

In the narrow sense, when we wanted to understand how the world looked with and without conservation programs, we did very much more than simply sort of wire today into the model. There was a collection of offline models that were part of an integrated community of policy assessment models used in the policy assessment activity which we carried out in '74 and '76 and then again in the summer of '77. The models I have in mind are the Residential Sector Model, where they are levers that represent policy actions and process standards and so forth and the Commercial Sector Model, a set of models operated by ICF and EIA that, respectively, estimated impacts of regulatory and pricing provisions. There was a documented analysis behind these shifts and there was a great deal of worrying about how the representation of these shifts within the consistent specifications of the demands models and the rest of the model.

So if you wanted to do what we always did functionally, that is recognize that these modeling efforts were part of the policy analysis system which was in play at that time, then they were integrated but not formally integrated in a code sense. And that is not add factoring. That is what you do when you have lots of big models which have lots of detail specificity-is that you construct many model representations in some sense that are representable between various pieces of code and you worry a lot about the internal consistency of those analyses. So that kind of add factoring, which is not add factoring but is noncomputerized model integration is, I think, always an essential part of any rich, sophisticated policy evaluation carried on inside the government. The modeling assessment process has to understand that and recognize what it is and its importance.

REFERENCES

Gass, Saul I., "Evaluation of Complex Models," published in Computers and Operations Research, Vol. 4, No. 1, Pergamon Press, Oxford, England, 1977.

Gass, Saul I., "A Procedure for the Evaluation of Complex Models," presented at the "First International Conference of Mathematical Modeling," August 29-September 1, 1977, St. Louis, Missouri, published in the proceedings of the conference.

Halter, A. N., "Strengths and Limitations of Models from a User's Viewpoint: Modeling the Modeler's Model," presented at the "Modeling Symposium: Building Modeling Bridges Over Information Gaps," Continuing Education Center, University of Houston, April 29, 1977.

Holloway, Milton L., Texas National Energy Modeling Project: Volume I Project Summary, Texas Energy Advisory Council, Austin, Texas, draft report, March, 1979.

The M.I.T. Model Assessment Group, "Independent Assessment of Energy Policy Models: Two Case Studies," Energy Laboratory in cooperation with The Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, draft report, May 15, 1978.

Selvidge, Judith, "Panel Discussion - Management Audit of Quantitative Models," Northeast AIDS Conference, Washington, D.C., June 1-2, 1978.

[blocks in formation]
[blocks in formation]

Previous studies sponsored by the Department of Defense, The National Science Foundation, The Government Accounting Office, and the National Bureau of Standards have raised questions concerning the utility of computer based models developed for use by agencies of the Federal Government [4], [5], [6], [10]. These reports and others contain ideas and suggestions for improving the development, management and maintenance of a model during its life cycle. Based on our analysis, we identified eighteen broad areas by which current modeling improvement ideas can be grouped. These areas are as follows:..

1.

2.

Data collection and availability for model improvement
Standardized procedures for model developers

3. Model user training

4. Model documentation plan and guidelines Definition of large-scale models

5.

6.

8.

RFP statement of work for model development
Model verification and validation plan

Relationship between model user and developer

9. Phased management approach to model development Government in-house model development

10.

11. Model post-review panel

12.

13.

14.

15.

16.

17.

Model ongoing review panel

Upgrading of the Government contract officer's technical
representatives (COTR's)

Financial and milestone model management review techniques
Central model clearinghouse

Government model testing, verification and validation center
Government modeling research center

18. Modeling forums of users and developers

1/This paper is based on the report "A Study for Assessing

Ways to Improve the Utility of Large-Scale Models," S.I. Gass,
Z.F. Lansdowne, R.P. Harvey, and A.J. Lemoine, National
Bureau of Standards, December 1978. Due to time constraints,
this paper was not presented at the Workshop.

Up to the time of our study, there had been few attempts to determine (a) which specific model improvement proposals would work and be accepted by the modeling community, (b) what modeling research activities should be supported, and (c) if the Government was to support such research, what priorities should be established? This study was an attempt to answer these questions.

Thus,

It is estimated that about 80% of the models developed for non-DOD agencies are constructed by organizations external to the Government; the figure for DOD models is 55% [4], [5]. it is clear that any attempt to impose ideas or guidelines or standards for improving the utility of Government modeling must take into account the concerns and interests of the modeling community at large. This community includes Government sponsors, users and model builders; private contractors; non-profit organizations; and university researchers and grantees.

STUDY METHODOLOGY

1.

The basic problem was "How to Improve the Utility of Large-Scale Computer-Based Models?" We determined why this a problem, who was involved and prepared a problem

description paper.

2. We next defined the eighteen model improvement areas (as previously listed). For these areas we identified alternatives open to the Government in its desire to improve model utility. These alternative were generated from other studies and surveys, and from discussions with model users and developers.

3. For each area we developed a "model improvement proposition." We then discussed each proposition in terms of arguments for and arguments against. For example, one proposal was "for the Government to develop suitable standards for model development and to require the developer to conform to these standards whenever appropriate. Another proposition was "to require the model developer to prepare a verification and validation test plan and to report the results of the test plan in a technical report." While a third proposition was "as a means of improving the development and utilization of models, this proposal is to establish a central model clearinghouse.

4. For each proposition we developed related statements that highlighted and specified certain aspects of the proposition. For example, the proposition that dealt with standardization had the following three statements:

A joint Government/industry committee should be established to investigate what aspects of the Government's modeling projects can be subjected to some form of standardization. The committee report would include a statement as to the costs and benefits to be obtained by standardization of particular modeling elements.

« PreviousContinue »