PANEL SUMMATION

The final event of the Workshop was a panel summation that addressed the future of model assessment. The panelists were Martin Greenberger, William Hogan, George Lady, David Nissen, Richard Richels, and David Wood. The following is an edited version of the discussion.

DR. WOOD: I would have a few summary comments. First, with all of you, I have quite enjoyed the last two days and Saul's organization of this Workshop. Some things that I already knew have been reaffirmed. For example, whenever you organize an activity such as this, you must be sure to invite David Nissen and Larry Mayer to keep it lively, as well as informative.

Some issues discussed during the Workshop seemed especially provocative to me. John Weyant's discussion of model assessment versus the Forum, substitutes or complements, helped me to focus more clearly on what the differences between those two enterprises might be, and why in the future these apparently separate activities are likely to merge.

I thought Bud Cherry's observation yesterday that much of what goes on in the Forum process is not reportable in the traditional documentation and, in fact, internalized in the alumnae of that process, is suggestive for further organizational initiative. If the specific product of the Forum is policy research studies, then the value added is an alumnae sensitized to the uses of policy models in policy research.

Finally, I want to emphasize a view I have expressed several times during this Workshop. Policy model evaluation seems to me to be scientific analysis and peer review organized and presented to provide for the information requirements of all groups--not just modelers--involved in the policy research process. What may appear new or distinctive is the effort to satisfy non-modeler needs for information about models, their scientific validity and applicability to particular policy issues. As this evaluative aspect of the policy sciences matures, the apparent distinctions between scientific analysis and review and policy model evaluation will evolve into good scientific practice with whatever particular methods and practices are appropriate to satisfy the information needs of non-modelers involved in the policy process.

DR. GASS: Thank you Dave. Next, Martin Greenberger.

DR. GREENBERGER: My remarks can be very brief because I spoke a little bit earlier today. I would like to join with Dave in congratulating Saul on bringing together an excellent group of people. The National Bureau of Standards and the Department of Energy deserve our thanks for sponsoring this very constructive Workshop.

A member of the Workshop told me he felt the future of model assessment hinged on whether modeling is more like writing or physics. What he meant was that if it is like writing, then just as writing is going to continue to have its critics for a long time, so modeling will continue to have its model analysts.

But in physics, he did not see the analogy. I pointed out to him that there is in face a parallel in physics. There is the theoretical physicist, who corresponds to the modeler, and then there is the experimental physicist--the fellow in the laboratory testing the theories--who corresponds to the model analyst. In physics, many more theories are proposed than survive because of the efforts of the experimentalists.

The question is, can you have the same person serving both functions? In physics, the answer is generally no. Theoretical and experimental physicists tend to be different people. They work cooperatively, if not always harmoniously, and there is a very productive symbiosis between them. It seems to me that it is entirely possible for the same kind of symbiosis to develop as between model developers and model analysts for the benefit of the policy-makers and model users.

I am very much encouraged by what has taken place over the past four years. The field of policy modeling four years ago revealed a very clear deficiency which it has begun to fill with the development of model analysis. There is still a way to go, but now we are not asking the question, "What can we do?" but "What form will it take?" That is encouraging.

DR. GASS: Thank you Martin. Next, Bill Hogan.

DR. HOGAN: Well, the first thing I would like to do is to ask Larry Mayer if he has any plans for dinner? I don't have to catch a plane until 8 o'clock, so we can see if we can arrange something later on. This is to disprove his statements about his ostracism. So I told him privately his remarks are very provocative, but although I think almost everything he said as a factual matter is correct, the general thrust of his comments was not in keeping with what I viewed as the pleasant progress of the discussions of this workshop. That progress has been in laying out the heterogenous nature of models in use and the different kinds of applications and the different kinds of uses.

I won't repeat all of the taxonomies that were proposed. There were many and they are developing. If I may use an analogy we are trying to make the distinction here between the processes that physicists are involved in vs. the processes that lawyers are involved in. This is the distinction between policy research and policy analysis models. We are concerned about people using the rules of the physicist in a lawyer's game or using the practices of lawyers to claim the benefits of physics. We don't want to confuse these two activities.

I did ask Larry about this and he agreed. I think making the distinction is an important component of what we are doing.

That also leads me to another dimension. As we are developing a more sophisticated view of what we mean by modeling, the modeling process and modeling assessments, we want to make sure that we keep our standards in mind, particularly when we are talking about the end of the spectrum that I refer to as the lawyer's view, i.e., the policy analysis process. I don't know if that analogy is going to hold up, but it certainly is true that the absolute standards that we might appeal to for the scientific evaluation are not going to be relevant for the lawyer.

I was happy to see that a lot of the discussion, at least formally (for example, in the paper by Fred Murphy and Harvey Greenberg), was about validity as a relative concept: this model is more valid than that model; this component is more valid than that component. For the policy analysis purpose, I think that is the only useful piece of information. If you are confronted with a situation where you must make a decision, you want to know how useful the model might be compared with something else that you might have to use in its absence.

The implications of this are many, but I think the most important one in the short run is that we are not very close to being able to specify standards for modeling or model assessments; standards in the rigid sense of being able to give grades and to have necessary conditions, and so forth, for the use of some models. At least this is true in the spectrum of policy analysis. We should continue the kind of work that obviously has been indicated in the discussion here.

In terms of my preferences, I don't know what the future is going to be like, but my preferences for the future would be that there be much more creative energies in trying to understand the end of the spectrum concerned with the use of models, how decisions are influenced by analysis, no matter how formal that analysis may tend to be. That is going to get us into areas of behavioral research. There is some work going on in that. Not all people think about problems the same way, so it is probably true that not all models are going to influence all people the same way and we ought to develop a better understanding of these differences.

The discipline of empirical tests is useful, not only at the scientific end, but also at the policy analysis end. It is an uncomfortable discipline. I endorse the view that we should try to test our models, our concepts, and our ideas at every opportunity. Sometimes those opportunities may be difficult to create, but that is an important element that should be continued.

And then, if we talk about the models and the model use process, we ought to have more information from our own experiences, not from the side of decision makers, but from the side of the modelers about how models are actually used. I would like to see more papers written on the applications of good models and how they are, in fact, applied. I have a

particular interest in this--I recently assumed an area editorship for Operations Research on energy and environmental problems, and I would be very anxious to get good papers on the applications of models in discussing empirical tests, how the models are used, what contributions have been made, and so forth. We need to publicize that kind of information in addition to the theoretical descriptions of models that we find so much easier to write and much easier to criticize.

MR. LADY: I really will be brief as I view my role here more as a customer than as a contributor and, as a customer, I want to first say I feel I have been well served, thank Saul Gass for his efforts, and thank everyone else. I thought it was a very good program. It is very ambitious to have a program that lasted this long, for two days in a row, and still have this many people here.

MR. RICHELS: I think they are all getting rides back home with the people that stayed!

MR. LADY: Anyway, Anyway, I stayed! So, I will be brief, but let me mention a few things that I think are important.

First, I liked Martin's chart that organized the who and the why part of assessments. I am impressed and believe, and others have said it different ways, that the major first order impact of the process that we are in will be felt in the upper left hand corner of that particular organization. That is, what we are really doing is we are in the process of changing what "modelers normally do" and, in the end, a lot of the so-called third-party activities and associations, with the ideas we have been talking about, will disappear. In the end, I think that the modelers will behave differently than they have so far due to the process that we are now experiencing.

In my own mind, a lot of the ambiguity that still exists--in terms of the technical issues and different ways of looking at things--will go away if scientific principle survives all of this. Model results and things associated with evaluating model results must be reproducible in general. That was not talked about too much, but I think that is very critical. I believe it will be true, far more so in the future than has been in the past, that the systems that are used to support decisions are going to be in some sense tractable and available to anyone who wishes to examine the process that led to the information that formed the decision. This even extends to the technical issue of model portability.

There have been estimates that assessing a model more or less takes as many resources as developing the model. I have no basis for questioning that. Mike Shaw says documentation takes about 25 percent of the resources to develop a model. This means that, in general, it costs more than twice as much to do what we have been doing since it is agreed that we have not really documented or assessed up to the level that we are proposing. It

is very expensive. We should emphasize to people that are going to buy the services that, to do it right, and we have been instructed by the users to do it right, that it is very expensive.

DR. WOOD: Could I just mention a footnote to that. In production, in general, the initial production cost is typically just a very small fraction of the total cost of commercializing a product, bringing it to market, and so forth. I think there is a parallel here, so maybe we should not be so surprised that something like assessment that makes the model more usable, more understandable, is going to be expensive and the balance between the cost of assessing the model and the cost of producing it may not be excessive at all.

DR. GASS: Thank you George. Next, Dave Nissen.

DR. NISSEN: I would like to thank Saul Gass, the National Bureau of Standards, and George Lady for making this workshop possible. It has been a very exciting workshop for me and I am pleasantly surprised at how much better we understand our ignorance in this area now than we did two days ago.

Unfortunately I feel compelled to conclude here on a down note.

What

I have gathered here is that we do not, either collectively or individually, understand the role of science in policy modeling very well at all in any way that is operational. I have two examples of this and, then, a hint of the reason. One example is that I found myself nodding very interestedly at David Freedman's description of the READ model. I don't want to get into the merits of the detailed READ model assessment, but I was saying to myself, "Gee, it would be terrific to have a model like that.' You could fix some of the things that David said was wrong with it, and I would conjecture that READ would look a lot like other models. It would suffer about the same degree of disability and shakiness, it would be estimated on about the same kinds of data bases, but people would find such a model cast in a consistent accounting framework to be fabulously useful. I could make a long list of things for which you might want to have that kind of regional economic model disaggregation--environmental assessments, for example, at the air quality control region in analysis, but aggregated to state-level impacts for reporting. (You need a complete accounting of economic activity at a fine level of regional disaggregation to understand the conversion of pollutants to pollution).

But David Freedman concluded that, as a professional statistician, there was sufficient reason to not build the model essentially because of econometric problems. It occurred to me that my view of what one did with models had to be very different from his and that I and my other colleagues in policy analysis had done a very bad job at communicating how policy models are actually used, what you do when you build them, and what they turn out to be valuable for in the policy-making process.