ON A PERSPECTIVE FOR ENERGY MODEL VALIDATION Lawrence S. Mayer Department of Statistics 1. INTRODUCTION A recent article in the Chronicle of Higher Education suggests that one indicator of the health of a discipline is the proportion of its scholars regarded as "big thinkers." I am pleased to report that by this criterion, model validation is in marvelous health. The previous presentations have convinced me once again that we are blessed with a copious supply of splendorous notions of validation and its uses. But although this state of affairs may be a sign of healthy originality, it masks a number of fundamental problems, including a lack of formal, rigorous definitions of the basic terms employ. we A group of scholars, no matter how sophisticated, is not likely to agree on relationships between ill-defined concepts, and, moreover, such agreement would be meaningless in the absence of basic definitions. We are often far too eager to define the relationships between grand concepts such as Validation, Evaluation, Verification, and Ventilation before we have even agreed on the meanings of these terms. I would be the first to admit - and I am sure that many here will bear witness that I do not know the complete meaning of the simplest of these terms, "validation." My goal here is to introduce a perspective on validation that will make absolutely and explicitly clear its assumptions and definitions the precondition of reasoned debate and credibility in the scientific and political worlds in which we operate. I ask my colleagues, then, to be tolerant, rest their minds, and listen to a few ruminations on this question of perspective. Traditionally, we have paid a great deal of solicitude to the question: "Are modelers doing a good job?" And after weighing the evidence, most of us have concluded that given the current state of information about energy processes, modelers are doing as well as can be expected. I endorse this conclusion but suggest the question itself is misleading because it confuses and confounds issues vital to the energy analyst struggling to assess the value of a model. In particular, the question as framed ignores the dual nature of energy models as products of science and agents of policy. These natures are often confused; they must be assessed separately using distinct sets of criteria, outlines of which I will explore in this paper. Pretend for a moment, if you will, that as modelers (and validators are all modelers in another incarnation) our purpose is not to convince our sponsors to increase our funds, not to persuade our academic brethren and sistren of the legitimacy of our discipline, not to secure fame and glory, and not to propagate our species by recruiting and training students but actually to estimate the scientific and political merits of our products. Stripped of extraneous issues, then, the question of validation divides in two: A. B. To what extent, and in what way, does an energy model teach us about the world we live in? To what extent, and in what way, is an energy model an important agent in the political process? In answering question A, we find ourselves in the realm of the energy model as science. We would not pose the question at all if we did not believe, as scientists, that the stronger the scientific underpinnings of a model, the more likely it is to tell us something about the world. To answer the question, we must judge the model by the degree to which its methodology conforms to the spirit and canons of contemporary science. It is wrong, and probably fatal, to continue to apologize for serious violations of scientific method. We can, should, and must withstand the scrutiny of our peers in science. Question B, on the other hand, takes the model from the womb of scientific creation into the world of political life. To evaluate a model as an actor in this world, we must isolate its uses as as a political tool, judge the aptness of these uses, and estimate the ability of the model to contribute to political victories. To start with, we must ask how the model's advertised accuracy compares with the actual needs of those who commissioned it. But as a supplement, we must also ask about its other roles, which, though not based in science, can be recorded and assessed scientifically. For example, we can and should the degree to which political debate is enhanced or restricted by the use of a particular model. Similarly, the political implications of the perspective underlying a model should be appraised, since all such perspectives contain political suppositions and biases. can no longer afford to dismiss critics of the political uses of models as anti-technological cranks. And so, as much as I welcome our discipline's day in the court of science, I also encourage its appearance in the chamber of public policy. Just as we must examine the scientific integrity of our work by probing what it can actually tell us about the world, so examine its political integrity by asking to what political uses it is adaptable and to which misuses it is vulnerable. All the while, we must bear in mind that the answer to one of these questions does not necessarily follow from the answer to the other. The best scientific model can prove unacceptable for a variety of political reasons: it may not provide policy makers with the forecasts they need, those they expected, or those which are robust against unforeseen changes in exogenous variables. And conversely, the least scientific of soothsayers, acting to satisfy a policy maker's needs, could produce a forecasting mechanism that turns out to be accurate. As scientists, we believe that on the average, science is the best approach for modeling empirical processes. The evidence for this belief lies in the aggregate, however. It does not and cannot rest on the supposition that science will produce an accurate forecasting model for each and every problem. It is this dual perspective on validation that I intend to develop in this paper and to supplement with an example drawn from the evaluation of a singleequation model for the rate of production of domestic crude oil. The establishment of validation as a legitimate enterprise involves questions in need of serious study. Before anyone begins to repine, let me admit that I have no complete answers. But if the most critical step in understanding a process is focusing attention on the most pertinent questions, and if the process at hand is energy modeling, then these are the questions to work on. Asking them will not replace or displace the work of my colleagues, but it will require work different from that being reported here or, in fact, any currently funded by the Department of Energy. Answering these questions addresses issues that must be considered if energy modeling is to develop what it so sorely lacks: the cumulative nature of a true scientific enterprise and the accompanying respect of policy makers, policy analysts, and, most importantly, the public at large. The perspective on model validation that I have tendered draws heavily on three experiences. The first is having been a Principal Investigator of the Department of Energy-sponsored Princeton Residential Energy Conservation Project, a six-year interdisciplinary study of the end-use of energy in a single residential community. Some results of this study are summarized in a recent book (Socolow, 1978); additional statistical analyses are presented in Mayer (1978a, 1979a), Horowitz and Mayer (1977), Mayer and Horowitz (1979), and Tittman (1978). The second is directing the Energy Information Administration-sponsored Princeton Resource Project, which is validating and improving methodologies for estimating domestic and international resources of crude oil and natural gas (e.g., Mayer, et al., 1979). The third is having directed a study for the Department of Commerce which produced a critical review of large-scale econometric energy models (Mayer, 1979b). The perspective owes its theme to John Shewmaker, Deputy Assistant Administrator for Energy Information Validation of the Energy Information Administration, who asked me, "What does it mean for a model to be valid?" Before I could answer, he warned me that he had recently posed the question to a dozen people in the business and received 12 different answers, none of them satisfactory. Well, I thought of the old Buddhist adage that says if 12 wise men, or women, give different answers to a single question, then it must be the wrong question. In the spirit of this adage, I have concluded that the question Shewmaker posed appears simple and direct but is actually compound and complex. To answer it honestly we must develop a new perspective on the issue. This perspective distinguishes components of the modeling process as indicated in Figure 1. The modeling of any empirical process begins with a conceptual approach. This approach includes a theory or a pre-theory about how the process functions and some prior expectations about the kind of evidence that could disconfirm the theory. The conceptual approach is joined with a methodology, a set of procedures for developing the theory into a metaphor for the empirical process. The methodology is applied to an information base, which includes a set of data, to produce a "model, a system of equations, or other analytic system, and a set of rules governing the use of that system. The term model, used in this sense, is put in quotes because the entire intellectual product is, in some sense, the model, and confusion may arise from blurring the distinction between a "model" and a model. The former is an analytic structure and a set of rules. The latter includes the former but also includes the conceptual approach, methodology and information base used to produce the "model." The uses made of the model comprise the other half of the perspective. These include all uses, both advertised and unadvertised, those that depend directly on the forecasts of the "model" and those that depend on the existence of the model and only indirectly on its forecasts. There is a tendency among modelers to assess model use in the most esoteric fashion, as if models were used only by angels involved in rational debate over zoning the environs of heaven. Models are policy agents and political weapons and must be studied as such. Seen through this perspective, the components of the modeling process have been confused habitually in validation studies. The very question, "What is a valid model?" tends to blur the distinctions among them and should be replaced by questions like these: i) What does it mean for a conceptual approach, ii) iii) methodology, and information base to be appropriate? What does it mean for a methodology to be optimally What does it mean for a model to provide accurate iv) What does it mean for a model to be an effective As these questions indicate, the issue of validity can be split into three separate problem areas. The first area deals with the validity and appropriateness of the conceptual approach, the methodology, and the information base. These are problems of science and must be treated as such. The second problem area concerns the optimality of the application of the methodology as it is used to produce the "model" and the operating characteristics of the "model" produced. These are problems of verification (dear to statisticians), including: model specification, alternative functional forms, aggre |