[9] In IEEE D. W. North. "A Tutorial Introduction to Decision Theory." [10] H. Raiffa. Decision Analysis: Introductory Lectures on Choices Under Uncertainty. Reading, MA: Addison-Wesley Publishing Co., 1968. [11] E. G. Cazalet. Generalized Equilibrium Modeling: The Methodology of the SRI-Gulf Energy Model. Final Report prepared by Decision Focus Incorporated for the Federal Energy Administration, May 1977. [12] C. S. Spetzler and C.A.S. Stael von Holstein. "Probability Encoding in Decision Analysis." Management Science, Vol. 22, No. 3 (November 1975). Reprinted in Readings in Decision Analysis. R. A. Howard et. al. (eds.). Menlo Park, CA: Stanford Research Institute, Decision Analysis Group, 1976, pp. 403-427. [13] C.A.S. Stael von Holstein. Assessment and Evaluation of Subjective Probability Distributions. [14] C.A.S. Stael von Holstein. Probability Distributions.' Research Institute, 1972. " [15] A. Tversky and D. Kahneman. Frequency and Probability." 207-232. "A Bibliography on Encoding of Subjective Unpublished manuscript, Stanford "Availability: A Heuristic for Judging Cognitive Psychology, Vol. 5, 1973, pp. "Judgment Under Uncertainty: [16] A. Tversky and D. Kahneman. Heuristics and Biases." Science, Vol. 185 (September 27, 1974), pp. 1124-1131. [17] R. L. Winkler. "The Assessment of Prior Distribution in Bayesian Analysis." Journal of the American Statistical Association, Vol. 62 (1967), pp. 776-800. [18] R. L. Winkler and A. G. Murphy. "Experiments in the Laboratory and the Real World." Organizational Behavior and Human Performance, Vol. 10 (1973), pp. 252-270. [19] K. Arrow. "Aspects of the Theory of Risk Bearing." Yrjö Johnson Lectures, Helsinki, 1965. [20] D. W. Boyd. A Methodology for Analyzing Decision Problems Involving Complex Preference Assessments. Ph.D. Dissertation, Stanford University, 1970. Available from University Microfilms (No. 71-2737), Ann Arbor, Michigan. [21] D. W. Boyd and C. E. Clark, Jr. Multi-Party Decision Analysis for Social Decisions. Decision Focus Incorporated Working Paper No. 2, August 1978. [22] R. A. Howard. "Life and Death Decision Analysis." In Proceedings Second Lawrence Symposium on Systems and Decision Sciences, 2:271277, 1978. [23] R. A. Howard. "Risk Preference." Readings in Decision Analysis. R. A. Howard et. al. (eds.). Menlo Park, CA: Stanford Research Institute, Decision Analysis Group, 1976, pp. 429-465. [24] Ralph L. Keeney and H. Raiffa. Decision Analysis with Multiple Conflicting Objectives: Preferences and Value Tradeoffs. New York: John Wiley and Sons, 1976. [25] von Neumann and Morgenstern. Theory of Games and Economic Behavior, Second Edition. Princeton, New Jersey: Princeton University Press, 1947. [26] Ronald J. Adler et al. The DFI Energy-Economy Modeling System. Final Report prepared by Decision Focus Incorporated for the U.S. Department of Energy. DFI Project No. 1023, December 1978. [27] E. G. Cazalet et. al. DFI Model Data Management System. Working Paper. DF I [28] E. G. Cazalet, C. E. Clark, Jr. and T. W. Keelin. Costs and Benefits of Over/Under Capacity in Electric Power System Planning. Report prepared by Decision Focus Incorporated for the Electric Power [29] E. G. Cazalet et al. Decision Analysis of Nuclear Plants in Electrical System Expansion. Final Report prepared by Stanford Research Institute for Comision Federal de Electricidad, Mexico City, D.F., 1968. [30] R. A. Howard, J. E. Matheson, and D. W. North. "The Decision to Seed Hurricanes." Science, Vol. 176 (June 16, 1972), pp. 1191-1202. [31] J. E. Matheson and W. J. Roths. "Decision Analysis of Space Projects: Voyager Mars." National Symposium of the American Astronautical Society, June 11, 1967. Reprinted in Readings in Decision Analysis. R. A. Howard et. al. (eds.). Menlo Park, CA: Stanford Research Institute, Decision Analysis Group, 1976, pp. 309-340. [32] [33] D. W. North and M. W. Merkhofer. "Analysis of Alternative Emissions Synfuels Interagency Task Force. Recommendations for a Synthetic VALIDATION ISSUES--A VIEW FROM THE TRENCHES* W. Marcuse, F. T. Sparrow,** D. A. Pilati Economic Analysis Division I. Introduction A great deal of attention has been directed towards model evaluation and assessment. A bibliography compiled by Saul Gass lists 37 articles and monographs and 14 books and reports devoted to model evaluation or assessment. (Gass, undated) Most of these, in dealing with verification and validation, discuss means and mechanisms by which "outside" parties can perform peer review to provide verification (model behavioral response is as intended and publicized) and establish the validity (model produces results one would expect, e.g., in the case of most models, it will recreate history) of models. (Gass, 1977) Little attention is paid to activities performed by the user modeling team itself to improve the ability of the model to provide information useful in the decision making process, and to provide confidence that the information is meaningful. This paper presents a number of case histories describing our experience with this type of model improvement activity which we have called internal validation. Our experiences are illuminating since they were learned in the context of formulating, developing, and exercising a specific set of process models. This experience has convinced us that internal validation schemes (our definition) should be incorporated in the project description and that they be used in part to answer questions of formulation. Having discovered the need to perform explicit internal validation, we recommend that modelers incorporate sufficient funding in their project plans to carry out this function and to fully document it. In general, this will be an unwelcome addition to sponsors already unhappy with the size of their modeling budget. Increasingly, we turn to government to intercede in areas where economic equilibrium is subject to market failures, where externalities previously ignored are now considered socially undesirable, or where political goals have to be satisfied. These activities require the manipulation of enormous data bases. This has prompted an increased acceptance of information provided by quantitative models capable of such manipulation by the actors in the decision process and an increased demand for such tools. It is not surprising that model builders and users have evidenced increased concern with regard to the quality of their products. Figure 1 indicates where such models can fit into the decision process. Decision makers are faced with a wide range of policies and actions (Box 1). They also are acutely aware of the political and institutional limitations *Work supported by the U.S. Department of Energy Under Contract No. EY-76-C02-0016. **School of Industrial Engineering, Grissom Hall, Purdue University, West Lafayette, Indiana 47907. 337 |