A DECISION ANALYST'S VIEW OF MODEL ASSESSMENT Edward G. Cazalet Decision Focus, Incorporated INTRODUCTION As a decision analyst, I must be very skeptical about the usefulness of model assessment and validation for two reasons: First, any assessment activity should focus on the quality of the decision process; the quality of any models used in the process is only one area of assessment. Second, assessment and validation are extremely difficult tasks to do well because of the necessity for hindsight. Despite this skepticism, I shall try to make a positive contribution to this workshop on model validation and assessment by using the framework of decision analysis to outline an approach to assessment. I will begin by first reviewing the basic concepts of the decision analysis framework. THE FRAMEWORK OF DECISION ANALYSIS Decision analysis is a term used to describe a professional practice and methodology for aiding decision making [1-10]. The framework of decision analysis is designed to improve a decision process but is also can be viewed as a framework for assessing the quality of a decision process. In non-technical terms, the framework of decision analysis is outlined in Figure 1. Good Decisions Versus Good Outcomes The first step in describing the decision analysis framework is to define a decision. A decision is an irrevocable allocation of resources in the sense that it would require a large amount of additional resources to change the allocation. The next step is to distinguish between a good decision and a good outcome. A good outcome is one that is favorably regarded by those with the power to make the decision. A good decision, however, cannot be defined as one that produces a good outcome. Because of uncertainty, a good decision may produce either a good or bad outcome. A good decision must be defined in terms of the process of decision making. Loosely speaking, we would like to increase the likelihood of good outcomes by doing all we can to gather information, create new But the process of Therefore, we must alternatives, and contemplate what is a good outcome. decision making itself consumes resources and time. define a good decision as one that is the result of a process that economically balances all aspects of the decision problem including the cost of the process itself. Decomposition of the Decision Problem The basic idea of decision analysis is to gain insight into complex decision problems using a "divide and conquer" or decomposition approach. We proceed by decomposing a complex decision problem into a number of elements or subproblems, each of which is easier to analyze than the original problem. Then we combine the analyses of the subproblems into an overall analysis of the original problem. Figure 1 shows the decomposition of the decision problem into three basic elements; information, alternatives, and preferences. Each of these elements is analyzed independently and recombined in an iterative process of analysis that is designed to provide a better understanding of the original, complex decision problem. Information. Information describes "what we know." Information can be represented in two ways: by means of relationships structured in the form of a model and by means of probability assignments. Structural information consists of information describing how things are connected to other things. For example, we know that the electrical energy produced by an electric power plant is related to its fuel use. Typically, structural information can be represented in terms of equations relating several variables. The value of a quantitative model is greatest when we have many equations and variables. Here the unaided human mind is unable to cope with the solution of many equations in many variables, whereas a computer model can solve thousands of equations [11]. We will later consider the role of structural models in more detail, particularly as their quality relates to the quality of the decision process. The second way of representing information is by means of probability assignments. There is only one way to communicate uncertainty and that is the language of probability. Decision analysis views probability as a state of mind rather than things. This subjective view of probability includes situations where quantity of experimental data is influential in the probability assignment. A major area of concern in decision analysis is compensating for the human motivational and cognitive biases that may influence the assignment of probabilities [12-18]. As we shall see, the psychological and analytical techniques that have been developed for assigning probabilities provide a number of useful insights into model validation and assessment. Alternatives. Alternatives describe "what we can do." It is important that an analysis consider the full range of decision alternatives. Decision analysis is normally thought of as a procedure for selecting among a set of well-defined alternatives. However, an important aspect of decision making is the creative process of generating new alternatives. Often, an analysis will facilitate the creation of new alternatives by focusing attention on the important aspects of the problem. For example, the inclusion of uncertainty in an analysis may suggest hedging alternatives and contingency plans that might otherwise not be considered. Preferences. Preferences describe "what we want." The importance of preferences from the perspective of model assessment is to identify the important role of preferences in the use of models for analysis. In a decision analysis it is useful to distinguish between four types of preferences: value, time, risk, and equity [19-25]. Value assignment concerns trade-offs between the known consequences of a decision; uncertainty and risk preference are treated elsewhere. In public decision problems, values might be set on the health, mortality, and esthetic consequences as well as the monetary consequences of a decision. Often it is convenient to assign values in monetary terms, but it is not necessary to do so. Time preference concerns trade-offs between outcomes distributed over time. When values are expressed in monetary terms it is often useful to use a discount rate to characterize time preference. Risk preference is a term used to describe the fact that most people are not willing to choose among alternatives simply on the basis of the expected value of each alternative (the probability weighted values of all possible outcomes of a decision). Risk preference is therefore a reflection of attitude towards uncertainty; uncertainty itself being described in probabilistic terms in the information element of the analysis. Equity trade-offs are relevant in decision problems where the outcomes of more than one party are of concern. Equity trade-offs describe how value to one party is to be traded off against value to each other party for purposes of making a decision. In making equity trade-offs it may be difficult or unnecessary to get general agreement among the parties. Logical Process of Analysis A decision analysis proceeds by iteratively decomposing a decision problem into its basic elements (information, alternatives, and preferences) and then combining these elements into an overall analysis. At each stage of this process the decision makers or appropriate specialists are involved in developing and analyzing each element. final result is not so much identification of a good decision as it is development of insight into what makes a good decision. If the analytical process is effective then the intuition of the decision makers should be consistent with the insight from the analysis and the resulting decision |