MODEL ACCESS AND DOCUMENTATION Michael L. Shaw Logistics Management Institute (LMI) 4701 Sangamore Road Washington, D.C. 20016 Introduction The objectives of this paper are to describe the needs for the documentation that is necessary to perform evaluation of models and to touch upon the topic of access by outsiders to models in as far as these topics relate to model evaluation. Although our own experience at LMI has been in the documentation and analysis of access issues related solely to energy models these comments should be applicable to most types of models. Documentation We must commence by asking the question, "Why is documentation an important topic to discuss at a meeting on model evaluation?" The answer simply is that without documentation it is impossible to evaluate the work that others have done on models and the nature of the documentation will either facilitate or hinder the evaluation process. Documentation is a topic discussed in several of the other presentations in this workshop, and in some of them, formats or requirements for documentation have been proposed. It is documentation that permits outsiders to learn about the model, the input data, and the forecasts that are obtained with the model. The ease with which an outsider is able to use the documentation to obtain the understanding that he or she desires is a reflection of the quality of the documentation. The credentials with which I discuss this topic are that some three years ago, LMI produced an initial documentation of the PIES Integrating Model. This task has become similar to that of painting a large suspension bridge. As soon as one completes painting the far end, one has to go back to the beginning and start to repaint. This is wholly analogous to the problem of documenting a large and evolving model like PIES which is constantly in a state of change. I should stress that my insights, are far from unique. In particular, I commend you to a comprehensive, well written and entertaining paper on the subject by Saul Gass (Ref. 1); and also works by House and McLeod and Rafael Ubico (Ref. 2 & 3). In addition, most government agencies have standards for documentation of computer models and computer software including, of course, The National Bureau of Standards (Ref. 4), which Saul Gass describes as being the most comprehensive. It is important to ask is--"What are the objectives of the documentation, what is it that one is trying to communicate?" The question must be answered by saying that there is no single objective or set of objectives; the specific objectives depend partly on the background of the individual who needs the documentation, partly on the environment, and partly on the use for which the individual needs the documentation and perhaps the model. As examples of the first of these three factors, mathematicians are clearly interested in precise mathematical statements of the problems that are being modeled; economists are interested in what models represent in an economic sense; regulators might be interested in what the model represents in the way of regulation; and so on. The second and third factors governing the objectives of documentation are the environment in which the user finds oneself and the uses to be made of it. Depending on whether one is a government analyst who is going to have to work with the model; an entrepreneur or businessman whose business environment may be impacted by policy decisions based upon forecast from the model; a policymaker; or the taxpayer who is concerned that the government is using its modeling funds in an appropriate fashion, one's objectives for the documentation will differ. In consequence I would suggest there exist the following three major objectives for documentation: The provision of a description of the model for policy level The documentation should present the model in such a way that is The documentation should provide an archive which permits the I suggest that the issues to be answered through documentation are: What does it do? This should describe the applications and What doesn't it do? This should describe the limitations of the How does it do it? This should contain a narrative description sary to use the model. Each of these items should in turn tion should describe the language, the machine on which the How well does it do it? This should include any information What assumptions are made? What data are used and whence come they? What results or forecasts exist? What plans are there for the model? What resources does it require to run the model? What is the organizational environment in which the model is What must a potential user do to access the model? It is worth repeating that each of these issues must be addressed on several different levels. We can now identify a list of required documents, they are: An executive summary description of what the model does. A detailed narrative description of what the model does including the principles, structure, and assumptions of the model. A complete mathematical statement of the model. The computer implementation; perhaps including the model codes. A user's guide describing not only how to run the model on a The data base. A description of the validation, verification and audit record The future development schedule for the model. A record of changes made to the model. The results of using the model; both the raw outputs and analyses This comprehensive list of documents is a goal. It is probably true that no model will ever have all of the appropriate documentation up to date. However, even though this is an ideal or goal, in most instances, the current status of energy model documentation leaves considerable room for improvement. The National Bureau of Standards FIPS Pub 38 (Ref. 4) suggests that the level of documentation required should be linked to the uses that are made of a model and the costs of the modeling effort. This is not something I have pursued here. In addition to these external documentation needs, there will be internal documentation, not necessarily of primary interest in the model evaluation and assessment process. This documentation would typically include sub-program specifications and internal management matters. Because such documentation may only be of limited value in evaluation, it is not discussed further. I suggest that the documentation requirements presented here are more comprehensive than suggested elsewhere and seek to illustrate this by the use of Table 1 which shows the approximate correspondence between the 11 types of document proposed here, the 10 types of document suggested in FIPS Pub 38 (Ref. 4), the 12 types of document identified by Ubico (Ref. 3), and the five documents called for in the recent EIA interim documentation standards (Ref. 5). It seems worth noting that the EIA standards are completely concerned with describing what existing models do. A stronger emphasis on the originally specified goals of the models might be desirable. It is worth discussing some of these proposed documents in more detail starting with the model specification. Ideally, the specification would be written before any development was done on the model. The model specification should be written to at least two levels. First, there should be the broad requirement of what the model should do and secondly there should be a more detailed specification of the mathematics and the structure the model will have when it is implemented. One issue is who should write a model specification, and my suggestion is that it should be a collaborative effort between the potential user and the modeler. The potential user will express his awareness in one set of terms; the modeler will have a different awareness set, which, in all probability, will constrain the implementation of the user's broad objectives. I do not think that it is inappropriate to point out that the greater the specificity of a specification the better it will be. 1of these 12, seven are included in the category 'Programming Documents.' |