ANALYSIS AND CONCLUSIONS A main purpose of this study was to develop a basis for discussing possible model guidelines, their feasibility and acceptability by the model developer and user groups. From the results of this study and our review of previous surveys and model research, certain directions are clear. An analysis of the available information should convince all interested parties that certain model improvement possibilities should be initiated, some dropped, and others put aside for the time being. Further research on improving model utility now must be focused on specific proposals. Future analysis should be directed towards the determination of the costs and effectiveness of these proposed activities. Then, the Government, in conjunction with the modeling community, should move to develop and implement the most beneficial activities. To aid in the analysis, we have grouped the model utility proposals by six modeling activities. These are model initiation (propositions 4, 5, and 6), model development (propositions 1, 2, 8, and 12), model implementation (proposition 3), model management (propositions 9, 10, 13, and 14), model assessment (propositions 7, 11, and 16), and model research (propositions 15, 17, and 18). We next present our conclusions using these groupings. A. Model Initiation (priority counts are in parentheses) Conclusions: Proposition 4, dealing with model documentation plan and guidelines, should be selected for future development. If guidelines are developed, the results of proposition 5 indicate that they should be applied selectively, based on how the model There is good support for proposition 6 and its is to be used. doing this should be investigated. statements on improving the RFP statement of work. Ways for B. Model Development Proposition 1: Data Collection and Availability for Model Proposition 2: Proposition 8: Standardized Procedures for Model Developers (2 to 3) Relationship Between Model Users and Developers (15 to 0) Proposition 12: Model Ongoing Review Panel (0 to 2) Conclusions: Propositions 1 calls for distinct data availability, collection and assessment tasks to be made part of a modeling project. of ways for improving the total data aspects of a project. This proposition should be selected for further development in terms There appears to be no real strong support for model standardization, Any standards should be directed towards computer proposition 2. Procedures for strengthening the but these should be voluntary. bpects (languages, routines, structured programming techniques), relationship between model developers and users are an overwhelming choice for development, proposition 8. In particular, formal meetings between the developer and user should be an RFP requirement. 12 calls for ongoing model review panels. be pursued. 249 Proposition It should not, in general, Proposition 3: Model User Training (3 to 0) However, Conclusions: Proposition 3, dealing with the establishing of training tasks, should be investigated at a future date. major modeling projects that require the transfer of a modeling system from developer to a Government user should incorporate training and maintenance tasks in the contract. D• Model Management Phased Management Approach to Model Development Proposition 10: Government In-House Model Development (0 to 2) Conclusions: None of the model management proposals received very strong support. However, proposition 9, on the phased approach to model management, appears to have some good support from all groups. We suggest that it be selected for future development, depending on the availability of resources. An industry/Government group established for the purpose of determining the form and function of phased model management would be a low-cost way to continue the investigation in this area. E. Model Assessment Model Verification and Validation Plan (6 to 0) Conclusions: Model assessment is becoming an important aspect of the modeling process. Proposition 7 requires a model developer to prepare a verification and validation test plan and to carry it out. This proposition should be further developed. review panels should not be implemented. Proposition 16 received very little support and was the second lowest in priority. It should be dropped from further consideration. F. Model Research Post Proposition 15: Central Modeling Clearinghouse (2 to 5) Conclusions: The clearinghouse and modeling research center proposal should be dropped from any future consideration. The center received the largest low priority count of all proposals. The concept of the modeling forum is endorsed strongly, especially forums established by professional organizations and industrial groups. The forum proposition should be selected for future development. SUMMARY Based on our analysis of the above material and reviewer comments, all categories of reviewers are against those model propositions that would tend to increase Government bureaucracy. The unfavorable reaction to propositions 15, 16 and 17 attests to this conclusion. However, there is recognition that the Government must begin efforts to improve the utility of the models it sponsors. Thus, strong support is given to model documentation plan and guidelines, and good support to model phased management and the RFP statement of work. At the same time, the responsibilities of the model developers are recognized in the support of the propositions dealing with verification and validation, and data collection and availability. The joint needs of the users and developers are recognized by the strong support of the user/developer interaction and the modeling forums. We next summarize the above discussion as follows: Propositions for Further Research Strong Support Proposition 4: Data Collection and Availability for Model Model Documentation Plan and Guidelines (11 to 1) Proposition 18: Modeling Forums of Users and Developers (6 to 0) Support Proposition 6: Proposition 9: Possible Support RFP Statement of Work for Model Development Phased Management Approach to Model Development Proposition 3: Model User Training (3 to 0) Propositions Not To Be Supported for Further Research Proposition 15: Central Model Clearinghouse (2 to 5) Proposition 16: Government Model Testing, Verification and Proposition 17: Government Modeling Research Center (3 to 18) We feel that future activity on how to improve the utility of models should be concerned with the above strongly supported propositions. Ways for accomplishing these propositions, in a cost-effective manner, need to be explored and tested. A final summation of the propositions and the reviewer information is, we feel, the following. Most of the supported propositions and their statements represent good modeling practices. It is not clear why these practices are not put to use on a regular basis. What the supported propositions call for is a better professional attitude toward modeling by all facets of the modeling community -- developers, users and sponsors. The reviewers have expressed how they believe the Government and industry modelers can improve the professional field of modeling and thus, increase the utility of models. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] "Advantages and Limitations of Computer Simulation in Decision-Making,'| B-163074, U.S. GAO, Washington, D.C., May 3, 1973. "Computer Simulations, War Gaming, and Contract Studies," B-163074, U.S. GAO, Washington, D.C., February 23, 1971. "Improvement Needed in Documenting Computer Systems," B-115369, "Models, Simulations, and Games -- A Survey", M. Shubik and "Federally Supported Mathematical Models: Survey and Analysis," "An Approach for Developing Feasible Guidelines for Large-Scale March 1977. " "Improvements Needed in Managing Automated Decisionmaking by Computers Throughout the Federal Government, " FGMSD-76-5, U.S. GAO, Washington, D.C., April 23, 1976. "Review of the 1974 Project Independence Evaluation System,' [10] [11] "Auditing a Computer Model: A Case Study," Division of Financial "Guidelines for Documentation of Computer Programs and Automated [12] "Guidelines for the Practice of Operations Research," "The Operations Research Society of America, Operation Research, Vol. 19, No. 5, September 1971. [13] "Evaluation of Complex Models," Saul I. Gass, Computers and Operations Research, Vol. 4, pp. 27-35, 1977. [14] "Models in the Policy Process," M. Greenberger, M.A. Crenson and B.L. Crissey, Russel Sage Foundation, N.Y., 1976. [15] "Homilies for Humble Standards," D.T. Ross, CACM, Vol. 19, No. 11, November 1976. [16] "Requiem for Large-Scale Models," D.B. Lee, Jr., American Institute of Planners Journal, May 1973. 17] 18] 19] 201 21] "Politicians, Bureaucrats and the Consultant," G.D. Brewer, "Systems Analysis in Public Policy," I.R. Hoos, U. of California "Large-Scale Models for Policy Evaluation," Peter House and J. McLeod, John Wiley & Sons, N.Y., 1977. "A Procedure for the Evaluation of Complex Models," Saul I. Gass, Proceedings, First International Conference on Mathematical Modeling, August 29-September 1, 1977, University of Missouri-Rolla. "Evaluation of Policy Simulation Models," R.E. Pugh, Information Resources Press, Washington, D.C., 1977. |