Determination.-Problems of program management and relationships between Centers and with the contractor have led in some cases in insufficient response to changing program requirements.

Recommendation.-Every effort must be made to insure the maximum clarification and understanding of the responsibilities of all the organizations involved, the objective being a fully coordinated and efficient program.

Action. A number of steps has been taken to improve understanding between all organizations to improve implementation of practices and procedures.

APOP 0-202, Preparation and Release of Test and Checkout Procedures, was released July 12, 1967, and describes the MSC/KSC system requirements for reviewing and approving all spacecraft test procedures. All spacecraft test procedures are forwarded from KSC to MSC for review and comment. The MSC flight crew checklist information is required to be furnished to KSC 40 days before the scheduled test date with all MSC procedures_comments due no later than 15 days before the scheduled test date. MSC will also approve by signature, all spacecraft test procedures that include participation of the flight crew. All revisions to actual test plans must be in the hands of the test team two days before the test. No last-minute revisions to operational test procedures are acceptable and this limitation can only be waived by the KSC Director, or his designated alternate. Other key documents defining and clarifying KSC/contractor relationships include:

APOP No. 0-222, TCP Open Item Reviews, Constraint Listings, Pre-Test Briefings, and Post-Test Debriefing released August 2, 1967. This APOP requires pretest open item reviews, documentation of constraint items with proper approvals, open item closeout verification, and post-test anomaly critiques. This APOP includes a methodical means of evaluating the hazard level of all spacecraft procedures and assuring that safety practices are followed. The following safety reviews were recently conducted by NASA and contractor personnel:

(1) Launch vehicles for missions AS-501 and AS-204 had detailed electrical and corrosion inspections at KSC to determine any potentially hazardous conditions. All discrepancies noted have been formally closed out.

(2) All KSC operating procedures for launch vehicle AS-501 which was launched on November 9, 1967 were subjected to an Operational System Safety Evaluation. 1354 procedures were reviewed in detail and all problems noted were closed out by formal reports prior to the launch.

(3) A KSC Ad Hoc Safety Review of documentation involving Launch Complex 39 hardware and safety conditions was conducted prior to the AS501 launch. All potential problem areas are being corrected.

(4) A pre-operational safety review of spacecraft 017 was conducted by KSC and MSC personnel. All potential problems identified were acted upon prior to the AS-501 Flight Readiness Test.

Apollo Program Directive No. 26, Preparations of Test and Checkout Plans and Procedures at KSC, issued April 18, 1967 by the Apollo Program office defines specific requirements for:

(1) Test and Checkout Procedure (TCP) standardization;

(2) Review and use TCP's by KSC Safety and Quality control offices; (3) Timing and revision control of TCP releases;

(4) Open work, deviation and change reviews;

(5) Flight crew use of TCP's.

Center responsibilities are assigned for preparation, signature approval, deviation approval, participation in reviews and assurance that adequate tests are being accomplished for all hardware at KSC. This directive is in effect and has been used as the basis for development and use of all TCP's at KSC.

Apollo Program Directive No. 33, Center Responsibilities in the Apollo Program, M-DMA 1400.074 is now in effect. This directive assigns and augments responsibility and functions and defines intercenter relationships including those for: (1) Hardware qualification and acceptance;

(2) Delivering qualified hardware to KSC;

(3) Controlling hardware configurations at each stage of preparation or test in the factory, test site or launch site, including approval of changes at KSC;

(4) Providing quality control requirements and inspection criteria for use at the factory, test site or launch site;

(5) Conducting audits to evaluate contractor performance in accordance with inspection and quality control criteria.

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION

STATUS OF ACTIONS TAKEN ON THE APOLLO 204 REVIEW BOARD REPORT

Volume II-Status of Actions Taken on the Apollo 204 Panel Findings and

GLOSSARY

REFERENCE INFORMATION

The actions reported in this volume are based on the panel findings and determinations found in Appendix D of the Report of the Apollo 204 Review Board to the Administrator National Aeronautics and Space Administration, dated April 5, 1967.

The panel findings and determinations are reproduced at the beginning of each finding exactly as they appear in the report referenced above. The Action section describes the actions taken by the Apollo Program to correct the conditions described in the panel findings and determinations. The Responsible Organization heading refers to the NASA Center which was required to take action on the specific panel finding and determination.

The following AS204 panel findings contain determinations that do not require action on the part of the Apollo Program:

Finding. One hundred and sixty-four (164) Engineering Orders (EO's) were not accomplished at the time Spacecraft 012 was received at KSC. Six hundred and twenty-three (623) EO's were released subsequent to receipt at KSC. Of these, twenty-two (22) were recent releases which were not recorded in configuration records at KSC at the time of the accident.

Determination.-Continuing engineering changes indicate progressive development of the Spacecraft configuration through the time of the Space Vehicle Plugs Out Integrated Test. At the time of the test, the configuration could not have been complete with respect to the launch configurations.

Responsible Organization.-MSC.

Action. Spacecraft acceptance and DD-250 administration procedures will be strictly enforced to ensure that shipping documents accurately reflect hardware status. Further action has been directed towards rigidly observing configuration management procedures to ensure that no work is open unnecessarily at the time of shipment. Specifically, the revised CARR plan requires an updated configuration index and listing of open work for each spacecraft before acceptance. It also provides for a KSC representative to review and accept in writing all open work outstanding on each spacecraft. After delivery of the spacecraft to KSC, all changes will be reviewed and approved by NASA. CARR report requirements have been implemented for all spacecraft.

NAR has revised its Policies and Procedures (P&P's) on configuration management to ensure more complete and timely configuration identification, from manufacturing, through each major test to launch. These P&P's are:

(1) F-403, Change Verification Record System, April 13, 1967;

(2) C-432, Engineering Change System, July 24, 1967;

(3) C-432.1, Engineering Change Approval Processing, June 22, 1967; (4) C-432.3, Engineering Change Product Plans, June 22, 1967;

(5) C-432.4, Engineering Change Control for Subcontracted Equipment, July 7, 1967;

(6) C-432.6, Engineering Change Control for Interdivisional Work Authorizations, August 18, 1967.

Finding No. 2

Finding. The required Space Vehicle Plugs Out Integrated Test configuration was not explicitly defined by design engineering or test documentation. Definition of required test configuration was limited to test set-up and controls configurations specified in OCP FO-K-0021-1.

Determination.-The absence of explicit definition of Spacecraft test configuration requirements relegated such definition to the test organization. Further, it is the opinion of this Panel that the lack of timely and explicit design definition of the required test configuration precluded complete assessment of adverse configuration aspects as constraints to the test.

Responsible Organization-MSC.

Action.-The Test and Checkout Procedures (TCP's) define the spacecraft and GSE configuration required for each test. MSC will review all of the spacecraft TCP's written by KSC and will exercise signature approval of the document for those TCP's involving crew participation. An open items/constraints list review meeting (see Panel No. 1, Finding No. 3 for additional details) with the responsible systems engineers of KSC and the contractor will be held prior to initiating any TCP's. This review will include evaluations of

(1) Unqualified hardware list;

(2) Suspect failed hardware list;

(3) Temporary installation records (TIR's).

All nonflight hardware items will be recorded as temporary installations. The Customer Acceptance Readiness Review (CARR) has been modified to require that the spacecraft contractor determine any test constraints associated with each item of unqualified hardware. The CARR is the final review conducted between NAR and NASA to formally evaluate the spacecraft for NASA acceptance prior to shipment to KSC. The CARR is based primarily on the results of acceptance checkout, but also includes evaluation of existing open work, waivers and deviations, and quality control and reliability verifications. The timely and explicit definition of each required test configuration has been accomplished by requiring the spacecraft contractor to maintain and update the unqualified hardware constraints list and the suspect failed hardware list in support of the open item/constraint list review meeting. This CARR plan is effective for all manned spacecraft.

Finding No. 3

Finding. Eighty (80) EO's effective on S/C 012 were not accomplished at the time of the accident. Of these, twenty (20) were specified to be accomplished subsequent to the Space Vehicle Plugs Out Integrated Test. Four (4) of the open EO's were of a nature not affecting configuration. Three hundred and eighty-four (384) Parts Installation and Removal Records (PIRR's) and Temporary Installation Records (TIR's) were open, of which one hundred and twenty-five (125) were in compliance with requirements of the test documentation.

Determination. It is concluded that test requirements had no defined relationship to the open status of fifty-six (56) EO's and two hundred and fifty-nine (259) PIRR's/TIR's. It is the opinion of this Panel that all work items and EO's were not closed because of late receipt of changes or further work scheduled to be accomplished prior to launch.

Responsible Organization.-KSC.

Action.-Test requirements for all spacecraft testing will be related to all open work items. Closure of open work items will be expedited by implementing the instructions of APOP No. 0-222 TCP Open Item Reviews, Constraints Listings, Pre-Test Briefings, and Post-Test Debriefings, released August 2, 1967. This procedure includes and assigns responsibilities for

(1) A specific definition of the requirement for a pretest open item review. The review will be co-chaired by the NASA spacecraft test conductor or his designee and the spacecraft contractor test project engineer.

(2) A system of documenting the constraint items with proper approvals for the test in question or waivers as required.

(3) A procedure for indicating items that are closed out between the conclusion of the open item review and the scheduled start of the test.

(4) A post-test critique at which anomaly items discovered during the performance of the test are evaluated as possible constraints to closeout

of the subject test. All such constraints will be monitored during workoff by Quality Control.

(5) A preliminary constraints list for the next scheduled major test will also be defined at the post-test critique.

This procedure is now in effect.

Finding No. 4

Finding.-Items were placed on board the Spacecraft during preparation for the Space Vehicle Plugs Out Integrated Test which were not documented by quality inspection records.

Determination.-Procedures for controlling entry of items into the Spacecraft were not strictly enforced.

Responsible Organization.-KSC.

Action.-Apollo Preflight Operations Procedure No. 0-203, titled Access to the Command and Service Modules (CSM) was released on May 22, 1967. This procedure provides instructions and outlines responsibilities for controlling ingress egress of personnel and/or materials to the CSM, and it establishes the requirements for a NAR Q&RA Crew Compartment and Service Module Monitor. This procedure clearly defines the requirements for the maintenance of the ingress/egress log to include all personnel and each item by name, quantity, part number and work authorization. It further defines the type of equipment and materials which must be accepted in accordance with MSC specification MC 999-0058, titled Approved Materials for use in the Apollo Spacecraft.

NASA KSC Quality Control personnel will verify and enforce contractor performance relative to controlling ingress/egress as part of their periodic inspections of the CSM.

This procedure is now in effect.

Panel No. 2

Finding No. 1

TEST ENVIRONMENTS

Finding. All crew compartment equipment was not tested to be explosion proof.

Determination. There was insufficient testing of possible ignition sources.
Responsible Organization.—MSC.

Action. Numerous ignition tests have been conducted in support of the accident investigation and as part of the ongoing Apollo program engineering, tests and evaluation efforts. The results of the latest 11 ignition tests which were conducted as part of the AS204 Review Board activity, but released subsequent to the board's report, are described under Panel Finding 9-a(2).

All anomalies during the spacecraft manufacturing, certification and test operations which could represent possible explosion sources will be completely reviewed prior to acceptance of the spacecraft by NASA and corrective actions taken immediately. This is discussed under Panel Finding 2-3.

All equipment requiring electrical connection or disconnection that could be a potential fire-initiation or explosion hazard is capable of being turned off before the need for disconnection. This is discussed under Panel Finding 2-6.

Extensive reviews of electrical wiring design, fabrication, and installation were conducted to determine that the wiring is not an ignition source. This is discussed under Panel Finding 9-a(8).

Tests were conducted to determine the effect of water/glycol on connectors. These tests, discussed under Panel Finding 8-3g disclosed no evidence of electrical breakdown or degradation due to immersion in water/glycol.

Tests and evaluations were conducted on all spacecraft circuit breakers and other electrical equipment such as power inverters to ensure they can comply with new stringent size/load/circuit protection design criteria. These are discussed under Panel Finding 9-a(5).

Certification testing of equipment continues as part of the continuing program actions to ensure that all equipment anomalies that can potentially cause a fire hazard or explosion are identified and corrected.

Finding No. 2

Finding.-Crew compartment equipment of C/M 012 was exposed to water/ ethylene glycol contamination. Untested cleaning techniques were employed for that equipment discovered to be wet.

Responsible Organization.—MSC.