[41] Apollo 12 lifted off the pad at 11:22 a. m. e.s.t. on November 14, 1969. At 36.5 seconds after lift-off, lightning struck the command and service module (CSM), disconnecting all three fuel cells from the main buses and placing the main loads on two of the three batteries which ordinarily supply reentry power (fig. 5-1). Fuel cell disconnect flags popped up, and caution and warning lights winked on to alert the crew. With the decrease in the main bus power, the primary signal conditioning equipment ceased operating as it is meant to do when main bus voltages drop to approximately 22 volts. The ground simultaneously lost telemetry lock. At first, flight controllers thought the plume of ionized rocket-exhaust particles had blacked out the telemetry signal. However, they abandoned this theory when the crew reported the warning lights.
The primary signal conditioning equipment controls most electrical-power measurements; therefore, there was little information with which to diagnose the trouble. At 52 seconds after lift-off, the crew reported losing the spacecraft platform. At 60 seconds, the ground locked on to the telemetry signal again, and the CSM electrical and environmental systems engineer, John W. Aaron, asked the crew to switch to the secondary signal conditioning equipment to get additional insight into the electrical system. At 98 seconds, the crew made the switch, restoring all telemetry. Aaron then noted from his data display that three fuel cells were disconnected and requested the crew to reset them. Fuel cells 1 and 2 went back on the line at 144 seconds; fuel cell 3, at 171 seconds. Main bus voltages rose to approximately 30 volts, and all electrical parameters returned to normal.
Throughout the entire launch the Saturn launch vehicle performed normally. The spacecraft entered the proper orbit, and the crew and ground began preparing for translunar injection.
The quick response to the Apollo 12 outage came about not as a result of blind luck but of careful planning, training, and development of people, procedures, and data display techniques by those responsible for flight control.
The flight control organization devotes a majority of its time and resources to careful pre-mission planning and detailed training. This pre-mission preparation culminates in simulations of critical phases of the mission with the flight crew. These simulations prepare the flight controllers and the flight crew to respond properly to both normal and contingency situations.
[43] Following one of the basic [flight Control Division philosophies, operations personnel take part in planning a mission from its conception through its execution. They participate in two areas where their operational experience contributes greatly: (1) in the early stages of mission design and (2) in setting basic design requirements for various spacecraft systems. As a result, both spacecraft hardware and mission design have optimum operational qualities.
The true operational phases of the Flight Control Division begin after the spacecraft design reaches completion and NASA has committed itself to constructing flight vehicles and launch facilities. The development of mission operations starts after sufficient information about the detailed design of the space vehicles becomes available. The development takes 2 years and is divided into four phases, as shown in figure 5-2: mission development, detailed planning, testing and training, and real-time operational support of a flight.
The mission development phase begins approximately 2 years before the first launch. First comes the establishing of the conceptual operational guidelines for the flight. The guidelines for the Apollo Program were developed during Project Mercury and the Gemini Program.
[44] Throughout the Mercury and Gemini flight programs, teams of flight controllers at the remote tracking stations handled certain operational responsibilities delegated to them somewhat independently of the main control center. As the flight programs progressed, the advantages of having one centralized flight control team became more apparent. By the advent of the Apollo Program, two high-speed (2.4 kbps) data lines connected each remote site to the Mission Control Center (MCC), permitting the centralization of flight control there.
The spacecraft systems were of two functional categories: (1) electrical, environmental, and communications and (2) guidance and control and propulsion. A flight controller in the Mission Operations Control Room (fig. 5-3) of the MCC monitored each one. He received the backing of the Staff Support Room where every spacecraft system had its own man.
The flight dynamics team philosophy remained unchanged from earlier programs. The team would monitor the spacecraft trajectory and plan changes to it, monitor and manage the three spacecraft computers, and plan the return to earth.
The function of the Flight Director and his staff also remained unchanged. The Flight Director directs and coordinates the flight control team. He may, after analysis of the flight, take any action necessary to complete the mission successfully.
Following mission definition, flight control personnel participate in flight operations planning meetings. At this intercenter (Manned Spacecraft Center, Marshall Space Flight Center, Kennedy Space Center, Goddard Space Flight Center, and NASA Headquarters) forum, operations personnel can directly influence the first detailed mission design. As the mission design begins to take shape, they have an opportunity to analyze the possible effects of various failures on the successful completion of the mission. They can then suggest alternate mission designs or hardware changes (or both) which would improve chances for success. As the mission plan solidifies, the operations personnel identify certain guidelines and constraints forming boundaries within which the detailed mission planning will take place.
Concurrent with the development of the operations concepts and the mission guidelines and constraints, systems flight controllers begin to gather detailed systems information from the spacecraft manufacturers, The systems flight controllers get functional schematics and engineering drawings which they translate into a handbook of spacecraft systems. Flight control personnel and the flight crew use this handbook during the mission. On one page, like that reproduced in figure 5-4, each schematic shows all system or subsystem interfaces, together with the power sources, onboard and ground instrumentation, the various controls and displays necessary for the operation of that system, and pertinent notes on system performance. As a spinoff, preparation of the systems handbooks provides a chance to spot potential systems design [46] inadequacies. On completion of the systems handbooks, the long-lead-time items comprising the mission development phase come to a close.
The phase II detailed planning begins 6 to 9 months before launch when NASA assigns detailed mission objectives to the particular mission. The objectives are assigned to a particular phase in the mission time line, and flight control personnel work with the Apollo Spacecraft Program Office to establish their order of priority.
Flight controllers begin defining the specific data necessary to carry out the detailed mission objective and monitor the operational spacecraft systems. They do this parameter by parameter, assigning a priority to each. Programmers can then write the software required to provide these data to the flight controllers in a usable form.
Meanwhile, the mission rules begin to take shape. The rules specify in great detail how to conduct the mission in both normal and contingency situations. The final list constitutes a three-way agreement among flight control personnel, the flight crew, and management personnel.
Flight crew safety overrides all else. Then come into play complex tradeoffs between mission objectives and spacecraft design, the reliability and maturity of all elements associated with the conduct and control of the mission, the mission-objective priorities, and risk tradeoffs.
Mission rules fall into two distinct categories: (1) general guidelines formulated by the Flight Director and his staff and (2) detailed rules formulated by individual flight controllers in response to the general guidelines. One general guideline reads as follows:
During lunar module powered descent, if a systems failure occurs, a choice is available.
Early in powered descent when descent-propulsion-system-to-orbit capability is available (up to powered descent initiation plus 5 minutes), it is preferable to abort in flight rather than to continue descent. Redundant capability of critical lunar module systems is required to continue powered descent during this period.
During the remainder of powered descent, it is preferable to land and launch from the lunar surface rather than to abort. Only those system failures or trends that indicate impending loss of the capability to land, ascend, and achieve a safe orbit from the lunar surface, or impending loss of life-support capability will be cause for abort during this period.
This rule came about because up to 5 minutes after powered descent initiation, one can abort and reach orbit by using only the descent stage. One can keep the descent stage and use its consumables (oxygen, water, and batteries) if it takes a long time to rendezvous or if one loses the ascent consumables. After 5 minutes in powered descent, the descent propellant remaining could not return the craft to orbit. It then becomes more desirable to continue on and land. By landing and lifting off one revolution (2 hours) later, one gains sufficient time to analyze the malfunction, perform any system reconfigurations necessary, and perform a nominal ascent and rendezvous. Since the flight crew and the flight controllers emphasize nominal activities most, they know them best and always use them if possible.
[47] Detailed rules, such as the ones listed in figure 5-5, expand upon the broad philosophy of the general guidelines to cover single failures of all individual systems and subsystems throughout the spacecraft down to the individual instrumentation points necessary to making mission-rule decisions.
The specific format for mission rules has remained the same since Project Mercury. A short statement describes the condition or malfunction which may require action. The ruling follows. A third section of the rule contains applicable notes, comments, or standard operating procedures. Documents carefully delineate the reasoning behind each mission rule, including tradeoffs which may not be otherwise apparent.
Standard operating procedures receive as careful attention. They divide into interface procedures and personal procedures. Interface procedures include all flight control procedures involving more than two console positions. Together, the procedures comprise the Flight Control Operations Handbook (FCOH). Like the mission [48] rules, the FCOH procedures appear in a standard format and are as brief as possible without becoming ambiguous. Checklists, like the one shown in figure 5-6, are used whenever possible.
Personal procedures go into the Flight Controller Console Handbook prepared for each console position. This handbook describes the job of the flight controller manning that position and details the procedures he must follow. As does the FCOH, it uses, whenever possible, checklists of the kind shown in figure 5-7.
The testing and training phase has two main purposes: (1) integrating the flight control team and (2) testing the procedures and mission rules prepared for the specific mission. Six distinct techniques are used to train flight controllers for a mission. Several are used simultaneously. As described earlier, documentation development proved to be one of the most successful techniques. The system flight controllers gain an intimate knowledge of the hardware design and what it will and will not do through their long weeks and months of preparing the spacecraft systems handbooks. Likewise, the flight controllers learn the various ramifications of the mission rules by discussing the various merits of alternate courses of action. Preparing the procedural documents instills in the flight controller the capabilities and responsibilities of his position and the methods of using his position most efficiently.
Formal classes convene periodically to examine spacecraft systems in detail, to present basic orbital mechanics laws and principles, and to give instruction on the capabilities and limitations of the ground systems. Programmed instruction methods, as developed in 1965 by the Air Force, underwent slight modification for flight control training. Programmed-instruction courses covered all spacecraft systems and, ultimately, all functional job positions in various levels of detail, thus providing two benefits. The courses provided easily assimilated material, such as that shown in figure 5-8, which helped individual flight controllers understand the problems and responsibilities of other flight controllers and assisted in training controllers who could move rapidly from one job to another.
During the Gemini and Apollo Programs, cockpit-system trainers were developed for flight control training. These automated trainers allow the flight controllers to become familiar with the problems encountered by the flight crew. With the trainers, flight controllers identify, develop, and exercise critical crew procedures.
Through his support of program office design reviews, single-point failure reviews, and detailed mission requirement reviews (to name a few), the flight controller could associate directly with the designers, builders, and operators of the flight hardware. This often provides a rare insight to critical information that may affect all areas of planning and preparation of a mission.
Simulation began 2 or 3 months before launch. The entire flight control team took part. Artificially created spacecraft and network data flowed into the control center, providing real-time responses and displays as if a mission were actually in progress.
The first mission simulations worked with computer-generated mathematical models of space vehicles. These provided flight controller training in exercising documented procedures and mission rules in response to live data and a dynamic mission situation. Approximately 30 percent of the total simulation time goes into working with the mathematical model.
During the remainder of the time, controllers worked with the night crew spacecraft mission simulators. This promotes the integration of the flight control team and the flight crew team. Simulation personnel introduce spacecraft and Manned Space Flight Network problems and failures to test the documented procedures and mission rules. Sometimes the simulations lead to changes in them. The time spent on each period depends on its criticality and the amount of past experience available.
For an Apollo mission, simulations cover these periods of major activity: launch through translunar injection; lunar orbit insertion; lunar module activation, checkout, and descent; lunar surface operations; ascent through rendezvous; transearth injection; and reentry.
We found in the Apollo Program that having groups specialize in these periods gave better support. Thus, four teams divide the periods among themselves. Each gets to know critical mission phases better, but does not have to spend such long hours on the simulators.
The culmination of all this preparation follows lift-off. Throughout the Apollo Program, approximately 80 percent of all problems encountered in flight, whether large or small, had been previously discussed, documented, and simulated before the flight. This made choosing the correct course of action almost automatic. The remaining 20 percent of the problems readily yielded to the same logic, and decision-making procedures followed to arrive at pre-mission decisions. The logic of flight control decisions is diagramed in figure 5-9.
A functional organization has emerged which is flexible enough to meet unexpected problems, but is structured enough to provide continuity of operation from mission to mission. The basic principles of flight control are not unique to manned space night. They apply to any field where one can visualize malfunctions, document solutions, and rehearse the resulting actions. They could find use in any field where one monitors equipment or procedures by remote sensing devices. Application of the basic principles could increase efficiency in any field where one can write standard operating procedures. In the Apollo Program, they helped carry man to the lunar surface and bring him safely home again.
Home - NASA
Office of Logic Design
Last Revised:
February 03, 2010
Web Grunt: Richard
Katz
