Pitched Unfairly

Incident review / Manned Spacecraft Center, Houston

Houston, we've had a post-mortem.

55:55:42LovellHouston, we've had a problem.

Incident
AS-508
Detected
55:54:53 MET
Severity
Crew safety
Status
Resolved
Format
Blameless

01 / Summary

One tank failed. Three crew came home.

Apollo 13 Review BoardTL;DR

What happened
At 55:54:53, oxygen tank 2 in the service module failed and took tank 1 down with it.
Impact
Lunar landing cancelled. Three crew lived four days in a lander built for two people and two days.
Root cause
Heater switches rated for 28 V met a 65 V ground supply, welded shut, and cooked the wiring.
Resolution
Lifeboat ops in the lunar module. Splashdown at 142:54:41, April 17, 1970.

02 / Impact

We lost the landing. We kept the crew.

Lunar landing
CancelledFra Mauro, not visited
Distance at failure
205,000 miFrom Earth, outbound
Lifeboat load
3 crew, 4 daysIn a lander rated for 2 and 2
Time to resolve
87 hours55:55 to 142:54 MET

3 /3

Crew recovered: Lovell, Swigert, Haise

03 / Timeline

143 hours, logged to the second. Scrub any of them.

MET142:54:41

Fri Apr 17 1970, 1:07 pm EST

Splashdown142:54:41

Mid-Pacific, 3.5 nautical miles from USS Iwo Jima.

04 / Detection

We detected the failure in 27 seconds. The odd reading came nine hours sooner.

O2 tank 2 quantitySchematic, MET hours

46:40:02Gauge pegs off-scale high 55:54:53Bang 100%0
Time to detect
27 sBang to first crew report
Time to diagnose
1.5 hTank 1 leak judged unstoppable
Time to mitigate
2h 45mLifeboat live by 58:40
Anomaly age
9h 15mGauge fault, unexplained

05 / Response

The lander became the lifeboat.

58:40
Aquarius powered up as the lifeboat
20%
Lander power, cut to a fifth of normal
4
Course corrections on a lander engine
Earth Moon 55:55Tank 2 fails 61:30Back on free return 79:28Burn: home 9 h sooner 105:18Course correction 137:40Final correction 142:54Splashdown

05 / Response, continued

Spare filters: square. Sockets: round.

0.1 mm Hg

Fix installed at 93 h

LM cabin CO2, mm HgCurve illustrative

Mission rule limit, 7.5 93 h Saturated, about 100 h
Spare filterCommand module, square
The mailboxBags, a manual cover, duct tape
SocketLunar module, round

06 / Root cause

Five whys, from the bang back to a 1965 spec change.

  1. Why 1

    Why did tank 2 fail at 55:54?

    The crew stirred it. Damaged fan wiring shorted, and its Teflon insulation ignited in pure oxygen.

    1970
  2. Why 2

    Why was the wiring damaged?

    Before launch, the tank heaters ran for 8 hours straight. Parts of the heater tube reached about 1,000 F.

    Mar 1970
  3. Why 3

    Why did the heaters run for 8 hours?

    Tank 2 would not drain after the countdown test, so the team boiled the oxygen off. Its fill tube was likely knocked loose when the tank was jarred in 1968.

    Oct 1968
  4. Why 4

    Why didn't the thermostats cut the heat?

    They were rated for the spacecraft's 28 V. The ground supply was 65 V. The first time they opened, they welded shut.

    Mar 1970
  5. Why 5

    Why were 28 V switches on a 65 V tank?

    A 1965 spec change moved the heaters to 65 V. The switch spec never followed, and no test cycled the switches under load.

    1965

Root cause A spec changed and a part didn't. The board: a serious oversight in which all parties shared.

07 / Retro

Some of it was skill, some of it was process, and some of it was luck.

Went well

  • Lifeboat checklists already existed. Mission Control stretched them to three crew and four days.
  • The ground team built, tested and read up a CO2 fix before the limit hit.
  • Swigert joined the crew two days before launch and flew it like he had trained for months. He had.

Went wrong

  • The tank 2 gauge failed at 46:40. Nobody had to explain it.
  • The heater temperature sensor read to 85 F. The heater reached about 1,000 F.
  • Prelaunch reviewers never saw an 8-hour heater run as a risk.

Got lucky

  • It happened outbound, with the lunar module attached and fully stocked.
  • Tank 2 was hazardous from the pad onward. It held for 56 hours.
  • When Aquarius was let go, it had about 4.5 hours of power left.

08 / Action items

Nine recommendations. Every one has an owner.

ShippedApollo 14 waited three months for these

#ActionOwnerType
01 Get wiring and unsealed motors out of the oxygen. Minimize Teflon near it. Apollo Program Prevent
02 Requalify the modified oxygen system, operations included. Apollo Program Prevent
03 Stop one alarm masking another. Add a second alarm level in Mission Control. Apollo Program Detect
04 Stock both spacecraft for lifeboat mode. Apollo Program Mitigate
05 Finish the special tests. Look hard at the lunar module power anomalies. MSC Learn
06 At launch reviews, show every prior anomaly, with a subsystem expert in the room. Apollo Program Process
07 Re-examine every NASA system holding high-density oxygen. NASA Prevent
08 Research ignition and combustion in strong oxidizers. Write new standards. NASA Learn
09 Reassess subsystems down to vendor level. Strengthen the teams that own them. MSC Process

09 / Follow-ups

Status: resolved. Crew: home.