Command and Control

Disaster by Design/Safety by Intent #17

Disaster by Design

Command and control is often used to describe the authority of military leaders in directing armed forces in battle. It can also refer to senior managers at nuclear power plants and the resources they command and control to fend off safety challenges.

Faulty intelligence, or flawed situational awareness, undermines command and control when leaders have the wrong understanding of hazards and/or response capabilities.

A recurring situational awareness problem affecting nuclear safety involves indications of plant conditions from monitoring instruments. A contributing factor in the meltdown of the Unit 2 reactor core at the Three Mile Island nuclear plant outside Harrisburg, Pennsylvania on March 28, 1979, was the operators turning off the emergency makeup pumps. An indicator in the control room led them to believe the system had too much water and to turn off the pumps to solve that perceived problem. Actually, the indicator was reading falsely high—the system had too little water. Their response—proper for their understanding of the situation but entirely improper for the real condition—transformed a minor problem into a nuclear nightmare.

Three examples among too many instances since that accident show that nuclear safety remains susceptible to managers relying on faulty indications and either taking the wrong steps or not taking the right steps in mitigate the problem.

Perry

As described in Fission Stories #88, workers testing the offgas system at the Perry nuclear plant northeast of Cleveland, Ohio on June 18, 1986, encountered a problem. The test checked the beds of charcoal that reduced the levels of radioactivity in gases being discharged to the environment during normal operation. Workers placed space heaters near the vaults containing the charcoal to warm the charcoal up to the minimum temperature of 150°F specified in the test procedure. There was a lot of charcoal to warm. By the morning of the following day, thermocouples showed the charcoal temperatures to have only risen to 112°F and 126°F. Over the next hour, the indicated temperatures quickly rose to 250°F—the maximum reading on their scales.

An instrument and control technician was dispatched to figure out why these two instruments had failed. The technician hooked a broader scale to the thermocouples and saw them both to be reading over 1,000°F. The technician also observed smoke in the area, which was attributed to insulation smoldering on the wires from the failed thermocouples.

Workers dug into blueprints and equipment manuals trying to figure out what was causing the thermocouples to fail.

The following morning, workers measured a temperature of 254°F on the outer surface of the charcoal vault. They finally realized that the thermocouples had not failed—the charcoal was burning. They stopped routing air flow through the charcoal beds and began purging them with nitrogen gas to put out the fire.

Seabrook

Fission Stories #137 described a situation facing workers at the Seabrook nuclear plant 13 miles south of Portsmouth, New Hampshire. The reactor’s operating license required the level of water in the cooling tower basin to be at least 42.15 feet above mean sea level. Two different gauges monitored the level and continuously sent level readings to the control room.

On November 2, 2012, the operators in the control room observed that the two instruments showed the water levels to be about a foot apart. Both readings should have been the same, but one instrument showed the water level to be above the minimum allowable level while the second instrument indicated that the level was too low. The operators chose to believe the instrument indicating that the level was too low had failed.

On December 7, 2012, workers determined just the opposite—the instrument showing that there was adequate water level had failed. The operators added water to the basin until the unfailed instrument showed that level was once again above the minimum level.

Millstone

Workers shut down the Unit 2 reactor at the Millstone nuclear plant in Waterford, Connecticut, on October 3, 2015, to enter a refueling outage. The following morning, the operators turned on the shutdown cooling system to complete cooling down the water circulating through the reactor core to less than 212°F. Shortly after turning on this system, the water level in the pressurizer dropped and the operators encountered problems keeping the water level high enough. The operators assumed the level in the pressurizer dropped due to “shrinkage” as the water temperature decreased.

Actually, the level dropped because a relief valve had opened in the shutdown cooling system and was allowing over 80 gallons per minute of reactor cooling water to flow into the Equipment Drain Storage Tank. Operators checked the water level in the Equipment Drain Storage Tank as a possible reason for the level drop in the pressurizer. But the tank’s water level was indicating -9 percent, below empty.

Actually, the tank’s water level was reading falsely low. It had been 52 percent before the shutdown cooling system was turned on. The flow into the tank from the stuck open relief valve increased the tank’s pressure from about 20 pounds per square inch to over 260 pounds per square inch. The higher pressure pushed down on the float in the water level gauge, causing it to show a falsely low level.

About three hours later, the operators finally realized there was a loss of coolant accident in progress. Within ten minutes, they took the steps necessary to stop the leak. An estimated 16,570 gallons (or about 63 tons) of cooling water leaked from the reactor during that time.

Safety by Intent

Nuclear power plant instruments have failed and falsely indicated problems when conditions were normal. The fail-safe aspect of the reactor protection system has resulted in several automatic reactor shut downs due to instrument failures. For example:

• Waterford (LA): The reactor automatically shut down on October 3, 2015, due to an indication of reactor cooling problems. Workers determined that a failed resistor on a circuit card caused a false indication of a problem when no such problem actually existed.

• Hope Creek (NJ): The reactor automatically shut down on September 28, 2015, due to an indication of a reactivity problem. Workers testing the reactor protection system inadvertently introduced a false signal of high pressure in the reactor vessel. Per design, the plant responded to indicated high pressure condition by rapidly shutting down the reactor.

All too often (meaning more, way more, than once), instruments indicating potential problems are dismissed by workers as having failed. The Seabrook incident is a classic example—operators received an indication from one instrument that a safety limit had been violated and an indication from a different instrument showing that the safety limit was being met. More out of convenience than conservatism, they opted to believe the “good news” and reject the “bad news.”

Safety dictates that all indications of potential problems be taken seriously and only dismissed when solid homework shows them to be invalid.

—–

UCS’s Disaster by Design/ Safety by Intent series of blog posts is intended to help readers understand how a seemingly unrelated assortment of minor problems can coalesce to cause disaster and how effective defense-in-depth can lessen both the number of pre-existing problems and the chances they team up.