Disaster by Design/Safety by Intent #20
Disaster by Design
In nuclear power safety, training has nothing to do with steam engines, diesel engines, passenger cars, freight cars, and cabooses. In nuclear power safety, training encompasses education, experience, and qualifications seeking to ensure that workers know what to do, and what not to do. Training is not just a good idea, it’s the law.
Fig. 1 The Kawakawa Steam Train (Source: Sids1)
Nuclear training is challenging. Among many complications, training must help workers handle a seemingly endless combination of configurations. It is impossible to develop procedures and provide associated training that covers all the various scenarios that workers will encounter. Over time, and with considerable trial and error, nuclear training has matured to prepare workers much better for the tasks they get assigned. The following examples show that nuclear training remains a work in progress.
Brunswick Unit 2
Workers attempting to restart the boiling water reactor on November 15, 2011, following an 11-day outage encountered indications that cooling water was leaking from the reactor vessel. As they increased the reactor power level, the associated higher pressure increased the cooling water leak rate. The increasing leak rate prompted the operators to manually scram the reactor early the following morning.
As described in Fission Stories #80, workers were able to hand turn several of the nuts fastening the reactor vessel’s head onto the reactor vessel. The reactor vessel’s head had been removed during the outage and improperly re-installed. There were two different methods for assuring proper fastening of the reactor vessel’s head, but the owner had stopped training workers on either method more than a decade earlier. The untrained workers misunderstood what the two monitoring systems were indicating and failed to properly tighten the nuts. During the reactor startup, the increasing pressure lifted the head off the reactor vessel and allowed cooling water to escape.
Browns Ferry Unit 3
The boiling water reactor had shut down on April 27, 2011, when a nearby tornado knocked down a support tower for power transmission lines, disconnecting the nuclear plant from its electrical grid.
The reactor remained shut down for maintenance. Residual Heat Removal (RHR) pump 3B (see Fig 2) was running in shutdown cooling mode to cool the reactor water. A connection to one of the two recirculation loops inside the containment supplied water to the RHR pump. The RHR pump sent the water through a heat exchanger where it was cooled by water from the ultimate heat sink (in this case, the Tennessee River.) The cooled water was returned to the same recirculation loop which carried it back into the reactor vessel.
Fig. 2 (Source: UCS adapted from NRC graphic)
On May 12, workers were replacing electrical relays in the primary containment isolation system (PCIS). The PCIS includes sensors that monitor plant parameters like containment pressure and reactor vessel water level and relays that automatically close valves in pipes that pass through the containment’s walls when indications of trouble are detected. The PCIS serves to close pathways that drain cooling water from the reactor as well as close pathways for potentially radioactive water to get out of containment.
At 6:25 pm on May 12, alarms in the control room alerted the operators that RHR pump 3B had stopped running and that some of the PCIS-controlled valves had closed. Workers replacing relays in the PCIS lifted the wrong wire, causing the valves in the RHR shutdown cooling lines to close. Deprived of its source of water, RHR pump 3B automatically turned itself off. Workers reconnected the wire. The Operators reset PCIS, re-opened the valves, and restarted RHR pump 3B. In the 40 minutes that shutdown cooling was off, the reactor water warmed to 122°F from 112°F. The time for the reactor water to boil, had shutdown cooling not been restored in time, was calculated to be 4.13 hours.
The instructions for the PCIS relay replacements did not match the actual plant conditions. The instructions were for conditions where RHR was not running in shutdown cooling mode. The owner determined that a contributing cause was “There was inadequate training on electrical fundamentals in the area of plant wiring and plant configuration.” The wiring diagrams for the PCIS relays did not match what the workers saw out in the plant—yet when confronted with more wires than appeared in the drawings, the workers just removed all the wires instead of questioning the discrepancy.
Perry
In July 2007, workers removed a 3-foot by 7-foot section of metal grating inside the containment of the boiling water reactor to allow one of the two recirculation pump motors to be replaced. On August 27, 2007, an operator conducting routine rounds discovered that the grating had not been properly re-installed. Fasteners to hold the grating in place had not been reconnected. If the nearby recirculation pipe had ruptured, the force of the water jetting from its broken ends could have dislodged the grating. The dislodged grating could have struck the connection where several of the emergency core cooling pumps draw their makeup water, disabling one of more of them. The owner determined that the instructions for the work lacked sufficient details and that workers received inadequate training on maintaining the plant’s configuration.
Millstone Unit 2
The pressurized water reactor was operating at 100% power on December 21, 2006, when workers discovered that scaffolding had been installed over a High Energy Line Break (HELB) blowout panel. Pipes containing steam or water at elevated pressures are termed high energy lines. If such a pipe were to rupture, the fluid jetting from the broken ends can damage equipment in the vicinity. If the pipe ruptured inside a small room, the discharged fluid could pressurize the room, causing its structural failure. The HELB blowout panel in this case was designed to open in event of a pipe rupture to relieve pressure buildup and protect structures.
The scaffolding could have limited the movement of the HELB blowout panel. Had a pipe ruptured, the pressure buildup could have caused a wall to fail, admitting steam into the adjacent room housing the auxiliary feedwater pumps. The auxiliary feedwater pumps are standby emergency pumps that provide makeup cooling water in event of an accident. But the pumps are not designed to operate in a steam environment, so the pumps were declared inoperable. The scaffolding was removed and the pumps declared operable about three and a half hours later.
The owner determined that procedures covering scaffolding lacked details on “do’s and don’ts” and that workers installing scaffolding received inadequate training on potential consequences.
DC Cook Unit 1
Workers inspecting the containment divider barrier seal at the pressurized water reactor on October 5, 2006, discovered that parts were missing. The reactor has an ice condenser containment featuring bays filled with baskets of ice. Steam discharged from a ruptured pipe inside containment would push open hinged doors into the ice condenser bays. The ice would cool the steam and convert it back into water, limiting the pressure rise inside containment. The containment divider protects against steam bypassing the ice condenser and overpressurizing (i.e., failing) the containment structure.
A check of the records revealed that the missing parts had been identified back in November 1998, but had not been replaced. Because it had happened so many years ago, the owner was not able to determine why nothing had been done about the problem, but attributed it to inadequate training.
Perry (second appearance)
On December 23, 2004, both of the recirculation pumps on the boiling water reactor unexpectedly shifted from high speed to low speed. The pumps slowing down reduced the reactor power level from 100 percent to about 44 percent within seconds.
As described in Fission Stories #8, boiling water reactors can become unstable when operating at high power and low flow conditions. As suggested by their name, water flowing through operating boiling water reactors boils. Steam bubbles tend to reduce power. The power reduction decreases the number of steam bubbles, which increases power. The power increase produces more steam bubbles, which reduces power. And so on. This effect, termed power oscillations because the reactor power level swings up and down on a two-second frequency, is more pronounced at the high power and low flow conditions that Perry encountered following the drop in pump speeds. Consequently, procedures direct the control room operators to take immediate steps to get reactors out of high power and low flow conditions as quickly as possible.
That did not happen at Perry. Three minutes later, an alarm alerted the operators that the reactor was unstable. The alarm re-sounded again in three minutes, again a minute later, and a fourth time two minutes later. Shortly after the fourth alarm, the reactor’s instability triggered an automatic shut down.
The owner concluded that inadequate training on reactor instability events and the system installed at Perry to guard against it.
Safety by Intent
When I worked at nuclear plants, I commented more than once that I never made the same mistake twice, because I was too busy making initial mistakes to find time for repetitions. Hopefully, there’s a little hyperbole involved. But the point was that workers tackle assignments governed by a myriad of procedures and undertaken across widely varying situations.
Nuclear training doesn’t even try to take all these inputs and provide workers with black and white guidance on how to handle every situation. Nor should it. Instead, nuclear training strives to make as many situations as possible black and white for workers, and equip them with the skills necessary to successfully navigate the gray areas.
—–
UCS’s Disaster by Design/ Safety by Intent series of blog posts is intended to help readers understand how a seemingly unrelated assortment of minor problems can coalesce to cause disaster and how effective defense-in-depth can lessen both the number of pre-existing problems and the chances they team up.