Disaster by Design/Safety by Intent #4
Disaster by Design
If you had a dollar for every foot of pipe—or even just a quarter for every three inches of pipe—used in the nation’s nuclear power plants, you would probably not be reading this post. That chore would be delegated to one or more of your many minions.
Pipes at nuclear power plants carry cooling water to the reactor vessel and spent fuel pool, transport steam to the main turbine, provide hydrogen gas to cool the main generators, supply fuel and lubricating oil to the emergency diesel generators, maintain the fire sprinklers ready to extinguish fires, and numerous other vital functions. Given so many pipes, a success rate of 99.99%—remarkably similar to a failure rate of one broken pipe out of ten thousand pipes—would result in lots of piping failures.
The Electric Power Research Institute’s report revealed lots of piping failures at U.S. nuclear power plants between 1961 and 1997 (Fig. 1). The non-leaking failures are identified by inspections indicating that safety margins had been compromised, forcing the pipes to be replaced before they leak. The leaking failures are identified by puddles on the floor or other obvious signs, again forcing pipes to be replaced.
Fig. 1 (Source: Electrical Power Research Institute)
The Electric Power Research Institute’s report identified numerous reasons why pipes break (Fig. 2). MIC under corrosion stands for microbiologically induced corrosion—tiny little bugs that eat metal. Pipes can be designed wrong, installed wrong, or weakened via an array of methods during use.
Fig. 2 (Source: Electrical Power Research Institute)
Whatever’s Inside the Pipes Needs to Stay Inside
Early in my career, I came across this parody of a design specification for piping (Fig. 3).
Fig 3 (click to enlarge) (Source: Dave Lochbaum)
Real design specifications provide the details necessary to conform to regulatory requirements and building codes. Layered on top of the regulations, codes, and design specifications are an equally voluminous set of initial and recurring tests and inspections designed to ensure one thing—whatever is inside the pipes stays inside.
The initial post in this Disaster by Design series described threats to nuclear power plants from flooding. Several of the incidents involved piping failures that flooded parts of the plant to disable or jeopardize safety equipment. Broken pipes can drain water out of places it should be. Piping failures can therefore undermine safety by putting water places it should not be and by removing water from where it needs to be. This post complements the initial post by describing some incidents of the latter variety.
Draining Water From Where It Needs to Be
Dresden Nuclear Plant
Fission Stories #65 described the January 25, 1994, discovery by workers at the Dresden nuclear plant in Illinois of 55,000 gallons of water in the basement of the Unit 1 containment building. The Unit 1 reactor had been licensed to operate in September 1959 and was permanently shut down in October 1978. The Unit 1 containment building had not been heated since the winter of 1988/1989. The unheated building allowed stagnant water inside a section of service water system piping to freeze and expand, rupturing the pipe. The unheated building contained the spent fuel pool and its 560 irradiated fuel bundles. The NRC’s investigation into this event identified the potential for a water-filled pipe connected to the spent fuel pool to also freeze and rupture, draining water from the spent fuel pool and uncovering the upper few feet of the fuel bundles. Because the spent fuel had not been inside an operating reactor core for at least 15 years, it had cooled sufficiently such that the partially uncovered fuel bundles would not have overheated. But water also serves to shield radiation. Had the spent fuel pool’s water level dropped, the radiation dose on the refueling floor was estimated to cause a fatal exposure in considerably less than an hour.
Browns Ferry Nuclear Plant
On August 14, 1984, workers tested the core spray system for the Unit 1 reactor at the Browns Ferry nuclear plant. The core spray system is installed to provide low pressure makeup to the reactor vessel in event a pipe ruptures and drains water. The core spray system consists of two redundant loops with each loop having two motor-driven pumps that transfer water from the suppression pool (also called the torus) or the condensate storage tank into the reactor vessel. The loop involved in this event is shown in Fig. 4.
Fig. 4 (click to enlarge) (Source: Tennessee Valley Authority)
The test was conducted with the reactor at 100% power. During the test, workers installed jumpers on electrical circuits to simulate accident conditions (e.g., low pressure and low water level in the reactor vessel and high pressure inside the containment) to see whether the pumps would automatically start. The water from the pumps would recirculate back to the suppression pool via the test line. At least that was the plan.
This test procedure specified that the electrical breaker for valve 75-25 was to be opened to prevent it from opening during the test as it would during a real accident. With valve 75-25closed and valve 75-22 open the piping would direct the flow back to the suppression pool. But an operator skipped that procedure step and failed to open the electrical breaker. Because of this mistake, valve 75-25 actually opened during the test.
The reactor was operating at more than double the pressure the core spray system piping was designed to withstand. Valve 75-25 being inadvertently opened should not have caused significant consequences because of the check valve installed in the piping between the valve and the reactor vessel. Check valves are designed to allow flow in only one direction—from the core spray system into the reactor vessel—and to prevent any flow in the outward direction. But the check valve had been improperly assembled and was then mis-wired to appear in the control room as if it was closed. The mis-positioned check valve and the inadvertently opened 75-25 valve allowed high pressure reactor cooling water to flow into the core spray system piping. The relief valve, designed to open at 500 pounds per square inch (psig) pressure and protect the core spray system piping from excessive pressure did just that, discharging reactor cooling water to the radwaste system.
The elevated pressure inside the piping also sprayed water from the seals on the core spray pumps. Several workers responding to the event were contaminated by water spraying, ironically, from the core spray pumps. The paint on the piping between valve 75-25 and the relief valve reached about 400°F and began smoking. Nuclear plants are designed such that a single malfunction of a component or a single failure by a worker cannot compromise safety margins. Here, a pre-existing component malfunction (the check valve) and a subsequent worker miscue (the unopened breaker) combined to bend but not break low pressure piping. Had the piping broken, it would have opened a large hole for water to drain from the reactor vessel while at the same time depriving the plant of a primary means of refilling the reactor vessel with water.
Surry Nuclear Plant
On December 9, 1986, the 18-inch diameter pipe supplying water to main feedwater pump A for the Unit 2 reactor at the Surry nuclear plant in Virginia ruptured. This feedwater pipe provided water to the steam generators where it absorbed the heat produced by the reactor core. The pipe contained water at 370°F and pressurized to 450 pounds per square inch. The water flashed to steam as it jetted from the broken pipe ends, scalding eight workers installing insulation nearby. Four workers died from their injuries. The steam also tricked the fire detection system into actuating the fire suppression systems. Water from fire sprinklers shorted out computer card readers at locked doors, impeding responders trapped on the wrong side of doors that could not be opened with their keycards. Emergency systems unaffected by the broken pipe protected the reactor core. Investigation revealed that the pipe, which was installed with a wall thickness of 1/2-inch, had thinned to about 1/4-inch by erosion from the water rushing through it. The section that ruptured was an elbow where the pipe bent, forcing an accelerated erosion rate.
Mihama Nuclear Plant
A 22-inch diameter pipe in the condensate/feedwater system ruptured on August 9, 2004, at the Mihama nuclear plant in Japan for the same reason—internal erosion of the pipe’s wall by flowing water until it became too thin and burst open. The water flashed to steam upon leaving the broken pipe ends, scalding 11 workers in the vicinity. Five workers died from their injuries. The emergency systems safely shut down the reactor.
Oyster Creek and Dresden Nuclear Plants
Fission Stories #162 described other events where pressurized hot water flashing to steam caused problems. Following unplanned automatic shutdowns of the Oyster Creek nuclear plant in New Jersey on June 12, 1985, and the Dresden Unit 3 reactor on September 19, 1985, malfunctions allowed reactor cooling water to flow for at least twenty minutes through valves that were supposed to be closed into the reactor building sumps. The hot water flashed to steam. The steam set off the fire sprinklers at Oyster Creek and contaminated the lower three levels of the reactor building at Dresden Unit 3. So, not only was cooling water leaving the reactor vessel, it was causing problems in the reactor building housing all the emergency core cooling system pumps designed to mitigate the inventory loss.
LaSalle Nuclear Plant
On May 27, 1985, a security guard at the LaSalle nuclear plant in Illinois notified a supervisor in operations about “water bubbling out of the ground near the Off Gas Filter Building.” The Unit 2 High Pressure Core Spray pump had been operating for several hours in test lineup returning flow to the condensate storage tank to improve the water quality of that tank. Operators stopped the pump and closed valves to isolate the underground piping between the pump and the tank. Subsequent investigation revealed that the pipe had ruptured due to biological corrosion and leaked 200,000 gallons before being isolated.
Oyster Creek Nuclear Plant
Fission Stories #29 described how 133,000 gallons drained from the condensate storage tank at the Oyster Creek nuclear plant in New Jersey in September 1996 during a refueling outage. Workers installed a temporary pump and piping system during maintenance work on the normal system. The next day, the operators began investigating why they had to refill the 275,000 gallon condensate storage tank more often than usual to maintain its required inventory. The condensate storage tank has a minimum allowable inventory because its water is the preferred supply for several emergency core cooling systems. The operators found an opened valve that was supposed to be closed. The mis-positioned valve acted much like a broken pipe by draining water from the condensate storage tank into Barnegat Bay. A daily balancing of water supply and demand had identified the problem several hours prior to the operators’ finding the mis-positioned valve, but the worker performing the accounting attributed the missing tens of thousands of gallons of water to a typographical error and adjusted the numbers until everything matched; except for the slightly radioactive water illegally released into the bay.
Davis Besse Nuclear Plant
Fission Stories #131 described the March 2002 discovery by workers at the Davis-Besse nuclear plant in Ohio that a crack in a pipe allowing a control rod inside the reactor vessel to be connected to and manipulated by its electric motor outside the vessel had been leaking cooling water from the reactor for as long as six years. Making the loss of cooling water worse was the fact that it was borated water. When the leaked water evaporated, it left behind boric acid that proceeded to eat its way through more than six inches of the reactor vessel’s metal wall exposing the thin (less than a quarter-inch thick) layer of stainless steel applied to the inner side of the reactor vessel. The picture in Fig. 5 looks down at a section removed from the reactor vessel showing the 4-inch diameter hole where the pipe penetrated the vessel and the adjacent damage caused when the leak dropped acid. Researchers at the Oak Ridge National Laboratory estimated that the widening hole would have reached the bursting point within as little as 60 more days of reactor operation. Had the degraded reactor vessel failed, the result could have been worse than Three Mile Island but not as bad as Chernobyl—a good safety tip is never to put yourself in a situation where those events are your bookends.
Fig. 5 (Source: NRC)
Byron Nuclear Plant
On October 19, 2007, workers brushing away rust on the outer surface of a cooling water pipe at the Byron nuclear plant in Illinois poked a hole in it. The pipe, part of the essential service water (ESW) system, transports warm water from plant equipment to one of two mechanical draft cooling towers where it is cooled by air flow. The cooled water is returned to the plant to cool equipment protecting the reactor core and spent fuel pool from overheating. The 24-inch diameter pipe is designed to withstand earthquake forces, yet was broken by a worker with a wire brush.
Fig. 6 (Source: NRC)
The NRC’s Special Inspection Team sent to the site to examine this near-miss found that the pipe was originally specified to have a wall thickness of 0.375 inches. On June 14, 2007, workers measured the wall thickness of the pipe as thin as 0.124 inches and 0.122 inches. The response was to revise the acceptance criterion down to 0.121 inches. On October 10, 2007, workers measured the pipe’s wall thickness to be as little as 0.085 inches. The response was to revise the acceptance criterion down to 0.06 inches. On October 17, 2007, workers measured the pipe’s wall thickness to be as little as 0.047 inches. The response was to revise the acceptance criterion down to 0.03 inches—less than one-tenth of the thickness originally specified. Two days later, the thinned pipe broke as rust (i.e., its only remaining wall) was brushed away. To the owner’s credit, this time the response was NOT to reduce the acceptance criterion down to 0.000 inches or less. They replaced hundreds of yards of worn out piping. Had there been an accident at Byron, this vital cooling water system might have been irrigating the grounds instead of cooling emergency equipment.
Big Rock Point Nuclear Plant
The NRC described a broken pipe at the Big Rock Point nuclear plant in their annual report to the U.S. Congress on abnormal occurrences in 1998. The backup system to shut down the reactor core in event the primary system fails features a tank filled with sodium pentaborate solution. When injected into the reactor vessel, the boron in this solution absorbs neutrons to interrupt the nuclear chain reaction and shut down the reactor. In March 1998 after the reactor was permanently shut down, workers were unable to pump the solution out of its storage tank. They found the pipe was completely severed inside the tank and estimated that the pipe had broken at least 13 years ago. The plant therefore operated for the last third of its 39-year life with this backup safety system disabled by a broken pipe—as valuable as a car’s air bag with a hole in it.
Safety by Intent
The table above from the Electric Power Research Institute indicates that 1,816 failures were identified by testing and inspection at U.S. nuclear power plants between 1961 and 1997 while 2,247 failures were found after pipes had leaked.
This data reinforce a theme too often appearing in nuclear safety posts to our All Things Nuclear blog—testing and inspection efforts are less effective than they need to be. A federal regulation requires that plant owners have extensive testing and inspection programs that find and fix safety problems in a timely and effective manner. If compliance with this regulation were fact rather than fiction, the data should show more piping failures are found via tests and inspections than by puddles on the floor.
The NRC must figure out why testing and inspection efforts are violating federal safety regulations by failing to find and fix piping failures in a timely and effective manner.
—–
UCS’s Disaster by Design/Safety by Intent series of blog posts is intended to help readers understand how a seemingly unrelated assortment of minor problems can coalesce to cause disaster and how addressing pre-existing problems can lead to a more effective defense-in-depth protection.