Regulation and Nuclear Plant Safety #3
In July 2004, Nuclear Regulatory Commission (NRC) inspectors at the Waterford nuclear plant in Louisiana discovered that a portion of piping in a standby emergency system that would provide makeup water to cool the reactor in event of an emergency had been kept emptied of water, jeopardizing the ability to prevent core damage. This finding was shared with owners of similar reactors across the country. Days later, workers at the Palo Verde nuclear plant in Arizona discovered that sections of the emergency system piping for all three reactors was being deliberately emptied of water. The company tried arguing that there was no written requirement that water be maintained inside the emergency water makeup piping. The NRC disagreed and issued the company a yellow finding for the violations, the second most serious infraction in the agency’s color-coded system. The NRC also issued a $50,000 fine for an improper procedure change in 1992 that caused workers to deliberately drain water from this piping.
Water-less in Waterford
NRC inspectors at the Waterford nuclear plant outside New Orleans, Louisiana during the week of July 12, 2004, reviewed a report on a problem identified by workers on April 18, 1999. The problem was that air collected within piping of the containment spray system during normal operation. During an accident in which a pipe ruptures and drains cooling water onto the containment floor, the design initially calls for emergency pumps to automatically start and transfer makeup water from a large storage tank into the reactor vessel. Before this tank empties, workers re-position valves to have the pumps instead draw water from the containment sump, which collects the water spilled from the broken pipe. Following the swap-over, the emergency pumps would pull water from pipes partially filled with air.
The problem report had been dispositioned in 1999 as being acceptable as-is based on engineering judgement that the slope of the pipes and the low velocity of water flow through the pipes would enable air bubbles to travel against the flow and be released inside containment. When the NRC inspectors challenged the robustness of this assessment, the owner hired a consultant who conducted analytical modeling of the system during a postulated accident that showed the air within the piping would not prevent the safety function from being fulfilled.
The NRC inspectors noted that the reactor’s safety studies assumed that the piping was filled with water when the accident began and that another system had been installed at the plant for the purpose of keeping this piping full of water. The NRC issued a green finding, the least serious of the agency’s four color-coded sanction levels, for operating the reactor outside the bounds of its safety studies.
Equally Dry in Arizona
Workers at Waterford notified their counterparts at the Palo Verde nuclear plant west of Phoenix, Arizona on July 22, 2004, about the NRC’s discovery. On July 28, workers at Palo Verde determined that a significant portion of the suction piping for the containment spray, low-pressure safety injection, and high-pressure safety injection pumps for all three reactors was empty of water. These emergency pumps have two sources of water for use mitigating an accident. Initially, the pumps pull water from the Refueling Water Tank. The piping this tank and the pumps was filled with water, as was the section of piping to a check valve in the second water source—the containment sump.
The piping between the inside and outside containment isolation valves and between the outside containment isolation valve and the check valve held no water. A change made to a testing procedure on November 16, 1992, had workers close the two containment isolation valves and drain the water from these piping sections. When the volume of water in the Refueling Water Tank dropped to about the 10 percent level, the low-pressure safety injection pump would be turned off automatically and valve repositioned to supply water to the containment spray and high-pressure safety injection pumps from the containment sump.
The theory behind this design is that if the contents of the Refueling Water Tank do not restore the reactor vessel water level to the desired point, there must be a pathway for water to drain from the vessel. If so, that water will flow by gravity to the containment sump where it can be recycled through the reactor vessel to sustain adequate cooling of the reactor core. The high-pressure and low-pressure injection pumps supply makeup water to the reactor vessel; the containment spray pump causes water to be spray within the containment structure to reduce its pressure and temperature.
Fig. 1 (Source: Nuclear Regulatory Commission)
Coming Up Empty at Palo Verde, Again
By the afternoon of July 29, the engineering staff at Palo Verde concluded that the emptied piping sections could prevent the containment spray and high-pressure safety injection systems from performing their safety functions during an accident. (The low-pressure safety injection system was not affected because its pump gets turned off before suction from the containment sump through the empty pipes is established.) They entered the problem into the plant’s corrective action program.
On the morning of July 30, the operations department at Palo Verde learned about the problem from the corrective action report. That evening, the operations department determined that the containment spray and high-pressure safety injection systems could perform their safety functions provided that operators manually open the inside containment isolation valve during an accident. Opening this valve would re-fill the largest volume of the intentionally drained piping sections.
The owner notified the NRC about the problem on July 31. Between August 1 and 4, workers took steps to refill the emptied piping sections on all three reactors.
The NRC dispatched a special inspection team to Palo Verde to investigate the causes and corrective actions of this problem. The special inspection team was onsite August 23-27 and issued its report on January 5, 2005. The team made four findings: (1) operating the reactors with the piping sections drained of water contrary to assumptions in safety studies, (2) untimely notification of operations by engineering of a problem potentially affecting safety system operability, (3) inadequate evaluation of replacing automatic accident responses with manual actions, and (4) inadequate evaluation of a 1992 revision to a testing procedure that had workers drain the piping sections when the test was completed.
Palo Verde Pleads Its Case
The company contested the NRC’s findings and requested a meeting with the agency to present its case. That meeting was conducted in the NRC’s Region IV offices in Arlington, Texas on February 17, 2005. The NRC provided a phone bridge for this meeting and I called into it. The company reported that there had never been a procedural requirement to fill the piping sections with water, implying therefore was it was not improper then to revise a procedure in 1992 to drain water from the sections. The company further reported that the technical specifications issued by the NRC with the reactor operating licenses only required verifying that the piping on the discharge side of the pumps be filled with water but said nothing about the contents of the piping on the suction side (perhaps implying that this silence permitted piping sections to be filled with air, helium, jawbreakers, cement, or anything they desired.)
The owner also described full-scale testing using transparent plexiglass piping to show what was happening inside that it had performed as part of that it called the most expensive engineering analysis in the plant’s history. The company even showed a video from this testing (although the video was a wee bit hard to see via the phone bridge). When the owner completed its presentation, an NRC senior manager (whom I believe was Bruce Mallett, then Regional Administrator of NRC Region IV) remarked that the video and testing only convinced him that the pumps in the scale model would not cavitate; they told him little about performance in the real plant.
The NRC Puts Palo Verde in Its Place
That statement pretty much telegraphed the NRCs final answer on the matter. On April 8, 2005, the NRC issued a yellow finding, the second most serious in the agency’s four color-coded classifications, for operating the three reactors with safety system piping sections emptied of water and a $50,000 fine for the inadequate safety evaluation for the 1992 procedure change that had workers drain water from the piping after testing.
The company paid a far larger price. The NRC’s special inspection team investigation into this event and an NRC augmented inspection team investigation into all three reactors tripping on June 14, 2004, focused more NRC attention to the plant. More and more NRC inspectors identified more and more safety problems. In little time, Palo Verde went from all three reactors solidly in Column 1 of the Action Matrix within the NRC’s Reactor Oversight Process to Units 1 and 3 being in Column 3 and Unit 3 being in Column 4—the lowest safety performance rating in the country. It took over four years for the safety shortcomings to be remedied and all three reactors returned to Column 1. The cost of “volunteering” for more NRC scrutiny cost considerably more than the $50,000 fine.
Fig. 2 (Source: Union of Concerned Scientists)
The NRC Goes Big
NRC inspectors discovered a safety problem at Waterford. That discovery revealed a similar problem at Palo Verde. NRC inspectors determined the problem at Palo Verde to reflect systemic problems. The NRC’s responses remedied the specific problem at Waterford and the wider problems at Palo Verde.
But the NRC did not stop after these worthy regulatory achievements. They went big. Packaging the Palo Verde problem with other recent miscues, the NRC issued Bulletin 2008-01, “Managing Gas Accumulation in Emergency Core Cooling, Decay Heat Removal, and Containment Spray Systems,” to the owners of all U.S. operating reactors. It required owners to takes steps to ensure that safety systems at their plants did not have and were not likely to develop safety system impairments like that found at Palo Verde.
UCS Perspective
From the discovery at Waterford to the issuance of Bulletin 2008-01, the NRC exhibited just right regulation.
NRC inspectors found that workers knew about air collecting in piping but had not properly analyzed it. The ensuing analysis concluded that the air would not have prevented fulfilment of the necessary safety function. Despite that conclusion, the NRC issued a Green finding because public health was being protected more by luck than skill until the degraded condition was properly evaluated.
Whereas air was unintentionally collecting in piping at Waterford, workers followed procedures to drain water from safety system piping at Palo Verde and didn’t respond to the problem in a timely and effective manner. The NRC swung a bigger regulatory hammer.
The NRC then sought to avoid the problem across the U.S. fleet by issuing Bulletin 2008-01.
Some might contend that these events really reflect under-regulation by the NRC. After all, the air accumulation problem was first identified at Waterford in 1999 and not challenged by the NRC until 2004. The procedure was revised in 1999 to drain water from pipes at Palo Verde, but the NRC didn’t realize it until 2004. The Waterford and Palo Verde discoveries in 2004 joined by similar discoveries before then and afterwards didn’t prompt the NRC to cast a wider safety net until 2008. How can just right regulation entail such lengthy periods between creation of safety problems and their resolutions?
Blame the game and not its players. The NRC does not have the resources to inspect every corrective action report or review every procedure revision. Instead, the NRC audits samples. There’s no evidence that NRC inspectors looked at records at Waterford and Palo Verde prior to 2004 but missed seeing the problems or that NRC inspectors should have looked at these records but failed to do so.
As for the “delay” in getting Bulletin 2008-01 out, consider the adverse implications of a prompter response. Had the NRC issued the bulletin the day after the discovery at Waterford, owners would have been directed to look at the potential for air unintentionally collecting in piping. Since workers were intentionally draining water from piping at Palo Verde per an approved (albeit flawed) procedure, they would not have detected and corrected unintentional accumulation. By cultivating a number of similar events, the NRC required owners evaluate and manage a broader suite of potential problems—well worth the wait.
UCS’s Role of Regulation in Nuclear Plant Safety series of blog posts is intended to help readers understand when regulation played too little a role, too much of an undue role, and just the right role in nuclear plant safety.