The Nebraska Public Power District’s Cooper Nuclear Station about 23 miles south of Nebraska City has one boiling water reactor that began operating in the mid-1970s to add about 800 megawatts of electricity to the power grid. Workers shut down the reactor on September 24, 2016, to enter a scheduled refueling outage. That process eventually led to NRC special inspections.
Following the outage, workers reconnected the plant to the electrical grid on November 8, 2016, to begin its 30th operating cycle. During the outage, workers closed two valves that are normally open when while the reactor operates. Later during the outage, workers were directed to re-open the valves and they completed paperwork indicating the valves had been opened. But a quarterly check on February 5, 2017, revealed that both of the valves remained closed. The closed valves impaired a key safety system for 89 days until the mis-positioned valves were discovered and opened. The NRC dispatched a special inspection team to the site on March 1, 2017, to look into the causes and consequences of the improperly closed valves.
The Event
Workers shut down the reactor on September 24, 2016. The drywell head and reactor vessel head were removed to allow access to the fuel in the reactor core. By September 28, the water level had been increased to more than 21 feet above the flange where the reactor vessel head is bolted to the lower portion of the vessel. Flooding this volume—called the reactor cavity or refueling well—permits spent fuel bundles to be removed while still underwater, protecting workers from the radiation.
With the reactor shut down and so much water inventory available, the full array of emergency core cooling systems required when the reactor operates was reduced to a minimal amount. The reduction of systems required to remain in service facilitates maintenance and testing of out-of-service components.
In the late afternoon of September 29, workers removed Loop A of the Residual Heat Removal (RHR) system from service for maintenance. The RHR system is like a nuclear Swiss Army knife—it can supply cooling water for the reactor core, containment building, and suppression pool and it can provide makeup water to the reactor vessel and suppression pool. Cross-connections enable the RHR system to perform so many diverse functions. Workers open and close valves to transition from one RHR mode of operation to another.
As indicated in Figure 1, the RHR system at Cooper consisted of two subsystems called Loop A and Loop B. The two subsystems provide redundancy—only one loop need function for the necessary cooling or makeup job to be accomplished successfully.
Fig. 1 (Source: Nebraska Public Power District, Individual Plant Examination (1993))
RHR Loop A features two motor-driven pumps (labeled P-A and P-C in the figure) that can draw water from the Condensate Storage Tank (CST), suppression chamber, or reactor vessel. The pump(s) send the water through, or around, a heat exchanger (labeled HX-A). When passing through the heat exchanger, heat is conducted through the metal tube walls to be carried away by the Service Water (SW) system. The water can be sent to the reactor vessel, sprayed inside the containment building, or sent to the suppression chamber. RHR Loop B is essentially identical.
Work packages for maintenance activities include steps when applicable to open electrical breakers to de-energize components and protect workers from electrical shocks and close valves to allow isolated sections of piping to be drained of water so valves or pumps can be removed or replaced. The instructions for the RHR Loop A maintenance begun on September 29 included closing valves V-58 and V-60. These are valves that can only be opened and closed manually using handwheels. Valve V-58 is in the minimum flow line for RHR Pump A while V-60 is in the minimum flow line for RHR Pump C. These two minimum flow lines connect downstream of these manual valves and then this common line connects to a larger pipe going to the suppression chamber.
Motor-operated valve MOV-M016A in the common line automatically opens when either RHR Pump A or C is running and the pump’s flow rate is less than 2,731 gallons per minute. The large RHR pumps generate considerable heat when they are running. The minimum flow line arrangement ensures that there’s sufficient water flow through the pumps to prevent them from being damaged by overheating. MOV-M016A automatically closes when pump flow rises above 2,731 gallons per minute to prevent cooling flow or makeup flow from being diverted.
The maintenance on RHR Loop A was completed by October 7. The work instructions directed operators to reopen valves V-58 and V-60 and then seal the valves in the opened position. For these valves, sealing involved installing a chain and padlock around the handwheel so the valve could not be repositioned. The valves were sealed, but mistakenly in the closed rather than opened position. Another operator independently verified that this step in the work instruction had been completed, but failed to notice that the valves were sealed in the wrong position.
At that time during the refueling outage, RHR Loop A was not required to be operable. All of the fuel had been offloaded from the reactor core into the spent fuel pool. On October 19, workers began transferring fuel bundles back into the reactor core.
On October 20, operators declared RHR Loop A operable. Due to the closed valves in the minimum flow lines, RHR Loop A was actually inoperable, but that misalignment was not known at the time.
The plant was connected to the electrical grid on November 8 to end the refueling outage and begin the next operating cycle.
Between November 23 and 29, workers audited all sealed valves in the plant per a procedure required to be performed every quarter. Workers confirmed that valves V-58 and V-60 were sealed, but failed to notice that the valves were sealed closed instead of opened.
On February 5, 2017, workers were once again performing the quarterly audit of all sealed valves. This time, they noticed that valves V-58 and V-60 were not opened as required. They corrected the error and notified the NRC about its discovery.
The Consequences
Valves V-58 and V-60 had been improperly closed for 89 days, 12 hours, and 49 minutes. During that period, the pumps in RHR Loop A had been operated 15 times for various tests. The longest time that any pump was operated without its minimum flow line available was determined to be 2 minutes and 18 seconds. Collectively, the pumps in RHR Loop A operated for a total of 21 minutes and 28 seconds with flow less than 2,731 gallons per minute.
Running the pumps at less than “minimum” flow introduced the potential for their having been damaged by overheating. Workers undertook several steps to determine whether damage had occurred. Considerable data is collected during periodic testing of the RHR pumps (as suggested by the fact it was known that the longest a pump ran without its minimum flow line was 2 minutes and 18 seconds). Workers reviewed data such as differential pressures and vibration levels from tests over the prior two years and found that current pump performance was unchanged from performance prior to the fall 2016 refueling outage.
Workers also calculated how long it would take a RHR pump to operate before becoming damaged. They estimated that time to be 32 minutes. To double-check their work, a consulting firm was hired to independently answer the same question. The consultant concluded that it would take an hour for an RHR pump to become damaged. (The 28 minute difference between the two calculations was likely due to the workers onsite making conservative assumptions that the more detailed analysis was able to reduce. But it’s a difference without distinction—both calculations yield ample margin to the total time the RHR pumps ran.)
The testing and analysis clearly indicate that the RHR pumps were not damaged by their operating during the 89-plus days their minimum flow lines were unavailable.
The Potential Consequences
The RHR system can perform a variety of safety functions. If the largest pipe connected to the reactor vessel were two rupture, the two pumps in either RHR Loop are designed to provide more than sufficient makeup flow to refill the reactor vessel before the reactor core overheats.
The RHR system has high capacity, low head pumps. This means the pumps supply a lot of water (many thousands of gallons each minute) but at a low pressure. The RHR pumps deliver water at roughly one-third of the normal operating pressure inside the reactor vessel. When small or medium-sized pipes ruptured, cooling water drains out but the reactor vessel pressure takes longer to drop below the point where the RHR pumps can supply makeup flow. During such an accident, the RHR pumps will automatically start but will send water through the minimum flow lines until reactor vessel pressures drops low enough. The closure of valves V-58 and V-60 could have resulted in RHR Pumps A and C being disabled by overheating about an hour into an accident.
Had RHR Pumps B and D remained available, their loss would have been inconsequential. Had RHR Pumps B and D been unavailable (such as due to failure of the emergency diesel generator that supplies them electricity), the headline could have been far worse.
NRC Sanctions
The NRC’s special inspection team identified the following two apparent violations of regulatory requirements, both classified as Green in the agency’s Green, White, Yellow and Red classification system:
- Exceeding the allowed outage time in the operating license for RHR Loop A being inoperable. The operating license permitted Cooper to run for up to 7 days with one RHR loop unavailable, but the reactor operated far longer than that period with the mis-positioned valves.
- Failure to implement an adequate procedure to control equipment. Workers used a procedure every quarter to check sealed valves. But the guidance in that procedure was not clear enough to ensure workers verified both that a valve was sealed and that it was in the correct position.
UCS Perspective
This near-miss illustrates the virtues, and limitations, of the defense-in-depth approach to nuclear safety.
The maintenance procedure directed operators to re-open valves V-58 and V-60 when the work on RHR Loop A was completed.
While quite explicit, that procedure step alone was not deemed reliable enough. So, the maintenance procedure required a second operator to independently verify that the valves had been re-opened.
While the backup measure was also explicit, it was not considered an absolute check. So, another procedure required each sealed valves to be verified every quarter.
It would have been good had the first quarterly check identified the mis-positioned valves.
It would have been better had the independent verifier found the mis-positioned valves.
It would have best had the operator re-opened the valves as instructed.
But because no single barrier is 100% reliable, multiple barriers are employed. In this case, the third barrier detected and corrected a problem before it could be contribute to a really bad day at the nuclear plant.
Defense-in-depth also accounts for the NRC’s levying two Green findings instead of imposing harsher sanctions. The RHR system performs many safety roles in mitigating accidents. The mis-positioned valves impaired, but did not incapacitate, one of two RHR loops. That impairment could have prevented one RHR loop from successfully performing its necessary safety function during some, but not all, credible accident scenarios. Even had the impairment taken RHR Loop A out of the game, other players on the Emergency Core Cooling System team at Cooper could have stepped in.
Had the mis-positioned valves left Cooper with a shorter list of “what ifs” that needed to line up to cause disaster or with significantly fewer options available to mitigate an accident, the NRC’s sanctions would have been more severe. The Green findings are sufficient in this case to remind Cooper’s owner, and other nuclear plant owners, of the importance of complying with safety regulations.
Accidents certainly reveal lessons that can be learned to lessen the chances of another accident. Near-misses like this one also reveal lessons of equal value, but at a cheaper price.