The Nuclear Regulatory Commission (NRC) sent a special inspection team to the River Bend nuclear plant in St. Francisville, Louisiana on March 30, 2015, after a periodic test of the plant’s response to a simulated accident condition resulted in total loss of air conditioning for vital rooms in the control building. Equipment operating in these rooms could increase their ambient temperature to the point where electrical components failed or malfunctioned. Recurring problems with power supply breakers and control system circuit breakers caused the ventilation system’s failure. The NRC’s regulations permit safety components to be intentionally disabled for maintenance and testing while the reactor is operating, but only when properly justified by risk assessments. The NRC identified inadequate risk assessments when ventilations system components were disabled for testing warranted a White finding, the second most serious severity level in its Green, White, Yellow, and Red process. The NRC additionally identified three Green findings for ineffective corrective actions for the recurring breaker problems.
How the Event Unfolded
The control building chilled water system for the boiling water reactor at the River Bend Station near St. Francisville, Louisiana provides air conditioning for the control room, standby switchgear room, and equipment room within the control building. The safety function of the system is to maintain the ambient temperatures in these rooms below 104°F during both normal operation and accident conditions. Electrical equipment in these rooms—relays, switches, computers, etc.—may fail or malfunction if the ambient temperature rises above the design limit.
The control building chilled water system consists of two virtually identical divisions of components. The simplified drawing (Fig. 1) shows the major components in Division I. Division II is essentially identical, other than having the B and D components instead of the A and C ones.
Fig. 1 (Source: UCS)
Only one division is needed for the safety function to be met; a second division is provided for increased reliability. In fact, only half of a single division (i.e., one chiller and one chilled water pump) is needed to provide the necessary cooling for the rooms.
Each chiller consists of a condenser, compresser, and evaporator to cool the system’s water. The chilled water flows through the cooling coils of air handling units for the control room, standby switchgear room, and equipment room. The air handling units have fans that blow room air across the cooling coils. The water cools the room air. In turn, the room air heats the water. The chilled water pumps send the warmed water back to the chillers in this closed-loop cooling system.
Chiller A had been removed from service on August 11, 2014, for maintenance. A human performance problem (nukespeak for a worker’s mistake) damaged the chiller. It remained out of service until being repaired and returned to service in mid-2015.
On March 9, 2015, workers conducted a periodic test of the plant’s response to a simulated loss of coolant accident. The reactor was shut down at the time for refueling—a good time to conduct maintenance, inspection, and testing activities such as this test.
When the workers tested Division I, control building chiller A was already out of service because it was broken. Chiller C initially responded as designed. It was disconnected from its electrical power source. The automatic disconnection allows the emergency diesel generators to startup and supply power to the key electrical circuits, with safety equipment being reconnected to the power source in a staggered sequence to avoid overburdening the emergency diesel generators. (Large motors require a lot of electrical current to start, but their power needs drop once they are up and running. The staggered sequence of reconnecting electrical components allows the emergency diesel generators to handle the initially high current needs.)
But Chiller C did not restart as expected during the test. With both chillers in Division I now not working properly, the workers properly turned to a procedure written specifically for this situation: Abnormal Operating Procedure AOP-0060, “Loss of Control Building Ventilation.” Per this response procedure, the workers tried to start a Division II chiller. But neither Chiller B nor Chiller D could be started.
AOP-0060 also contained directions for steps to be taken when both divisions of the control building hilled water system were unavailable. The workers completed these steps (e.g., opening doors to the room and setting up portable fans) within 30 minutes. The temperature inside the control room increased from 65°F to 91.4°F in that half hour.
In the meantime, other workers were troubleshooting the problems preventing Chillers B, C, and D from being started. A couple of hours later, they determined that the electrical breaker for the power supply to Chiller C was getting a signal to close when the chiller’s start button was depressed, but the breaker was not closing. About 40 minutes later, workers depressed a reset pushbutton near the chiller and successfully restarted Chiller C in Division I. Workers later determined that its electrical breaker would not close due to a misaligned part. This was the first breaker failure due to this cause experienced at River Bend. To lessen the chances that other breakers fail for this reason, worker trimmed one-eighth of an inch off the part to make it less vulnerable to misalignment problems.
Workers determined that Chillers B and D in Division II could not be started because Air Handling Unit B for the control room would not start. An electrical interlock prevented the chillers from starting unless the air handling units were already running. Further troubleshooting determined that a circuit breaker in the control system for the air handling unit had failed.
The circuit breaker was a Masterpact breaker manufactured by Nuclear Logistics Incorporated. Workers replaced all of the original General Electric AKR circuit breakers between 2007 and 2009 with Masterpact breakers. The NRC’s special inspection team reported that the Masterpact breakers are “vulnerable to an intermittent failure mechanism under certain scenarios.”
The control logic for this type of circuit breaker sends closed breakers a continuous “close” command. When a breaker receives a signal to open, it briefly experiences simultaneous “open” and “close” commands. The circuit breaker has an “anti-pump” feature that prevents the breaker from rapidly flip-flopping due to the concurrent “open” and “close” signals. The “anti-pump” feature moves the close lever out of the way so the “open” signal dominates and the breaker opens.
In 2012, workers at the Hope Creek nuclear plant in New Jersey reported a binding problem with the “anti-pump” feature in Masterpact circuit breakers. They reported that minor vibrations would cause the close level to get stuck in position, preventing the “anti-pump” feature from moving the lever and allowing the breaker to open. Since the replacements, at least nine and perhaps as many as fifteen Masterpact breaker failures due to binding have occurred at River Bend. The uncertainty in the number of binding failures results from six failures due to unknown causes—perhaps due to binding or perhaps due to some other cause.
Nuclear Logistics Incorporated informed River Bend about the circuit breaker design vulnerability on December 9, 2014, and workers prepared the paperwork needed to investigate and resolve the problem. But they scheduled that task following restart from the refueling outage in spring 2015. Thus, the vulnerable circuit breakers remained unfixed when the test was conducted on March 9.
After the near miss, workers at River Bend modified the circuit breakers to eliminate the binding problem. It’s not clear from the report by the NRC’s special inspection team if they nailed that barn door closed or merely pushed it shut.
NRC Sanctions
The NRC’s special inspection team identified one preliminary White and three Green Findings:
- Preliminary White: Workers were not properly evaluating the risk associated with taking the control building chilled water system’s components out of service. The NRC’s Maintenance Rule allows workers to deliberately disable safety components while the reactor continues operating as long as the risk is evaluated and the components are restored to service in a timely manner. The NRC’s special inspection team identified times in the past when the risks of having control building chilled water system components out of service were higher than determined by plant workers.
- Green: Despite at least six Masterpact circuit breaker failures and perhaps as many as fifteen failures since 2007, workers had not initiated corrective actions per the NRC’s quality assurance regulations to resolve this safety problem in a timely and effective manner.
- Green: Workers used non-conservative data, contrary to station procedures, to conclude that the recurring Masterpact circuit breaker failures did not raise reliability or operability concerns. This violated another part of the NRC’s quality assurance regulations.
- Green: Despite a string of problems with Magne Blast electrical breakers, workers failed to identify the degrading trend and initiate timely corrective actions, violating another part of the NRC’s quality assurance regulations.
UCS Perspective
The biggest finding by the NRC’s special inspection team (i.e., the improper risk evaluations for planned maintenance while the reactor operated) was a very nice catch. The problem as detailed in the NRC’s report is black and white. But it took considerable homework by the special inspection team for this picture to emerge so clearly.
This near miss is yet another example of safety problems being introduced by recent modifications intended to manage the risk from aging components. As the bathtub curve shows, aging components face an increasingly greater chance of failure as they enter the wear-out region. But replacing aging components with brand new ones involves the risk of failure during the break-in region.
Workers replaced all the original GE AKR electrical breakers between 2007 and 2009 with Nuclear Logistics Incorporated Masterpact breakers. The GE AKR breakers certainly had at least their share of problems during the break-in phase, as evidenced by many warnings issued by the NRC to plant owners about them as well as reports of problems submitted by plants owners to the NRC. Installing replacement breakers made by a different vendor does not mean the break-in problems will be automatically revisited, but it opens the door for that potential. And some break-in problems entered via that door at River Bend.
The NRC’s special inspection team delivered the proper message with its sanctions in this case. Online maintenance and testing can be done when certain precautions are taken. By not taking the proper precautions, testing at River Bend sometimes involved greater risk than appropriate. And as the replacement components began failing, the adverse trend should have prompted the root cause to be identified and resolved in a timely and effective manner.
While these problems did not cause this near miss to have serious consequences, that outcome was more luck than it should have been. If the plant owner responds appropriately to the NRC’s message, the programmatic fixes will reduce the reliance on luck for nuclear safety at River Bend.