The Nuclear Regulatory Commission (NRC) reacted to a trio of miscues at the Grand Gulf nuclear plant in Mississippi by sending a special inspection team to investigate. While none of the events had adverse nuclear safety consequences, the NRC team identified significantly poor performance by the operators in all three. The recurring performance shortfalls instill little confidence that the operators would perform successfully in event of a design basis or beyond design basis accident.
The Events
Three events prompted the NRC to dispatch a special inspection team to Grand Gulf:
(1) failure to recognize that reactor power fluctuating up and down by more than 10% during troubleshooting of a control system malfunction in June 2016 exceeded a longstanding safety criterion calling for immediate shutdown,
(2) failure to recognize in September 2016 that the backup reactor cooling system relied upon when the primary cooling system broke was unable to function if needed, and
(3) failure to understand how a control system worked on September 27, 2016, resulting in the uncontrolled and undesired addition of nearly 24,000 gallons of water to the reactor vessel.
(1) June 2016 Reactor Power Oscillation Miscue
Figure 1 shows the main steam system for a typical boiling water reactor like Grand Gulf. The reactor vessel is not shown but is located off its left side. Heat produced by the reactor core boils water. Four pipes transport the steam from the reactor vessel to the turbine. The steam spins the turbine which is connected to a generator (off the right side of Figure 1) to make electricity.
Fig. 1 (Source: Nuclear Regulatory Commission)
Periodically, operators reduce the reactor power level to about 65% power and test the turbine stop valves (labeled SV in Figure 1). The stop valves are fully open when the turbine is in service, but are designed to rapidly close automatically if a turbine problem is detected. When the reactor is operating above about 30 percent power, closure of the stop valves triggers the automatic shutdown of the reactor. Below about 30 percent power, the main steam bypass valves (shown in the lower left of Figure 1) open to allow the steam flow to the main condenser should the stop valves close.
Downstream of the turbine stop valves are the turbine control valves (labeled CV in Figure 1.) The control valves are partially open when the turbine is in service. The control valves are automatically re-positioned by the electro-hydraulic control (labeled EHC) system as the operators increase or decrease the reactor power level. Additionally, the EHC system automatically opens the three control valves in the other steam pipes more fully when the stop valve in one steam pipe closes. The EHC system and the control valve response time is designed to minimize the pressure transient experienced in the reactor vessel when the steam flow pathways change.
The test involves the operators closing each stop valve to verify these safety features function properly. During testing on June 17, 2016, however, unexpected outcomes were encountered. The EHC system failed to properly reposition the control valves in the other lines when a stop valve was closed, and later when it was re-opened. The control system glitch caused the reactor power level to increase and decrease between 63% and 76%.
Water flowing through the core of a boiling water reactor is heated to the boiling point. By design, the formation of steam bubbles during boiling acts like a brake on the reactor’s power level. Atoms splitting within the reactor core release heat. The splitting atoms also release neutrons, subcomponents of the atoms. The neutrons can interact with other atoms to cause them to split in what is termed a nuclear chain reaction. The neutrons emitted by splitting atoms have high energy and high speed. The neutrons get slowed down by colliding with water molecules. While fast neutrons can cause atoms to split, slower neutrons perform this role significantly better.
The EHC system problems caused the turbine control valves to open wider and close more than was necessary to handle the steam flow. Turbine control valves opened wider than necessary lowered the pressure inside the reactor vessel, allowing more steam bubbles to form. With fewer water molecules around to slow down the fast neutrons, more neutrons went places other than interacting with atoms to cause more fissions. The reactor power level dropped as the neutron chain reaction rate slowed.
When turbine control valves closed more than necessary, the pressure inside the reactor vessel increased. The higher pressure collapsed steam bubbles and made it harder for new bubbles to form. With more water molecules around, more neutrons interacted with atoms to cause more fissions. The reactor power level increased as the neutron chain reaction rate quickened.
Workers performed troubleshooting of the EHC system problems for 40 minutes. The reactor power level fluctuated between 63% and 76% as the turbine control valves closed too much and then opened too much. Finally, a monitoring system detected the undesired power fluctuations and automatically tripped the reactor, causing all the control rods to rapidly insert into the reactor core and stop the nuclear chain reaction.
The NRC’s special inspection team reported that the control room operators failed to realize that the 10% power swings exceeded a safety criterion that called for the immediate shut down of the reactor. Following a reactor power level instability event at the LaSalle nuclear plant in Illinois in March 1988, Grand Gulf and other boiling water reactors revised operating procedures in response to an NRC mandate to require reactors to be promptly shut down when the reactor power level oscillated by 10% or more.
EHC system problems causing unwanted and uncontrolled turbine control valve movements had been experienced eight times in the prior three years. Operators wrote condition reports about the problems, but no steps had been taken to identify the cause and correct it.
Consequences
Due to the intervention by the system triggering the automatic reactor scram, this event did not result in fuel damage or release of radioactive materials exceeding normal, routine releases. But that outcome was achieved despite the operators’ efforts but because of them. The operators’ training and procedures should have caused them to manually shut down the reactor when its power level swung up and down by more than 10%. Fortunately, the plant’s protective features intervened to remedy their poor judgement.
(2) September 2016 Backup Reactor Cooling System Miscue
On September 4, 2016, the operators declared residual heat removal (RHR) pump A (circled in red in the lower middle portion of Figure 2) to be inoperable after it failed a periodic test. The pump was one of three RHR pumps that can provide makeup cooling water to the reactor vessel in case of an accident. RHR pumps A and B can also be used to cool the water within the reactor vessel during non-accident conditions. Grand Gulf’s operating license only permitted the unit to continue running for a handful of days with RHR pump A inoperable. So, the operators shut down the reactor on September 8 to repair the pump.
Fig. 2 (Source: Nuclear Regulatory Commission)
The operating license required two methods of cooling the water within the reactor vessel during shut down conditions. RHR pump B functioned as one of the methods. The operators took credit for the alternate decay heat removal (ADHR) system as the second method. The ADHR system is shown towards the upper right of Figure 2. It features two pumps that can take water from the reactor vessel, route it through heat exchangers, and return the cooled water to the reactor vessel. The ADHR system’s heat exchangers are supplied with cooling water from the plant service water (PSW) system. Warmed water from the reactor vessel flows through hundreds of metal tubes within the ADHR heat exchangers. Heat conducted through the tube walls gets carried away by the PSW system.
By September 22, workers had replaced RHR pump A and successfully tested the replacement. The following day, operators attempted to place the ADHR system in service prior to removing RHR pump B from service. They discovered that all the PSW valves (circle in red in the upper right portion of Figure 2) to the ADHR heat exchangers were closed. With these valves closed, the ADHR pumps would only take warm water from the reactor vessel, route it through the ADHR heat exchangers, and return the warm water back to the reactor vessel without being cooled.
The operating license required workers to check each day that both reactor water cooling systems were available during shut down. Each day between September 9 and 22, workers performed this check via a paperwork exercise. No one ever walked out into the plant to verify that the ADHR pumps were still there and that the PSW valves were still open.
The NRC team determined that workers closed the PSW valves to the ADHR heat exchangers on August 10 to perform maintenance on the ADHR system. The maintenance work was completed on August 15, but the valves were mistakenly not re-opened until September 23 after being belatedly discovered to be mis-positioned.
Consequences
Improperly relying on the ADHR system in this event had no adverse nuclear safety consequences. It was relied upon was a backup to the primary reactor cooling system which successfully performed that safety function. Had the primary system failed, the ADHR system would not have been able to take over that function as quickly as intended. Fortunately, the ADHR system’s vulnerability was not exploited.
(3) September 2016 Reactor Vessel Overfilling Miscue
On September 24, Grand Gulf was in what is called long cycle cleanup mode. Water within the condenser hotwell (upper right portion of Figure 3) was being sent by the condensate pumps through filter demineralizers and downstream feedwater heaters before recycling back to the condenser via the startup recirculation line. A closed valve prevented this water from flowing into the reactor vessel. Long cycle cleanup mode allows the filter demineralizers to remove particles and dissolved ions from the water. Water purity is important in boiling water reactors because any impurities tend to collect within the reactor vessel rather than being carried away with the steam leaving the vessel. The water in the condenser hotwell is the water used over and over again in boiling water reactors to make the steam that spins the turbine-generator.
Fig. 3 (Source: Nuclear Regulatory Commission)
Workers were restoring RHR pump B to its standby alignment following testing. The procedure they used directed them to open the closed feedwater valve. This valve was controlled by three pushbuttons in the control room: OPEN, CLOSE, and STOP. As soon as this valve began opening, water started flowing into the reactor vessel rather than being returned to the condenser.
The operator twice depressed the CLOSE pushbutton wanting very much for the valve to re-close. But this valve was designed to travel to the fully opened position after the OPEN pushbutton was depressed and travel to the fully closed position after the CLOSE pushbutton was depressed. By design, the valve would not change direction until after it had completed its full travel.
Unless the STOP pushbutton was depressed. The STOP pushbutton, as implied by its label, caused the valve’s movement to stop. Once stopped, depressing the CLOSE pushbutton would close the valve and depressing the OPEN pushbutton would open it.
According to the NRC’s special inspection team, “operations personnel did not understand the full function of the operating modes of [the] valve.” No operating procedure directed the operators to use the STOP button. Training in the control room simulator never covered the role of the STOP button because it was not mentioned in any operating procedures.
Not able to use the installed control system to its advantage, the operator waited until the valve traveled fully open before getting it to fully re-close. But the valve is among the largest and slowest valves in the plant—more like an elephant than a cheetah in its speed.
During the time the valve was open, an estimated 24,000 gallons of water overfilled the reactor vessel. As shown in Figure 4, the vessel’s normal level is about 33 inches above instrument zero, or about 201 inches above the top of the reactor core. The 24,000 gallons filled the reactor vessel to 151 inches above instrument zero.
Fig. 4 (Source: Nuclear Regulatory Commission)
Consequences
The overfilling event had no adverse nuclear safety consequences (unless revealing procedure inadequacies, insufficient training, and performance shortcomings count.)
NRC Sanctions
The NRC’s special inspection team identified three violations of regulatory requirements. One violation involved inadequate procedures for the condensate and feedwater systems that resulted in the reactor vessel overfilling event on September 24.
Another violation involved crediting the ADHR system for complying with an operating license requirement between September 9 and 22 despite its being unable to perform the necessary reactor water cooling role due to closed valves in the plant service water supply to the ADHR heat exchangers.
The third violation involved inadequate verification of the ADHR system availability between September 9 and 22. Workers failed to properly verify the system’s availability and had merely assumed it was a ready backup.
UCS Perspective
Th trilogy of miscues, goofs, and mistakes that prompted the NRC to dispatch a special inspection team have a common thread. Okay, two common threads since all three happened at Grand Gulf. All three miscues reflected very badly on the operations department.
During the June power fluctuations miscue, the operators should have manually scrammed the reactor, but failed to do so. In addition, operators had experienced turbine control system problems eight times in the prior three years and initiated reports intended to identify the causes of the problems and remedy them. The maintenance department could have, and should have, reacted to these reports earlier. But the operations department could have, and should have, insisted on the recurring problems getting fixed rather than meekly adding to the list of unresolved problem reports.
During the September backup cooling system miscue, many operators over nearly two weeks had many opportunities to notice that the ADHR system would not perform as needed due to mispositioned valves. The maintenance department could have, and should have, not set a trap for the operators by leaving the valves closed when maintenance work was completed. But the operators are the only workers at the plant licensed by the NRC to ensure regulatory requirements intended to protect the public are met. They failed that legal obligation again and again between September 9 and 22.
During the September reactor vessel overfilling event, the operators failed to recognize that opening the feedwater valve while in long cycle cleanup mode would send water into the reactor vessel. That’s a fundamental mistake that’s nearly impossible to justify. The operators then compounded that mistake by failing to properly use the installed control system to mitigate the event. They simply did not understand how the three pushbutton controls worked and thus were unable to use them properly.
The poor operator performance that is the common thread among the trio of problems examined by the NRC’s special inspection team inspire little to no confidence that their performance will be any better during a design basis or beyond design basis event.