Energy Northwest’s Columbia Generating Station near Richland, Washington has one General Electric boiling water reactor (BWR/5) with a Mark II containment design that began operating in 1984. In the late morning hours of Sunday, December 18, 2016, the station stopped generating electricity and began generating problems.
The Nuclear Regulatory Commission (NRC) dispatched a special inspection team to investigate the event after determining it could have increased the risk of reactor core damage by a factor of ten. The NRC team sought to understand the problems occurring during this near-miss as well as assess the breadth and effectiveness of the solutions proposed by the company for them.
Trouble Begins Offsite
The plant was operating at full power when the main generator output breakers opened at 11:24 am due to an electrical transient within the Ashe substation. The Ashe substation is owned and maintained by the Bonneville Power Authority and serves as the connection between electricity produced at the plant and the offsite power grid. At least three electrical breakers at the Ashe substation were supposed to have opened to de-energize the faulted transmission line(s). Had they done so, the loss of the transmission lines could have triggered protective devices at the Columbia Generating Station to automatically trip the main generator. But cold weather kept the breakers from functioning properly. Instead of the protective systems at the Columbia Generating Station responding on a system level (i.e., the de-energized transmission line(s) triggering a main generator trip), they responded at the component level (i.e., the main generator output breaker sensed the electrical transient and opened).
The turbine control valves automatically closed because the main generator was no longer fully loaded with its output breakers opened. The closure of the turbine control valves automatically tripped the reactor. The control rods fully inserted within seconds to stop the nuclear chain reaction. The output breakers, turbine control valves, and control rods all functioned per the plant’s design (see Figure 1).
Fig. 1 (Source: Nuclear Regulatory Commission annotated by UCS)
Before the trip, the main generator was producing electricity at 25,000 volts. The main transformer increased the voltage up to 500,000 volts for transmission out to the offsite power grid. The auxiliary transformers reduced the voltage to 4,160 volts and 6,900 volts for supply to equipment in the plant. The output breakers that opened to start this event are represented by the square box in the upper left corner of Figure 2.
Fig. 2 (Source: Nuclear Regulatory Commission annotated by UCS)
Trouble Begins Onsite – Loss of Heat Sink and Normal Makeup
The main generator was disconnected from the offsite power grid but continued to supply electricity through the auxiliary transformers to plant equipment. Because steam was no longer flowing to the turbine, the voltage and frequency of the electricity dropped. The voltages flowing to in-plant equipment dropped low enough to cause electrical breakers to automatically open at 11:25 am to protect motors and other electrical equipment from damage caused by under-voltage. For example, an electric motor requires an electrical current of a certain voltage in order to operate. Electrical current of lower voltage may not be enough to enable the motor to run, but that current flowing through the motor may be enough to heat it up and damage it. One of the de-energized loads caused the Main Steam Isolation Valves (MSIVs) to close. Their closure meant that steam produced by the reactor’s decay heat no longer flowed to the condenser where it got cooled by water from the plant’s cooling towers. Instead, the steam bottled up in the reactor vessel and piping until it increased the pressure to the point where the safety/relief valves opened to discharge steam to the suppression pool (see Figure 3).
The closure of the MSIVs also stopped the normal flow of makeup cooling water to the reactor vessel. The feedwater system uses steam-driven turbines connected to pumps to supply makeup cooling water to the reactor vessel. But the steam supply for the feedwater pumps is downstream of the now-closed MSIVs. The condensate and condensate booster pumps upstream of the feedwater pumps have electric motors and continued to be available. But collectively they only pump water at about two-thirds of the pressure inside the reactor vessel, meaning they could not supply makeup water unless the pressure inside the reactor vessel decreased by nearly one-third its normal pressure.
Fig. 3 (Source: Nuclear Regulatory Commission annotated by UCS)
Troubles Onsite Grow – Loss of Normal Power for Safety Buses
At 11:28 am, the safety buses SM7 and SM8 tripped on low voltage, causing their respective emergency diesel generators to start and provide power to these vital buses. This was not supposed to happen during this event. By procedure, the operators were directed to manually trip the turbine and generator following the automatic trip of the reactor. They tripped the turbine at 11:27 am, but never tripped the main generator. Tripping the main generator as specified in the procedures would have immediately caused electrical breakers to close and other electrical breakers to open to swap the supply of electricity to plant equipment from the auxiliary transformers to the startup transformers as shown in Figure 4. The startup transformers reduce 230,000 volt electricity from the offsite power grid to 4,160 volts and 6,900 volts for use by plant equipment when the main generator is unavailable. With electricity to plant equipment from the startup transformers, the MSIVs would have remained open and makeup cooling water supplied by the feedwater pumps as normally provided.
Fig. 4 (Source: Nuclear Regulatory Commission annotated by UCS)
Even More Trouble Onsite – Loss of Backup Makeup
The operators manually started the Reactor Core Isolation Cooling (RCIC) system (not shown on the Figure 3, but a smaller version of the High Pressure Coolant System) at 11:32 am to provide makeup cooling water because the feedwater system was unavailable. The RCIC systems’ primary function is to supply makeup cooling water when the feedwater system cannot do so. Like the feedwater pumps, the RCIC pump is connected to a steam-driven turbine. Unlike the feedwater pumps, the RCIC pump’s turbine is supplied with steam from the reactor vessel through a connection upstream of the closed MSIVs. The RCIC pump transfers water from a large storage tank to the reactor vessel.
The operators failed to follow the procedure when starting the RCIC system. The procedure called for them to close the steam admission valve (V-45) and then open the trip valve (V-1) as soon as V-45 was fully closed (see Figure 5). But they did not open V-1. The failure to open V-1 disabled the control system designed to bring the RCIC turbine up to desired speed in 12 seconds. Instead, the RCIC turbine tried to obtain the desired speed instantly. Too much steam too soon caused the RCIC turbine to automatically trip on high speed. This trip guards against the spinning turbine blades coming apart due to excessive forces.
It took about 13 minutes for workers to go down into the RCIC room in the reactor building’s basement and reset the mis-positioned valves to allow the system to be properly started. In that time, the water level inside the reactor vessel dropped about a foot as it boiled away. That still left 162 inches (13.5 feet) of water above the top of fuel in the reactor core. The operators had several hours to restore makeup cooling water flow before the reactor core started uncovering and overheating.
Fig. 5 (Source: Nuclear Regulatory Commission annotated by UCS)
The operators manually started the High Pressure Core Spray (HPCS) system at 12:09 pm to provide makeup cooling water with the feedwater and RCIC systems both unavailable. The main HPCS pump (HPCS-P-1) has an electric motor. The pump transfer water from the large storage tank to the reactor vessel. While RCIC is designed to supply makeup water to compensate for inventory boiled off after the reactor shuts down, the HPCS system is designed to also compensate for water being lost through a small-diameter (about 2 inches) pipe that drains cooling water from the reactor vessel. Consequently, the HPCS system flow rate is about ten times greater than the RCIC system flow rate. And whereas the RCIC system flow rate can be throttled to match the makeup need, the HPCS system makeup flow is either full or zero.
The HPCS system refilled the reactor vessel soon after it was started. The operators closed the HPCS system injection valve (V-4) after about a minute. The minimum flow valve (V-12) automatically opened to direct the pump flow to the suppression pool instead of to the reactor vessel (see Figure 6). The HCPS system ran in “idle” mode for the next 3 hours and 42 minutes.
Fig. 6 (Source: Nuclear Regulatory Commission annotated by UCS)
Yet More Trouble Onsite – Water Leaking into Reactor Building
On December 18, workers discovered that the restricting orifice (RO) downstream of V-12 had leaked an estimated 4.7 gallons per minute into the reactor building while the HPCS system had operated. The NRC team learned that the gasket material used in this restricting orifice had been the subject of an industry operating experience report in 2007. A condition report was written at Columbia Generating Station in 2008 to have engineering assess the operating experience report and gasket materials used at the plant. In early 2010, the condition report was closed out based on engineering’s evaluation to use the gasket material recommended in the industry report. But the “bad” gaskets were not replaced.
Operating experience cited in the 2007 industry report revealed that the original gasket material was vulnerable to erosion. The report described two adverse consequences from the material’s erosion. First, pieces of the gasket could be carried by the water into the reactor vessel where the material impacting the fuel rods could damage their cladding. Second, gasket erosion could allow leakage. The 2007 industry report thus forecast the problem experienced at Columbia Generating Station in December 2016. The solution recommended by the 2007 report was not implemented until after the forecast problem has occurred.
NRC Sanctions
The NRC’s special inspection team identified three safety violations at the Columbia Generating Station. Two violations involved the operators failing to follow written procedures: (1) the failure to trip the main generator which resulted in the unnecessary closure of the MSIVs, and (2) the failure to properly start the RCIC system which resulted in the unnecessary trip of its turbine. The third violation was associated with the continued use of gasket material determined nearly a decade earlier to be improper for this application.
UCS Perspective
Self-inflicted problems turned a fairly routine incident into a near-miss. Luck stopped it from progressing further.
The problem started offsite due to causes outside the control of the plant’s owner. Those uncontrollable causes resulted in the main generator output breakers opening as designed.
By procedure, the operators were supposed to trip the main generator. Failing to do so resulted in the unnecessary closure of the MSIVs and the loss of the normal makeup cooling flow to the reactor vessel.
By procedure, the operators were supposed to manually start the RCIC system to provide backup cooling water flow to the reactor vessel. But they failed to properly start the system and it immediately tripped.
Procedures are like recipes—positive outcomes are achieved only when they are followed.
The operators resorted to using the HPCS system. It took about a minute for the HPCS system to recover the reactor vessel water level—the operators left it running in “idle” for the next three hours and 42 minutes during which time about 5 gallons per minute leaked into the reactor building. The leak was through eroded gasket material that had been identified as improper for this application nearly a decade earlier, but never replaced.
Defense-in-depth is a nuclear safety hallmark. That hallmark works best when operators don’t bypass barriers and when workers patch known holes in barriers. Luckily, other barriers remained effective to thwart this near-miss from becoming a disaster. But luck is a fickle factor that needs to be minimized whenever possible.