Disaster by Design/ Safety by Intent #35
Disaster by Design
Fission Stories #58 described how control room operators prepared for a test to be conducted on the Unit 2 reactor at the Millstone nuclear plant in Connecticut. Each operator who would touch control switches during the test was assigned a peer checker who would have a copy of the test procedure in hand to verify that the operator conducted every step as specified. The entire group of operators and peer checkers went into the simulator—a full-scale, computer-controlled mockup of the control room—two days before the test to rehearse it a few times. What could go wrong?
Despite all these preparations to get the test right, the actual test went wrong. An operator turned a switch the wrong way. The peer checker did not notice that the right switch had been turned the wrong way. A control room supervisor also observed the operator turn the switch, but mistakenly thought it had been turned in the right direction.
The reactor responded to the mistaken control switch movement like it was supposed to, but not in the way that the operators expected and the test procedure specified. So, the operator turned the right switch three more times in the wrong direction. If two wrongs don’t make a right, four wrongs don’t either. The additional mistakes made things worse. By the time things stabilized, the reactor power level had increased by 8% during a test in which the power level was not supposed to change at all.
How had all the preparations failed to result in the test being conducted successfully? Some of the operators who participated in the dress rehearsals in the simulator did not participate in the actual test two days later. Their stand-ins performed the test “cold” without benefit of the dry runs. And the dry runs on the simulator did not truly test the role of the peer checkers and supervisors. When the operators performed steps properly, the ability of the peer checkers and supervisors to detect a wrong step was never demonstrated.
Emergency exercises
This example of well-intended preparations failing to achieve the desired performance may be replicated with the emergency exercises conducted every two years for every U.S. nuclear plant. The NRC evaluates the response of plant workers to the simulated accident, including communications with local, state, and federal entities and documents its findings in reports (see the report for the final exercise conducted at Vermont Yankee for an example.) The Federal Emergency Management Agency (FEMA) evaluates how well the local and state organizations respond (see FEMA’s report on a recent exercise at the Shearon Harris nuclear plant in North Carolina for the scope of their evaluation.)
The Incident Response Center during the Southern Exposure exercise conducted for the HB Robinson nuclear plant (SC) in October 2015. The NRC’s Executive Team is in the foreground with senior members of the agency’s reactor, engineering, and security offices. NRC staffers in the background monitor conditions at the plant. (Source: NRC Flickr Gallery)
The emergency exercises are scheduled many months in advance, ensuring all the primary responders are available rather than away on vacation or attending meetings.
The emergency exercises are conducted on weekdays during normal working hours, ensuring that all primary responders are readily and easily notified of the pretend emergency.
The emergency exercises almost always end within a handful of hours, ensuring that all primary responders can head home at the normal quitting times. The exercises’ short durations never evaluate how effectively the response can be sustained over the longer duration of real accidents.
But real nuclear plant accidents (as Three Mile Island, Chernobyl, and Fukushima demonstrate) do not always happen during normal working hours and never with months of advance warning. The biennial emergency exercises could be made more realistic if the NRC and FEMA employed simple, basic tactics such as:
- Periodically taking one or two of the primary responders out of the lineup to see how well the second team players can step in and perform the necessary duties.
- Simulating an event occurring in the middle of the night by holding up entry of some primary responders to the emergency response facilities, simulating the staggered arrivals of these individuals following notifications to report.
- Simulating a shift change three or four hours into a simulated accident to evaluate how effectively the second team can take over from the primary responders and sustain the effort. These multiple shifts would have the collateral benefit of enabling more individuals to role play their assigned response functions, expanding the training value of the biennial exercises.
Safety by Intent
The biennial emergency exercises are to accident response as force-on-force tests are to security readiness. The exercises and tests show how well a large number of measures work together to achieve the intended outcomes. Insights can be gained by inspecting the procedures governing each specific task and interviewing the individuals assigned with performed the tasks. But the exercises and tests are the surest way of determining whether there are seams or disconnects between the numerous tasks.
The biennial exercises have considerable value, both in ensuring the fidelity of many response measures and in further familiarizing individuals who conduct the responses. The biennial exercises could, and should, be made more valuable by simple tactics that have them better emulate real accident situations.
—–
UCS’s Disaster by Design/ Safety by Intent series of blog posts is intended to help readers understand how a seemingly unrelated assortment of minor problems can coalesce to cause disaster and how effective defense-in-depth can lessen both the number of pre-existing problems and the chances they team up.