We Surveyed Thousands of Federal Scientists. Here are Some Potential Reasons Why the Response Rate Was Lower than Usual

August 14, 2018 | 2:49 pm
Jacob Carter
Research Director

In February and March of this year, the Union of Concerned Scientists, in partnership with Iowa State University’s Center for Survey Statistics and Methodology, sent a survey to over 63,000 federal career staff across 16 federal agencies, offices, and bureaus. Our goal was to give scientists a voice on the state of science under the Trump administration as we had during previous administrations.

We worked diligently to maintain the anonymity of the federal scientists taking our survey, providing three different methods for participants to take the survey (online, phone, and a mail-in option). Scientists took advantage of all three methods.

We followed up with reminders nearly weekly. Some scientists who were invited to take the survey did reach out to confirm that UCS and Iowa State University were conducting a legitimate survey, and the link that we sent them was safe to click on. In addition, some agencies communicated to their staff that the survey was legitimate and that experts were free to take it on their own time.

And while we received enough responses for the results to be valid, the final overall response rate on this years’ federal scientists survey sits at 6.9%. Compared to response rates on prior surveys conducted by UCS over the past 13 years, which have typically ranged from 15-20%, this year’s rate is lower. Let’s unpack some potential reasons why, and what the impact may be on interpreting results.

Reasons Why the Response Rate was Low

  1. Fear

It is possible that federal scientists and scientific experts were fearful or reluctant to comment on the state of science under the Trump administration. This may be borne from some political appointees reprimanding career staff for speaking publicly about their work.

Additionally, it is possible that given the heightened threat of cyber-attacks in the modern era, scientists were afraid their information might be monitored or leaked. Survey respondents were given a unique identifier to ensure the integrity of the survey, and while these identifiers were deleted before the survey results were prepared for release, we heard reports that simply being associated with that unique identifier was too much of a barrier.

  1. Discouragement from Senior Leadership

At some offices within the Environmental Protection Agency (EPA) as well as at the Fish and Wildlife Service (FWS), senior leadership sent emails to employees that discouraged them from taking the 2018 UCS survey. FWS emails stated “Requests for service employees to participate in surveys, from both internal and external sources, must be approved in advance of the issuance of the survey.” But this is only true of surveys issued through the agency. Federal employees are not required to receive an ethics clearance to take an outside survey if they take it on their own time and with their own equipment. On the other hand, other offices within the EPA as well as the National Oceanic and Atmospheric Administration (NOAA) and the US Department of Agriculture (USDA) sent emails reminding employees that they were welcome to take the survey given that they took it using their own time and equipment.

  1. Larger Survey Sample

This is the largest survey that UCS has ever conducted. Our prior surveys have been administered to up to 4 agencies, whereas we surveyed 16 agencies, offices, and bureaus this year. It may be easier to achieve higher response rates with smaller survey samples because it is possible for researchers to devote more time to working with the survey sample and building trust.

  1. Lack of Public Directory and/or Job Descriptions

UCS can survey federal scientists because their name, email address, and job title are publicly available, or at least they should be. For some agencies that we surveyed, like the National Highway Traffic and Safety Administration (NHTSA) and the Department of Energy (DOE) who do not have public directories available, we submitted Freedom of Information Act (FOIA) requests for this information (it’s been a year and half, and we still don’t have the directory from DOE). For other agencies, such as the EPA, a public directory was available but didn’t have complete information (e.g., job titles). Having the job title of the career staffer is important as it allows us to narrow down our survey sample to those who are likely to be a scientist or scientific expert. In the case of the EPA, Census Bureau, and DOE’s Office of Energy Efficiency and Renewable Energy (EERE), we did not have this information, so we had to administer the survey to the entire agency, or to only offices that we assumed would do scientific work. This greatly increases the number of individuals in an agency sample such that response rates are likely skewed lower relative to other agencies.

Does this low response rate matter in the interpretation of survey results?

A low response rate can give rise to sampling bias, meaning that some individuals in our survey sample are less likely to be included than others (some suggest that only the most disgruntled employees would respond). However, there is a growing body of literature that suggests that this may not be the case. Counterintuitively, it’s possible that surveys with lower response rates may yield more accurate results compared to those with higher response rates. Another study showed that administering the same survey for only 5 days (achieving a 25% response rate) versus weeks (achieving a 50% response rate) largely did not result in statistically different results. Results that were significantly different across these surveys only differed between 4-8 percentage points.

Further, we have never suggested that the responses received at an agency represent the agency as a whole. Rather, the responses represent the experiences of those who chose to respond. And when hundreds or thousands of federal scientists report censorship, political influence on their work, or funding being distributed away from work just because the issue is viewed as politically contentious…well, we have a problem.

I’m very happy that we gave these scientists a voice, because they had a lot to say and it’s time that they’re heard.