Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

45
831

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

In 2016, the U.S. Census Bureau faced a pivotal choice in its plan to digitize the nation’s once-a-decade population count: build a system for collecting and processing data in-house, or buy one from an outside contractor.

The bureau chose Pegasystems Inc, reasoning that outsourcing would be cheaper and more effective.

Three years later, the project faces serious reliability and security problems, according to Reuters interviews with six technology professionals currently or formerly involved in the census digitization effort. And its projected cost has doubled to $167 million — about $40 million more than the bureau’s 2016 cost projection for building the site in-house.

The Pega-built website was hacked from IP addresses in Russia during 2018 testing of census systems, according to two security sources with direct knowledge of the incident. One of the sources said an intruder bypassed a “firewall” and accessed parts of the system that should have been restricted to census developers.

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

“He got into the network,” one of the sources said. “He got into where the public is not supposed to go.”

In a separate incident during the same test, an IP address affiliated with the census site experienced a domain name service attack, causing a sharp increase in traffic, according to one of the two sources and a third source with direct knowledge of the incident.

Neither incident resulted in system damage or stolen data, the sources said. But both raised alarms among census security staff about the ability of the bureau and its outside security contractor, T-Rex Solutions, to defend the system against more sophisticated cyberattacks, according to five sources who worked on census security, as well as internal messages from security officials that were reviewed by Reuters.

Among the messages, posted on an internal security registry seen by Reuters, was a note observing that T-Rex’s staff lacked adequate forensic capability as recently as June of this year. “In the event of a real-world event such as a significant malware infection,” the team would be “severely limited in its capability to definitively tell the story of what occurred,” the message said.

One of the sources with direct knowledge of the hack involving Russian IP addresses described the internal Census Bureau reaction as a “panic.” The incidents prompted multiple meetings to address security concerns, said the two sources and a third census security source.

Census Bureau spokesman Michael Cook declined to comment on the incidents described to Reuters by census security sources. He said no data was stolen during the 2018 system test and that the bureau’s systems worked as designed.

The work of Pega and T-Rex is part of the bureau’s $5 billion push to modernize the census and move it online for the first time. The project involves scores of technology contractors building dozens of systems for collecting, processing and storing data and training census workers for the once-a-decade count. T-Rex’s security work is projected to cost taxpayers up to $1.4 billion, according to the census budget, making it the largest recipient of the more than $3.1 billion that the bureau set aside for contracts.

The problems with Pega and T-Rex reflect the Census Bureau’s broader struggle to execute the digitization project. The effort has been marred by security mishaps, missed deadlines and cost overruns, according to Reuters interviews over the past several months with more than 30 people involved in the effort.

“The IT is really in jeopardy,” said Kane Baccigalupi, a private security consultant who previously worked on the census project for two years as a member of the federal digital services agency 18F, part of the General Services Administration. “They’ve gone with a really expensive solution that isn’t going to work.”

The potential costs of a hacking incident or a system failure go beyond busted budgets or stolen data. A technological breakdown could compromise the accuracy of the census, which has been a linchpin of American democracy since the founding of the republic more than two centuries ago.

The U.S. Constitution requires a decennial census to determine each state’s representation in Congress and to guide the allocation of as much as $1.5 trillion a year in federal funds. Census data is also crucial to a broad array of research conducted by government agencies, academics and businesses, which rely on accurate demographic statistics to craft marketing plans and choose locations for factories or stores.

In a worst-case scenario, according to security experts, poorly secured data could be accessed by hackers looking to manipulate demographic figures for political purposes. For example, they could add or subtract Congressional seats allocated to states by altering their official population statistics.

The Census Bureau says its information-technology overhaul is on-track. Systems supporting initial census operations – such as creating its address database and hiring workers – are “fully integrated with one another, performance-tested, and deployed on schedule and within budget,” bureau spokesman Cook said.

Cook said that the bureau had conducted a “bug bounty,” a bulletproofing practice in which benevolent hackers are invited to search for vulnerabilities. He called the effort successful but declined to provide details for security reasons.

Lisa Pintchman, a spokeswoman for Cambridge, Massachusetts-based Pega, said the company was selected through a “very rigorous process” and stands by its work. T-Rex, headquartered in Maryland, declined to comment.

The escalating costs and reliability concerns for Pega’s front-end website have prompted the bureau to consider reverting to an in-house system, which remains under construction as a backup, according to three technology professionals involved in the census project. Census spokesman Cook confirmed that the in-house system, called Primus, would be available for use if needed next year.

This exclusive account of the Census Bureau’s technology troubles comes after government oversight agencies have chronicled other security problems, delays and cost overruns.

The Government Accountability Office (GAO), the fiscal watchdog for Congress, has said the 2020 census is at high risk for a breach or system outage that could prevent people from filling out surveys. The GAO has also said the bureau’s information technology systems won’t be fully tested before the census kicks off for almost all Americans on April 1, 2020, and that 15 of the bureau’s systems – including Pega’s data collection mechanism – were at risk of missing development deadlines ahead of the census.

The Inspector General of the Department of Commerce, meanwhile, in October announced plans to audit the bureau’s technology operations, months after identifying mismanagement of its cloud data-storage system that left it vulnerable to hackers.

Cook declined to comment on the audit but said the bureau is poised to “conduct the most automated, modern, and dynamic decennial census in history.”

The effort to move the census online aims to streamline the counting process, improve accuracy, and rein in cost increases as the population rises and survey response rates decline. Adjusting for 2020 dollars, the 1970 census cost $1.1 billion, a figure that rose steadily to $12.3 billion by 2010, the most recent count. The 2020 tally is projected at $15.6 billion, including a $1.5 billion allowance for cost overruns.

The bureau’s technology woes mounted outside the limelight, as Washington focused on the Trump administration’s push to add a question asking census respondents if they were U.S. citizens, part of a larger effort to curb illegal immigration.

https://www.reuters.com

The president abandoned that effort in July after the U.S. Supreme Court rejected it, cheering civil rights groups who had worried it would dissuade immigrants from responding and cost their communities political representation and federal dollars. Still, an October 18 study by the nonpartisan Pew Research Center found that more than one-fifth of Hispanics say they may not participate in next year’s census, compared to 12% of whites.

‘SINGLE POINT OF FAILURE’

The census technology overhaul got off to a late start, in part because Congress gave the bureau less funding than it requested for most of the decade. Pressed for time, bureau leadership at times prioritized speed over security, according to four people familiar with the bureau’s security operations.

New technology systems, they said, were tested in settings that were vulnerable to hackers despite carrying unresolved risks that had been identified by the bureau’s in-house security team. The testing was authorized by bureau leadership and supported by T-Rex, over the objections of the in-house security officials, who wanted the vulnerabilities fixed first, three of the people said. It stoked internal tensions that ultimately led one security boss to quit his post, the people said.

The Census Bureau’s Cook declined to comment on whether the testing was done over the objections of in-house security officials but said that the bureau follows a strict protocol to minimize risk.

The bureau began rolling out its technology plans in 2014, promising a technological tour-de-force with 52 separate systems. Twenty-seven of them will be used for collecting census data, which include building the website where respondents submit forms and the tools used by door-knockers tasked with nudging stragglers.

Most of the Census Bureau’s $5 billion in technology spending has gone to seven main contractors, who together have tapped another 41 companies as subcontractors, according to public presentations by the Census Bureau in 2018.

Nato Summit Leaders Meet For 70th Anniversary Amid Rows

Within months of the rollout, government advisors from two outside agencies – the U.S. Digital Service and 18F – began warning officials off the sprawling approach, according to Baccigalupi and five other people familiar with the discussions. The outside advisers urged a simpler system, one that would be easier to defend against hacks and glitches.

The Digital Service was created in 2014 by President Barack Obama after the troubled launch of Healthcare.gov, the website meant to allow Americans to sign up for health insurance under Obamacare. Design flaws left the site overwhelmed by higher-than-expected traffic and prevented many users from registering for weeks. Digital Service officials saw the 2020 census as a potential repeat of that fiasco, two of the people said.

The General Service Administration’s 18F unit – named for the address of its Washington, D.C. office – functions like a private-sector consultant and is paid by agencies seeking technology help.

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

18F declined to comment for this story, and the Digital Service did not respond to requests for comment.

The debate between Census Bureau leadership and its advisors from the Digital Service and 18F focused on two broad approaches to software production: monolithic versus modular.

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

A monolithic framework – like the one envisioned by Census Bureau officials – bundles different functions into one system. In the case of the census, that could mean a system that allows people to answer the survey on a website, translates incoming responses into data and stores it. Monolithic systems can be easier to build, but critics say they become hopelessly complex when something goes wrong. A problem with one function can shutdown the whole process.

“It’s a single point of failure,” Baccigalupi said.

In a modular system, by contrast, engineers build different pieces of software for each function, then write code to allow them to interact. While it’s more challenging to move data through different components, the risk of a system collapse is much smaller. If one function breaks, others can still work while it’s repaired.

Census officials brought in 18F and Digital Service consultants on long-term secondments to help with aspects of the project but largely ignored their recommendations to take a more modular approach, said 18F’s Baccigalupi and Marianne Bellotti, a former agent at the Digital Service who consulted on the project in 2017.

“I told them pretty consistently in 2017: If you suffer a denial-of-service attack, I’m not sure your architecture can withstand it,” Bellotti said.

In a denial-of-service attack, a hacker tries to prevent legitimate users from accessing a program, often by overwhelming it with more connection requests than it can process. Any extended outages during the census would reduce response rates, compromising the accuracy of the data and making it more expensive to collect.

Cook, the Census spokesman, did not comment on why the bureau chose a more monolithic approach but said the consultants recommending against that path did not fully understand its systems.

“18F and USDS looked at portions of our systems and provided recommendations, but neither group had an overall understanding of how those systems integrated or their capabilities,” Cook said.

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

RISING COSTS

Bellotti and Baccigalupi say they told the bureau repeatedly in 2016 and 2017 that Pega’s technology wasn’t well-suited to its central tasks – building the self-response website and the mobile applications to be used by census door-knockers. Pega’s code, they argued, would require so much customization that the final product would be slow and prone to glitches.

“If you want to build the fastest car in the world, you build that car from scratch,” Baccigalupi said. “You don’t try to customize a tour bus until it’s the fastest car in the world.”

The Census Bureau’s outside advisers from Carnegie Mellon University’s Software Engineering Institute shared the concern and told the bureau in a 2016 memo, which was reviewed by Reuters, that commercial products such as Pega’s “are not designed to meet an organization’s specifications.”

Neither the bureau nor Pega commented on the assertion that the need for customization made the system expensive and unreliable.

Before hiring Pega, the bureau already had a workable system for data collection, built by in-house staff, Baccigalupi said. Starting in 2014, small teams had fashioned prototypes for online responses and mobile apps that seemed to work. The online response prototype, known as Primus, had been built at little cost beyond the salaries of the half-dozen or so coders.

The in-house systems were tested, and Primus was used in a real-world setting during smaller surveys conducted by the bureau. All performed well, John Thompson, who served as Census Bureau director from 2013 to 2017, said in an interview.

In a 2016 public report explaining its choice to go with an outside contractor, the bureau called Pega’s product a “commercial off-the-shelf solution” that could work with minimal alterations. Pega would do what Primus and the in-house mobile apps could do, but cheaper, with an estimated price tag of $84.5 million, compared to the $127 million forecast for building in-house. Pega would also supply other key functions, such as transferring user responses to data storage.

The reality was messier. Pega’s off-the-shelf solution has required so much modification that it has become “unrecognizable,” said one former Census Bureau official involved in the contracting process. In January 2018, the bureau nearly doubled Pega’s cost estimate to $167.3 million. It has spent about $149 million so far.

Contract documents reviewed by Reuters showed about $121 million of Pega’s contract has gone toward “contracting services,” a category that two former bureau contracting officials said typically refers to the labor required to write and customize code. The figure is more than 13 times Pega’s initial estimate for contracting services.

The bureau did not comment on the escalating costs. Pintchman, the Pega spokeswoman, said the work is “on budget” and that “any changes in estimates would be a result of changes in project scope as well as the Census Bureau identifying additional opportunities for us to add value.”

Thompson, who ran the bureau at the time it decided on Pega, described the decision as a “tough call.” While Thompson and his team viewed Primus as capable of scaling up for the 2020 Census, he said the prospects for scaling up the in-house prototypes for census-worker mobile apps were less certain.

As Pega’s problems have become more clear, Census officials have considered reverting to Primus, the in-house system, for data collection, said three sources familiar with the bureau’s thinking. As recently as this summer, they were instructing employees “to build Primus out, in case it was needed,” said one of those people.

SECURITY INCIDENTS

The only full-scale test of the system took place in Providence, Rhode Island, last year. The bureau conducted a kind of dress rehearsal – essentially a mini-census, with respondent data collected and stored online.

That’s when the system was accessed from IP addresses in Russia, the two census security sources said. Other hackers launched a domain name system attack on the website, which one source described as similar to a denial-of-service attack.

The domain name system attack was not as worrisome as what it revealed about the abilities of T-Rex to respond to such a threat, according to five people involved in census security.

T-Rex staffers “didn’t know how to access the cybersecurity defense tools that were in place, and they didn’t know what to look for,” said a person familiar with the operation. This source added that the bureau had purchased a license to use forensic-analysis software, called EnCase, to investigate hacks more than a year earlier, but T-Rex had yet to fully integrate EnCase into the security system when the security incidents occurred.

T-Rex’s security work had encountered trouble early on. The GAO reported that, by June of 2018, Census’ Office of Information Security (OIS) had flagged more than 3,000 security compliance deficiencies, 2,700 of which were related to components being developed by T-Rex.

OIS voiced concern over the flags and recommended addressing the bulk of them before testing, according to two security officials familiar with the matter. But bureau leadership authorized live-testing of the systems anyway to keep the project on schedule, the people said. The bureau’s Office of Information Security chief, Jeff Jackson, quit his post in October out of frustration over his office’s lack of influence on the project, two sources familiar with the matter said. Jackson did not respond to requests for comment.

A June report by the Department of Commerce’s Office of Inspector General called attention to other snafus. It revealed that, for a prolonged stretch in 2018, the bureau lost the codes needed to gain unrestricted access to its Amazon-based cloud data-storage system. Without the codes, the IG reported, the bureau could not have stopped a hacker from accessing or destroying data stored in the cloud.

The IG, in an October 17 letter to Census Director Steven Dillingham, said it would “immediately” begin auditing the bureau’s technology to “determine the effectiveness of security measures.”

Baccigalupi, the former 18F consultant, called the project’s problems to date “infuriating” given the high cost to taxpayers, and said the bureau’s internal staff could have built the systems better and cheaper.

“Those teams are eager to do it,” Baccigalupi said, “and demoralized to see bad and expensive software going out instead.”

Reporting by Nick Brown; Editing by Richard Valdmanis and Brian Thevenot

Special Report 2020 U.S. Census Plagued By Hacking Threats Cost Overruns

45 COMMENTS

  1. I?d have to check with you below. Which is not something I generally do! I take pleasure in reviewing a blog post that will make individuals believe. Additionally, thanks for enabling me to comment!

  2. One more thing. I believe that there are numerous travel insurance web sites of reputable companies than enable you to enter your vacation details and get you the estimates. You can also purchase this international travel insurance policy online by using your credit card. All that you should do is usually to enter your current travel specifics and you can start to see the plans side-by-side. Just find the program that suits your financial budget and needs after which it use your bank credit card to buy it. Travel insurance on the internet is a good way to check for a reputable company to get international holiday insurance. Thanks for giving your ideas.

  3. I’m not sure where you’re getting your information, but good topic. I needs to spend some time learning more or understanding more. Thanks for excellent information I was looking for this info for my mission.

  4. I like the valuable information you provide in your articles. I will bookmark your blog and check again here frequently. I am quite sure I’ll learn lots of new stuff right here! Best of luck for the next!

  5. As I site possessor I believe the content material here is rattling fantastic , appreciate it for your hard work. You should keep it up forever! Best of luck.

  6. My partner and I absolutely love your blog and find many of your post’s to be exactly what I’m looking for. Does one offer guest writers to write content for you personally? I wouldn’t mind publishing a post or elaborating on most of the subjects you write concerning here. Again, awesome website!

  7. I know this if off topic but I’m looking into starting my own blog and was curious what all is required to get set up? I’m assuming having a blog like yours would cost a pretty penny? I’m not very internet savvy so I’m not 100 certain. Any suggestions or advice would be greatly appreciated. Kudos

  8. I intended to post you a bit of observation just to say thanks a lot once again just for the wonderful knowledge you have discussed in this case. It was simply strangely generous of people like you to give extensively what exactly some people could have advertised as an electronic book to help with making some dough for their own end, even more so now that you might well have done it if you ever desired. The good tips additionally worked to be the fantastic way to be certain that most people have the same dream similar to my own to learn a whole lot more with regard to this issue. I am certain there are some more pleasurable opportunities in the future for people who look over your website.

  9. Thanks for your posting. I also feel that laptop computers have grown to be more and more popular today, and now tend to be the only type of computer employed in a household. This is because at the same time potentially they are becoming more and more very affordable, their processing power keeps growing to the point where they can be as potent as personal computers coming from just a few years back.

  10. Enjoyed examining this, very good stuff, regards . “Hereafter, in a better world than this, I shall desire more love and knowledge of you.” by William Shakespeare.

  11. I intended to post you this very small remark in order to thank you very much once again on the incredible things you’ve discussed on this website. It is really surprisingly generous of people like you to provide easily just what many individuals would have marketed as an ebook in making some dough on their own, precisely given that you might have tried it if you decided. Those good tips likewise worked like the good way to understand that other people have the same dream the same as my very own to know the truth good deal more concerning this matter. I believe there are millions of more enjoyable periods up front for people who scan through your blog.

  12. An intriguing discussion deserves comment. I think that you need to create much more on this subject, it may not be a forbidden subject yet generally people are not nearly enough to talk on such subjects. To the following. Thanks

  13. Please let me know if you’re looking for a article writer for your weblog. You have some really great posts and I believe I would be a good asset. If you ever want to take some of the load off, I’d really like to write some content for your blog in exchange for a link back to mine. Please blast me an email if interested. Thanks!

  14. There are some intriguing points in time in this post yet I don?t know if I see all of them facility to heart. There is some credibility yet I will take hold opinion up until I check out it even more. Good post, many thanks and we desire a lot more! Added to FeedBurner as well

  15. Fantastic web site. Plenty of useful info here. I’m sending it to some friends ans additionally sharing in delicious. And of course, thank you on your effort!

  16. I do believe all the ideas you’ve presented for your post. They’re very convincing and will definitely work. Still, the posts are very brief for beginners. May you please prolong them a little from next time? Thank you for the post.

  17. I have been surfing online more than 3 hours today, yet I never found any interesting article like yours. It is pretty worth enough for me. In my opinion, if all web owners and bloggers made good content as you did, the internet will be a lot more useful than ever before.

  18. Pretty section of content. I just stumbled upon your web site and in accession capital to assert that I acquire actually enjoyed account your blog posts. Any way I’ll be subscribing to your augment and even I achievement you access consistently rapidly.

  19. I do trust all the ideas you’ve introduced on your post. They are very convincing and can definitely work. Nonetheless, the posts are very short for starters. Could you please lengthen them a little from next time? Thanks for the post.

  20. I¡¦ll immediately clutch your rss as I can not to find your e-mail subscription link or e-newsletter service. Do you have any? Kindly permit me realize in order that I may subscribe. Thanks.

  21. I will immediately take hold of your rss as I can’t to find your email subscription hyperlink or newsletter service. Do you have any? Kindly permit me understand so that I may just subscribe. Thanks.

  22. I’m extremely impressed with your writing skills as well as with the layout on your blog. Is this a paid theme or did you modify it yourself? Either way keep up the excellent quality writing, it’s rare to see a nice blog like this one these days..

  23. Hey there would you mind sharing which blog platform you’re using? I’m planning to start my own blog in the near future but I’m having a hard time selecting between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your layout seems different then most blogs and I’m looking for something completely unique. P.S Sorry for getting off-topic but I had to ask!

  24. I really like your blog.. very nice colors & theme. Did you design this website yourself or did you hire someone to do it for you? Plz respond as I’m looking to construct my own blog and would like to know where u got this from. appreciate it

  25. Hey there, You’ve performed an incredible job. I’ll certainly digg it and individually recommend to my friends. I’m sure they’ll be benefited from this web site.

  26. Have you ever considered creating an e-book or guest authoring on other sites? I have a blog centered on the same ideas you discuss and would love to have you share some stories/information. I know my readers would enjoy your work. If you are even remotely interested, feel free to send me an email.

  27. Aw, this was a really wonderful article. In concept I would love to place in composing like this additionally? taking some time and actual effort to make a great write-up? yet what can I claim? I hesitate alot as well as never seem to get something done.

LEAVE A REPLY

Please enter your comment!
Please enter your name here