Russian Hack, Undetected Since Spring, Upends Government Agencies

WASHINGTON — The scope of a hack engineered by one of Russia’s premier intelligence agencies became clearer on Monday, when the Trump administration acknowledged that another federal agency, the Department of Homeland Security, had been compromised. Investigators were struggling to determine what parts of the military, intelligence community and nuclear laboratories were also vulnerable to the highly sophisticated attack.

United States officials did not detect the attack until recent weeks, and then only when a private cybersecurity firm, FireEye, alerted American intelligence that the hackers had evaded layers of defenses.

It was evident that the Treasury and Commerce Departments, the first agencies reported to be breached, were only part of a far larger operation whose sophistication stunned even experts who have been following a quarter-century of Russian hacks on the Pentagon and American civilian agencies.

About 18,000 private and government users downloaded a Russian tainted software update — a Trojan horse of sorts — that gave its hackers a foothold into victims’ systems, according to SolarWinds, the company whose software was compromised.

Among those who use SolarWinds software are the Centers for Disease Control and Prevention, the State Department, the Justice Department, parts of the Pentagon and a number of utility companies. While the presence of the software is not by itself evidence that each network was compromised and information was stolen, investigators spent Monday trying to understand the extent of the damage in what could be a significant loss of American data to a foreign attacker.

The National Security Agency — the premier U.S. intelligence organization that both hacks into foreign networks and defends national security agencies from attacks — apparently did not know of the breach in the network-monitoring software made by SolarWinds until it was notified last week by FireEye. The N.S.A. itself uses SolarWinds software.

One of the most embarrassing breaches came at the Department of Homeland Security, whose Cybersecurity and Infrastructure Security Agency oversaw the successful defense of the American election system last month.

A government official, who requested anonymity to speak about the investigation, made clear that the Homeland Security Department, which is charged with securing civilian government agencies and the private sector, was itself a victim of the complex attack. But the department, which often urges companies to come clean to their customers when their systems are victims of successful attacks, issued an obfuscating official statement that said only: “The Department of Homeland Security is aware of reports of a breach. We are currently investigating the matter.”

Parts of the Pentagon were also affected by the attack, according to a contractor who spoke on the condition of anonymity, but officials were similarly coy.

“The D.O.D. is aware of the reports and is currently assessing the impact,” said Russell Goemaere, a Pentagon spokesman. He added that for security reasons, the Pentagon would “not specify systems that may have been impacted.”

Investigators were particularly focused on why the Russians targeted the Commerce Department’s National Telecommunications and Information Administration, which helps determine policy for internet-related issues, including setting standards and blocking imports and exports of technology that is considered a national security risk. But analysts noted that the agency deals with some of the most cutting-edge commercial technologies, determining what will be sold and denied to adversarial countries.

Nearly all Fortune 500 companies, including The New York Times, use SolarWinds products to monitor their networks. So does Los Alamos National Laboratory, where nuclear weapons are designed, and major defense contractors like Boeing, which declined on Monday to discuss the attack.

The early assessments of the intrusions — believed to be the work of Russia’s S.V.R., a successor to the K.G.B. — suggest that the hackers were highly selective about which victims they exploited for further access and data theft.

The hackers embedded their malicious code in the Orion software made by SolarWinds, which is based in Austin, Texas. The company said that 33,000 of its 300,000 customers use Orion, and only half of those downloaded the malign Russian update. FireEye said that despite their widespread access, Russian hackers exploited only what was considered the most valuable targets.

“We think the number who were actually compromised were in the dozens,” said Charles Carmakal, a senior vice president at FireEye. “But they were all the highest-value targets.”

The picture emerging from interviews with corporate and government officials on Monday as they tried to assess the scope of the damage was of a complex, sophisticated attack on the software used in the systems that monitor activity at companies and government agencies.

After a quarter-century of hacks on the defense industrial establishment — many involving brute-force efforts to crack passwords or “spearphishing” messages to trick unwitting email recipients to give up their credentials — the Russian operation was a different breed. The attack was “the day you prepare against,” said Sarah Bloom Raskin, the deputy Treasury secretary during the Obama administration.

Investigators say they believe that Russian hackers used multiple entry points in addition to the compromised Orion software update, and that this may be only the beginning of what they find.

SolarWinds’s Orion software updates are not automatic, officials noted, and are often reviewed to ensure that they do not destabilize existing computer systems.

SolarWinds customers on Monday were still trying to assess the effects of the Russian attack.

A spokesman at the Justice Department, which uses SolarWinds software, declined to comment.

Ari Isaacman Bevacqua, a spokeswoman for The New York Times, said that “our security team is aware of recent developments and taking appropriate measures as warranted.”

Military and intelligence officials declined to say how widespread the use of Orion was in their organizations, or whether those systems had been updated with the infected code that gave the hackers broad access.

But unless the government was aware of the vulnerability in SolarWinds and kept it secret — which it sometimes does to develop offensive cyberweapons — there would have been little reason not to install the most up-to-date versions of the software. There is no evidence that government officials were withholding any knowledge of the flaw in the SolarWinds software.

The Cybersecurity and Infrastructure Security Agency on Sunday issued a rare emergency directive warning federal agencies to “power down” the SolarWinds software. But that only prevents new intrusions; it does not eradicate Russian hackers who, FireEye said, planted their own “back doors,” imitated legitimate email users and fooled the electronic systems that are supposed to assure the identities of users with the right passwords and additional authentication.

“A supply chain attack like this is an incredibly expensive operation — the more you make use of it, the higher the likelihood you get caught or burned,” said John Hultquist, a threat director at FireEye. “They had the opportunity to hit a massive quantity of targets, but they also knew that if they reached too far, they would lose their incredible access.”

The chief executive officers of the largest American utility companies held an urgent call on Monday to discuss the possible threat of the SolarWinds compromise to the power grid.

For the N.S.A. and its director, Gen. Paul M. Nakasone, who also heads the U.S. Cyber Command, the attack ranks among the biggest crises of his time in office. He was brought in nearly three years ago as one of the nation’s most experienced and trusted cyberwarriors, promising Congress that he would make sure that those who attacked the United States paid a price.

He famously declared in his confirmation hearing that the nation’s cyberadversaries “do not fear us” and moved quickly to raise the cost for them, delving deep into foreign computer networks, mounting attacks on Russia’s Internet Research Agency and sending warning shots across the bow of known Russian hackers.

General Nakasone was intensely focused on protecting the country’s election infrastructure, with considerable success in the 2020 vote. But it now appears that both civilian and national security agencies were the target of this carefully designed hack, and he will have to answer why private industry — rather than the multibillion-dollar enterprises he runs from a war room in Fort Meade, Md. — was the first to raise the alarm.

Analysts said it was hard to know which was worse: that the federal government was blindsided again by Russian intelligence agencies, or that when it was evident what was happening, White House officials said nothing.

But this much is clear: While President Trump was complaining about the hack that wasn’t — the supposed manipulation of votes in an election he had clearly and fairly lost — he was silent on the fact that Russians were hacking the building next door to him: the United States Treasury.

In the near term, government agencies are now struggling to get to the bottom of a problem with limited visibility. By shutting down SolarWinds — a step they had to take to halt future intrusions — many agencies are losing visibility into their own networks.

“They’re flying blind,” said Ben Johnson, a former N.S.A. hacker who is now the chief technology officer of Obsidian, a security firm.

David E. Sanger reported from Washington and Nicole Perlroth from Palo Alto, Calif. Zolan Kanno-Youngs and Alan Rappeport contributed reporting from Washington.