At the HEART of the SolarWinds Breach - Privileged Access in Active Directory

Folks,

The SolarWinds Breach is likely the most sophisticated, damaging and pervasive breach the world has witnessed yet.

This was a highly sophisticated supply-chain attack in which its perpetrators breached a prominent software vendor and planted a backdoor in an update patch, to very easily gain a foothold in the networks of thousands of organizations, and subsequently gain elevated credentials, ultimately gaining access to just about everything on-premises and in the Cloud. 

Objective

The SolarWinds Breach was discovered in December 2020. Since then, there have been numerous blog posts written on it, most notably those by FireEye, Volexity and Microsoft, and they describe various aspects of this breach in great detail.

The objective of this post is to pinpoint the most salient (cardinal) part of the SolarWinds breach i.e. the part that actually enabled and empowered its perpetrators to be able to access just about everything, both on-premises, and in the Cloud.  

Introduction

From the United States Government to Microsoft, the SolarWinds Breach likely impacted 17,000 organizations worldwide.

Do you know what these 17,000 organizations have in common?  >  They ALL operate on Microsoft's Active Directory.

Today, from the U.S. Government to the Fortune 1000, including almost all cyber security and cloud computing companies, at the very foundation of IT, cyber security and privileged access at 85% of organizations worldwide lies Active Directory.

I'm Sanjay Tandon, formerly Program Manager for Active Directory Security on Microsoft Corporation's Windows Server Development Team, and today, CEO of Paramount Defenses. These are my observations on the SolarWinds Breach -

Essence

Here's a highly simplified summary of how the perpetrators behind the SolarWinds Breach carried out these attacks -

I. Initial Access 

Step 1 - Identify a software vendor whose products are widely deployed worldwide  >  SolarWinds

Step 2 - Identify one of their products and become familiar with its inner workings  >  SolarWinds Orion

Step 3 - Compromise this vendor, gain access to its product's source-code and embed a covert C2 backdoor

Step 4 - Wait for malicious backdoor to be delivered to customers worldwide via legitimate software update

 

II. Execution, Command and Control

Step 5 - Load embedded backdoor during legitimate software application installation, evading suspicion  

Step 6 - Use calls from malicious DLL to remote network, to control payload and gain command in internal network

 

III. Privilege Escalation, Access and Data Exfiltration

Step 7 - Once in, target and gain administrative access in victimized organization's on-premises environment

Step 8 - Use gained administrative access to establish significant foothold in victim's on-premises environment

Step 9 - Use gained administrative access to forge SAML tokens and gain administrative access to the Cloud

Step 10 - Use gained administrative access to perform asset discovery and data exfiltration, on-prem and in Cloud 

 

The over simplification of the description of the above steps is deliberate, as various other notable blogs already provide a sufficiently detailed description of these steps. The objective of this post is to hone-in on and identify the most salient step, the enabling one, without which the perpetrators could not have gained access to or exfiltrated the vast majority of data.     

The Salient Step

The most important, enabling and salient (cardinal) step in the entire SolarWinds Breach attack methodology, is Step 7

i.e. the one in which the perpetrators gained administrative access in the victim organization's on-premises environment.

(The details of the ways in which the perpetrators may have gained administrative access are covered below in this post.)

Up until this step, all that the perpetrators had managed to accomplish was gaining a foothold in the victim organization's environment. Compromising the SolarWinds software to do so was merely a substantially efficient and ingenious way to breach the network of thousands of organizations. In itself, it did not provide them all-powerful administrative access.

From a technical perspective, up until this step, all that the perpetrators had managed to obtain was the same level of access that most employees and computers have in a network aka "Authenticated User" level access. Nothing more. 

This cannot be stressed enough and is proverbially a trillion dollar point - without this one defining step, i.e. targeting and gaining administrative access by escalating privilege from an authenticated user to an administrative user, the perpetrators simply could not have been able to gain access to or exfiltrate the vast majority of data, both on-prem and in the Cloud.

   

Evidence

Microsoft has extensively researched the SolarWinds Breach and published several blog posts on the SolarWinds breach.

The following snippets from a Microsoft blog post titled Customer Guidance on Recent Nation-State Cyber Attacks by the Microsoft Security Response Center, penned by my former Microsoft colleague John Lambert, on December 13, 2020, provide sufficient evidence of the above -

"In actions observed at the Microsoft cloud, attackers have either gained administrative access using compromised privileged account credentials (e.g. stolen passwords) or by forging SAML tokens using compromised SAML token signing certificates."

"In the cases we have determined that the SAML token signing certificate was compromised, common tools were used to access the database that supports the SAML federation server using administrative access and remote execution capabilities."  

"In other cases, service account credentials had been granted administrative privileges; and in others, administrative accounts have may have been compromised by unrelated mechanisms."

"Typically, the certificate is stored on the server that provides the SAML federation capabilities; this makes it accessible for anyone with administrative rights on that server, from storage or by reading memory."  

"Once the certificate has been acquired, the actor can forge SAML tokens with whatever claims and lifetime they choose... .... this includes forging a token which claims to represent a highly privileged account in Azure AD."

In short, it is abundantly clear that the perpetrators first gained administrative access in the on-premises environment, and once they had done so, they used that administrative access to compromise the SAML federation server, which provided them access to the SAML certificate, which they used to gain administrative access in the Microsoft cloud.

Now, The Most Important Thing

As mentioned above, to date several prominent cyber security companies including Microsoft, FireEye, Volexity and others have published detailed blog posts on the SolarWinds Breach, and go to great lengths to provide highly specific details.

However, most of these blog posts, especially those from Microsoft are missing ONE specific detail, and in my opinion, that one detail is the most important thing in the entire SolarWinds Breach incident and narrative, and that is the following - 

While they all refer to and mention administrative access and administrative privileges in on-premises environments numerous times, not once do they explicitly mention and/or tell you that by "administrative access/privileges in on-prem environments" what they are actually referring to is Active Directory Privileged User accounts i.e. domain user accounts that possess privileged access in foundational on-premises Active Directory environments!

Active Directory Domain Admins Security Group

It is Active Directory Privileged User accounts i.e. domain user, computer and service accounts, colloquially known as Domain Admin accounts, that possess domain-wide privileged access in on-prem Active Directory domains that the SolarWinds perpetrators (had to have) compromised to gain "administrative access" on-prem, and in the cloud!

In short, without having compromised an Active Directory Privileged User Account, it would be virtually impossible for the perpetrators in the SolarWinds Breach to gain "administrative access", and subsequently obtain access to and exfiltrate the vast majority of organizational data, on-prem and in the cloud.

Given the above, absolutely no mention whatsoever of  "Active Directory" in Microsoft's blogs, is a concerning mystery.

Note: It seems to me that a complete lack of mention of "Active Directory" may likely be deliberate, considering that given Microsoft's recent and complete focus on Azure Active Directory, they seem to be largely ignoring Active Directory, and no longer seem to have or provide any/sufficient guidance or solutions to help customers secure "Active Directory", even though over 85% of organizations worldwide are still operating on Active Directory.

Microsoft seems to have acknowledged this shortcoming during Brad Smith's testimony to the U.S. Congress. Quoting Mr. Smith - "And yet we only have direct visibility to the attack when it then moved to the cloud." In other words, the almost $ 2 Trillion Microsoft does not have the means to gain visibility in on-premises environments.    

 

Unequivocal Proof

The following snippets from yet another Microsoft blog post, titled Deep dive into the Solorigate second-stage activation, documenting observed attack behavior, unequivocally prove that the perpetrators specifically searched for and targeted privileged user accounts in Active Directory -

"Attackers executed multiple times the legitimate ADFIND tool to enumerate domains, remote systems, accounts and to discover federated domains. Specific example provided: [renamed-adfind].exe -h [machine] -f (name="Domain Admins") member -list" 

"Attackers obtained Ticket Granting Service (TGS) tickets for Active Directory Service Principal Names (SPNs) to crack offline (e.g., Kerberoasting)"

"Attackers used fsutil to check available free space before executing collection, enumeration, exfiltration, or launching attacks like DCSync which might create large files on disk."

"Attackers attempted to access Group Managed Service Account (gSMA) passwords with account credentials they have already obtained."

"Attackers leveraged privileged accounts to replicate directory service data with Domain Controllers"

The fact that the perpetrators used a tool like ADFIND to specifically search for domain admin accounts i.e. members of the all powerful Domain Admins security group, unequivocally proves that subsequent to having breached the perimeter, the perpetrators targeted and compromised Active Directory privileged user accounts.

Further, if, as suggested by Microsoft's research, the perpetrators in fact used DCSync in the SolarWinds Breach, they likely compromised the credentials of all domain accounts, and thus would have been able to easily gain access to and exfiltrate any and possibly every IT asset of choice, both on-premises and in the Cloud.

This also constitutes sufficient and additional proof that gaining access to a single Active Directory privileged user account is sufficient to gain unrestricted domain-wide access in an organization, and that could easily result in a colossal breach.

It cannot be overstated that access to a single Active Directory Privileged User account i.e. just ONE such account, is sufficient to obtain unrestricted domain-wide access to all IT assets, both on-premises and (by extension) in the Cloud.

SolarWinds Perpetrators Only Targeted Active Directory Environments

The perpetrators of the SolarWinds Breach were sophisticated, and they only targeted Active Directory environments.

According to additional research published by FireEye on the SUNBURST backdoor -

  

"The backdoor also determines if the system is joined to an Active Directory (AD) domain, and if so, retrieves the domain name. Execution ceases if the system is not joined to an AD domain." 

This clearly proves that while developing their malicious payload (SUNBURST), the perpetrators of the SolarWinds Breach specifically programmed their backdoor to determine and ensure that the system on which the payload was running was in fact joined to Active Directory. If it wasn't, the payload was instructed to cease execution. 

This is how important, pervasive and mission-critical Active Directory is today at thousands of organizations worldwide.  

    

Active Directory - The Heart of Privileged Access Worldwide

Today, from the entire United States Government to the global Fortune 1000, Active Directory is the very foundation of IT, bedrock of cyber security and heart of privileged access, at 85% of all government at business organizations worldwide.

Here's why -

1. The entirety of an organization's user accounts and their credentials reside in Active Directory


2. The entirety of an organization's computers are joined to and have a secure channel with Active Directory


3. The entirety of an organization's IT assets (files, folders etc.) are protected by Active Directory security groups


4. The entirety of an organization's end-point management and security policies are deployed from Active Directory


5. The credentials of the entirety of an organization's Active Directory accounts are synced with Azure AD in the Cloud

Further, to facilitate the management and protection of these organizational user and computer accounts, security groups and policies, OUs and containers, an ocean of privileged access is delegated and provisioned inside Active Directory.

Finally, the most powerful administrative (privileged) accounts and groups, i.e. all Domain Admin equivalent accounts and groups, that possess unrestricted organization-wide access, are all stored, managed and protected in Active Directory.

In other words, worldwide, not just the Keys to the Kingdom, the keys to every door in the kingdom lie in Active Directory.

(As such, in Windows based networks, Active Directory is the account/credential database used by Kerberos, the native authentication protocol in Windows, and every domain controller is a Kerberos Key Distribution Center (KDC). Based on this fact alone, Active Directory is also the foundation of cyber security in a Windows Server based IT infrastructure.)

Thus, factually speaking, an organization's Active Directory is the single most valuable IT and corporate asset, worthy of the highest protection, because it is the heart of privileged access and the foundation of an organization's cyber security.

Note: As described above, today Active Directory is the very foundation of IT, cyber security and privileged access at 85% of all organizations worldwide, and yet, there's virtually NO mention of "Active Directory" in Microsoft's blogs on the SolarWinds Hack, AND hardly any guidance provided anymore by Microsoft on "Active Directory Security," let alone guidance to organizations on the paramount importance of adequately protecting all their Domain Controllers and accurately identifying, minimizing and protecting all their Active Directory privileged accounts. 

Privileged Access in Active Directory

Speaking of privileged access in Active Directory, there exists an ocean of privileged access in every Active Directory.

Specifically, from the CEO's domain user account to the all-powerful Domain Admins security group, and from the domain computer account of every domain-joined computer to every domain security group that is used to protect millions of IT resources company-wide, literally everything in Active Directory is an object, protected by an ACL (access control list), within which reside hundreds of Active Directory security permissions, each one of which allows or denies one of over eighty different kinds of permissions to some user, service account, group, nested group, well-known security principal etc. etc., and it is together, i.e. collectively that millions of Active Directory security permissions in the ACLs of thousands of Active Directory objects, ultimately determine exactly who has what privileged access, where and how in Active Directory.

Avenues to Gaining Privileged Access in Active Directory

Obtaining privileged access in Active Directory is the new holy grail for perpetrators, and the #1 target today, because once such access is obtained, the perpetrator can obtain access to just about everything, on-premises, and in the Cloud.

It remains a less known fact that virtually all major recent cyber security breaches of the last decade, including JP Morgan, Sony Hack, Anthem Breach, the OPM Breach, Snowden, the United Nations Breach and now the SolarWinds Breach, involved the compromise and misuse of a single Active Directory privileged user account.

Traditional Techniques

Novice and intermediate perpetrators generally employ traditional techniques such as password guessing/brute-forcing, Kerberoasting and Pass-the-Hash (PtH) in their attempts to compromise Active Directory privileged user accounts. 

Fortunately for defenders, advances in protection measures have reduced the likelihood of success with such measures.

Advanced Techniques

Professional perpetrators seem to prefer employing advanced techniques that involve escalation of privilege based on the identification and exploitation of excessive access on privileged accounts, groups, and certain objects in Active Directory. 

Here are the Top-5 advanced techniques to gain privileged access in Active Directory -

1. Use Mimikatz DCSync to replicate secrets (i.e. password hashes) from an Active Directory domain 


2. Reset the password of any existing Active Directory Privileged User account e.g. the Administrator account


3. Change the membership of any existing Active Directory Privileged Group e.g. the Domain Admins group


4. Modify the ACL (access control list) protecting the special AdminSDHolder object in Active Directory


5. If Smartcards are in use, disable use of Smartcards on an AD Privileged User's account, then reset its password

The novelty of these five advanced privilege escalation techniques is that their use only requires the perpetrator to have sufficient Active Directory Effective Permissions to be able to enact these administrative tasks in a target Active Directory.

Specifically, the use of these advanced techniques does not require perpetrators to attempt a single move that could raise suspicion or be easily detected, such as moving laterally, compromising DCs, Kerberoasting, PTH etc. All a perpetrator needs to do is avail of the already gained Authenticated User level access to correctly analyze the ocean of security permissions that exists in Active Directory and identify privilege escalation paths leading to Domain Admin accounts.

Note: The risk posed by the use of these advanced techniques is adequately described in The Paramount Brief.  

These advanced techniques are already in use today, and often rely on the use of an inaccurate but freely available tool called Bloodhound. The only tools that can make such determinations accurately are Gold Finger and Gold Finger Mini.

Additional Irrefutable Evidence

On March 17, 2021 CISA (CyberSecurity and Infrastructure Security Agency), an agency of the United States Government released a document titled SolarWinds and Active Directory/M365 Compromise: Detecting Advanced Persistent Threat Activity from Known Tactics, Techniques and Procedures. (This document can be downloaded from CISA's website, here.)

This was very likely the first time anyone had publicly mentioned Active Directory in relation to the SolarWinds Breach !

There you have it!  The United States Government, and specifically CISA, clearly, unequivocally and irrefutably called the SolarWinds Breach what it actually and in fact was i.e. SolarWinds and Active Directory/M365 compromise. 

Here are the most salient parts from that document, that in my opinion, ultimately enabled the perpetrators to compromise the Active Directory Federation Server, which was then used to forge SAML tokens and gain access to the Cloud -

"The actor leveraged privileged accounts to replicate directory service data with domain controllers."

"Maintain awareness of which IPs belong to domain controllers. Add DCs to Replication Allow list, and then configure monitoring systems to alert on DsGetNCChange request originating from any IP not on the "Replication Allow List" (See AdSecurity reference for details.)"

That, right there, indicates that the perpetrators in the SolarWinds Breach had compromised and mis-used an Active Directory privileged account (one that at the very least, had sufficient privileges to replicate secrets from Active Directory,) and further that this AD privileged account was then likely used to to perform DCSync, which would have then led to the compromise of the credentials of every single Active Directory account, including all Active Directory privileged accounts. 

Twice (; Not Just Once)

Then, on May 14, 2021, CISA released eviction guidance to help organizations remove (, and I quote) "Russian State-Sponsored Threats from Compromised Networks" penned by Eric Goldstein, Executive Assistant Director at CISA.

This was the second time that someone has publicly mentioned Active Directory in relation to the SolarWinds Breach! -

"CISA encourages any organization that may gave been impacted by the SolarWinds and Active Directory/M365 compromise to consider implementing these steps across their applicable environments."

Incidentally, CISA's  first alert on the SolarWinds breach was issued on December 17, 2020, specifically, Alert AA20-352A titled Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations, and in its initial alert on the SolarWinds Breach, there was no mention of Active Directory.

Then, on January 06, 2021, an update was annotated to the alert description, and it read -       

"The adversary's initial objectives, as understood today, appear to be to collect information from victim environments. One method the adversary is accomplishing this objective is by compromising the SAML signing certificate using their escalated Active Directory privileges." 

There you have it again! To anyone who may have a shred of doubt left, it should now be abundantly clear that access obtained to resources in the Cloud was only made possible by the (mis)-use of escalated Active Directory privileges!

Irrefutable Evidence

As mentioned above, on May 14, 2021, CISA released eviction guidance on the SolarWinds Breach, which can be found online at Eviction Guidance for Networks Affected by the SolarWinds and Active Directory/M365 Compromise.

Here are some relevant snippets from their eviction guidance -

"Define the true scope by identifying trust boundaries (including between Active Directory [AD] forests and domains) and determining the enterprise assets to which this guidance applies."

"Ensure that all endpoints that will need to be updated are powered on for as long as possible during the eviction phase. Note: this action is necessary for all vital changes to AD to be pushed to all systems in the environment prior to reconnection and also to verify that all systems are rebooted." 

"Reset the Kerberos Ticket Granting Ticket account (krbtgt)"

"Rebuild and reimage systems"

"Regain control of the AD and ADFS"

"Ensure there are unique and distinct administrative accounts for each set of administrative tasks"

"Enforce the principle of least privilege. Remove all accounts that are unnecessary; remove privileges not expressly required by an account's function or role"

"Enforce MFA for all administrative accounts and functions"

"Create and establish PAWs for administrative accounts and mandate use for administrative functions (AD Administrators first, at minimum.)"

"Rotate all the secrets. Reset passwords for i. All AD accounts with elevated privileges (such as administrators, ii. All AD service accounts, iii. Directory Services Restore Mode (DSRM) account on domain controllers, iv. All AD accounts, v. Accounts with suspicious activity or whose credentials existed on compromised systems, such as affected SolarWinds servers, vi. Any account where Smartcard/PIN is not enforced, vii. All AD user accounts, viii. All Windows local administrative accounts, ix. Non-AD application privileged accounts, x. Network device administrative accounts, xi. Non-AD HVA user accounts"

I could go on and on, but I trust you'll get the drift i.e. - the vast majority of the advice/guidance that CISA is providing to organizations is that which one provides in the event of an Active Directory Security Breach/Compromise, and the reason this guidance is being given is because a single Active Directory Privileged User account was found to have been compromised in this breach, which as I have indicated, is tantamount to a complete Active Directory forest-wide breach. 

In other words, as I have been saying for months now, the most critical and salient aspect and element of the SolarWinds Breach was the one in which the perpetrators compromised an Active Directory Privileged User account. (Just ONE.)

Once they had done so, they had obtained the "Keys to the Kingdom." From that point on, obtaining access to any resource of choice, including the AD Federation Server on which the SAML Signing Certificate resided was child's play!

As a corollary,  it can also be stated with a high degree of confidence that had the perpetrators of the SolarWinds Breach not been able to compromise an Active Directory privileged user account, it would have been substantially difficult for them to compromise any other assets, including and notably the AD Federation Server, which is what ultimately granted them access to everything in the Cloud.

I cannot emphasize this enough - "The compromise of a single Domain Controller or that of a single Active Directory Privileged User Account is tantamount to a complete Active Directory Forest-wide compromise."

 

A Word about Microsoft

This post would be not complete without a brief mention of Microsoft's role in cyber security today. Microsoft is a great company, one I care for deeply and have had the privilege of serving in two vital capacities (from a security perspective.)

Microsoft Active Directory is quite simply the very foundation of IT and organizational cyber security worldwide, and as its developer, Microsoft has a responsibility to adequately support Active Directory, including its security, for years to come.

Unfortunately, it appears that in light of and since Microsoft's new-found focus on its "Cloud" offerings, it seems to have been ignoring/neglecting Active Directory and its security for years now, and I'd like to believe that it is unintentionally so.

If it is intentional, then all I can say is that while it may perhaps seem "convenient" for Microsoft to do so, perhaps with the hope that doing so could eventually force organizations to migrate to Azure, it is not the right thing to do for a two trillion dollar company with such a great responsibility.

In my opinion, a company cannot, one on hand, talk about "trust", "trustworthiness" and "security" and invest heavily in the security of its new ("cloud") products, while on the other hand, seem to not care equally/sufficiently about the security /trustworthiness of its existing currently deployed ("on-premises") products, upon which the majority of the world is still operating, likely knowing that doing so could endanger thousands of its own existing customers and result in breaches, responding to which would ultimately leave organizations with no real choice, but to adopt its new ("cloud") products.

I only say so because there have been several breaches now, wherein traditional on-premises software such as Active Directory and Exchange Server have been targeted and compromised, and it appears that instead of helping organizations understand how to adequately secure and defend these foundational technologies, Microsoft may perhaps have viewed and used these as an "opportunity" to pitch (suggest) its new Cloud offerings as a secure alternative to these ("its own") foundational on-premises technologies. If true, such sales strategies should be beneath a company such as Microsoft.

As such, one would hope that Microsoft would understand, recognize and appreciate that over the past two decades, across the world, thousands of organizations and vendors have collectively invested billions of dollars towards enabling /integrating all sorts of capabilities (SSO, management, security etc.) for/with Active Directory, and that most organizations do not wish to be forced to migrate to the Cloud, and as such, as Microsoft's customers have been paying it handsomely over the years, so they absolutely expect to be able to operate such foundational technologies securely for years.

There is nothing wrong with Active Directory or its security. It is a highly securable technology that has stood the test of time and has remarkably been quite secure, at least as long as Microsoft has provided sufficient guidance and support.

Note: The Cloud is not the answer to today's cyber security problems. Also, most organizations may not realize so, but any security offered by the Cloud comes at the great cost of privacy, and dependency. Securing and defending the very foundational technologies upon which thousands of organizations operate today, is part of the the answer.

If I were leading a two trillion dollar company that played such a foundational role in cyber security, even with towering ambitions for its new products, I would ensure that the company absolutely, substantially and adequately continued to support its existing products, no matter the cost, especially considering that the entire world was operating on them.

As such, the extended end date for Windows Server 2019 support is 2029, so I expect that Microsoft will absolutely and adequately continue to support a technology as foundational and as pervasive as Active Directory, until at least 2029.

I felt the need to share this because over the years I have found that there is hardly any adequate/sufficient guidance, support and/or focus on Active Directory Security or Active Directory Privileged Access, provided by Microsoft, anymore. 

That to me is concerning because, as I have been saying for years, if Microsoft had only continued its commitment to Active Directory and its security, in all likelihood, most of its customers would likely have had more secure Active Directory deployments, and that could possibly have prevented or reduced the likelihood of the occurrence of so many breaches.

For instance, I have been saying for years now that Microsoft needs to do more to at least help its customers understand the paramount importance of (correctly) attaining and maintaining least privileged access (LPA) in their Active Directory. If Microsoft and organizations worldwide had listened to my advice, so many organizations would have had a locked down state of access in Active Directory, and perpetrators would not have been able to use DCSync to compromise them.

In Microsoft's defense, I will also say that it is not Microsoft's sole responsibility to ensure the security of its customers. Ultimately, the responsibility of protecting organizations from getting breached, lies with the organizations themselves.

 

Public Enemy #1 - DCSync

As shared above, the most important (enabling) step in the SolarWinds Breach involved the perpetrators compromising and then misusing a single Active Directory Privileged Account to gain unrestricted access on-premises and in the Cloud.

In regards to the compromise and misuse of Active Directory privileged accounts, today, possibly the single biggest risk to organizations that operate on Active Directory, is that posed by the unauthorized replication of secrets (i.e. password hashes) from an organization's Active Directory, made effortlessly easy by the freely available tool, Mimikatz DCSync.

The unauthorized replication of secrets from Active Directory results in the instant compromise of the credentials of all Active Directory accounts, i.e. the user accounts of the entirety of an organization's employees, contractors, domain computer accounts, the accounts of all executives and all Active Directory privileged user accounts.

This threat is extremely dangerous and it can be enacted by anyone who has a domain user account and sufficient privileges in Active Directory to be able to replicate secrets (i.e. password hashes) from an Active Directory domain.

In terms of privileges, the only privileges that a perpetrator requires to enact this dangerous threat against an organization is to possess two Active Directory Effective Permissions on the domain-root object of an organization's Active Directory domain(s) - i.e. Get Replication Changes and Get Replication Changes All extended rights.

The successful enactment of this threat is tantamount to a complete, organization-wide security breach, and the only way to recover from it is to perform a complete Active Directory Forest-wide recovery.

The simplest and most effective means of protecting an organization from this threat is to identify (i.e. audit) the identities of all domain accounts that currently possess sufficient privileges to be able to successfully perform a DCSync operation against Active Directory, reduce this number down to an absolute bare minimum (e.g. ideally, only the DC computer accounts,) and then monitor for the use of DSGetNCChange requests coming from IPs that do not belong to DCs. 

The reason is simple - if a perpetrator does not have sufficient privileges to enact DCSync, he/she will not be able to enact the risk. Prevention is substantially more valuable than detection, because if you're relying on detection, the damage has already been done i.e. the replication has already started, and by the time you react, it will already have been completed.

Trying to determine who has sufficient privileges on the domain root in Active Directory, with a trustworthy level of accuracy is very difficult, because it involves accurately determining Active Directory Effective Permissions. Accuracy is paramount because one simply cannot afford to miss even one account that has sufficient privileges to be able to run DCSync.    

Note: Gold Finger Mini lets all organizations instantly and accurately find out (identify) exactly how many domain accounts currently possess sufficient privileges to be able to run DCSync against their Active Directory, for free.

In their own best interest, all organizations operating on Microsoft Active Directory are advised to assess and mitigate their own exposure to this serious risk, because the failure to do so could very likely result in an colossal cyber security breach.   

A Word about DCSync

I would like to add this much because it matters - if indeed, it was DCSync that the perpetrators of the SolarWinds Hack used, then I have to tell you that I have been pointing out for years now that Mimikatz DCSync is possibly the single most dangerous risk to Active Directory deployments worldwide.

In fact, here's unequivocal evidence that I have been warning about the danger posed by DCSync for years now -   

July 2016 - A Letter to Benjamin Delpy regarding Mimikatz and Active Directory Security

July 2016 - A Simple $100B Active Directory Security Question for Alex Simons at Microsoft 

August 2016 - How to Lockdown Active Directory to Thwart the Use of Mimikatz DCSync

January 2017 - Trillion-Dollar Cyber Security Insight for President Donald Trump

October 2017 - A Massive Cyber Breach at a Company Whilst it was Considering the Cloud

July 2018 - Mimikatz DCSync Mitigation

October 2018 - A Very Simple Trillion $ Cyber Security Multiple-Choice Question

November 2018 - Cyber Security 101 for the C Suite - Active Directory Security is Paramount

February 2020 - The ONE Question NO ONE knows the Answer to at RSA Conference 2020

In essence, if indeed DCSync was used by the perpetrators of the SolarWinds Hack, then it seems that the exact vector that I have been warning about for years now, was in fact used by the perpetrators to elevate privileges and gain the "administrative access" they needed to basically compromise everything at thousands of organizations worldwide.

In early 2016, I had informed the highest levels of the U.S. Government, the CIOs/CISOs of every major U.S. government federal agency, the CIOs of every U.S. State Government as well as the executive leadership of the world's top Fortune 100 companies and the world's top cyber security companies about the serious risk posed by the existence of excessive privileged access in the Active Directory deployments of these organizations, yet I doubt they took my warning seriously.

If only they had, in all likelihood, today, not a single additional domain account other than the domain computer accounts of their domain controllers would have had the privileges necessary to replicate secrets from their Active Directory domains, and, as a result, the perpetrators in the SolarWinds Hack would not have been able to successfully use DCSync, and so many (thousands of) organizations may not have suffered so substantially.

Concluding Thoughts

The sole purpose of penning this blog post was to help organizations worldwide understand that in fact what enabled the perpetrators of the SolarWinds Breach to be able to successfully gain unauthorized access to and exfiltrate vast amounts of data was their ability to compromise and then misuse a single Active Directory Privileged User account.

In the case of the SolarWinds Hack, the perpetrator's intentions seemed to be to exfiltrate substantial amounts of data. 

Likewise, a perpetrator could easily accomplish virtually any objective of choice, whether it be data exfiltration, unleashing ransomware domain-wide, tampering a highly sensitive asset (e.g. software source-code, blue-prints of a highly sensitive project, such as a Nuclear Reactor,) taking over the energy grid of a city/state, compromising a government agency (e.g. an embassy or a military deployment), stealing data (e.g. financial details, customer PII etc.) from a Fortune 100 company etc., if he/she could simply compromise ONE Active Directory Privileged User account. 

I cannot emphasize this enough, so I will say it once more, for the umpteenth time - the compromise of a single DC or a single Active Directory Privileged User account is tantamount to a complete, colossal, organization-wide breach, that can not only result in substantial damage, it can cost millions of dollars and weeks to recover from.

Securing DCs is easy for we know exactly how many we have; unfortunately, the same isn't true of privileged users in AD.

In that regard, it is my professional opinion as former Microsoft Program Manager for Active Directory Security that the accurate identification and subsequent reduction in the number of individuals that possess privileged access in Active Directory is the single most important step organizations can take to protect themselves from such colossal breaches.

Here's why - How can one possibly secure and defend something, if, to begin with, one can't even accurately identify it ?

I will also tell you that today, while there exist over a thousand cyber security companies in the world, including numerous prominent ones such as Palo Alto Networks (PANW), Palantir Technologies (PLTR), CyberArk (CYBR), FireEye (FEYE), CrowdStrike (CRWD), Check Point Software (CHKP), ZScaler (ZS), Splunk (SPLK), CloudFlare (NET), NortonLifeLock (NLOK), Sophos Group (SOPH), SolarWinds (SWI), Tenable (TENB), Varonis (VRNS), VMWare (VMW), Cisco (CSCO), IBM, Intel (INTC), Microsoft (MSFT) etc., today not one cyber security company in the world possesses the capability to help organizations accurately* identify and lockdown privileged access in their foundational Active Directory deployments.

Well, I shouldn't say not one, because there is one. The only company in the world that can do so is Paramount Defenses ; it can empower organizations to instantly and accurately identify privileged users/access, domain-wide, at a button's touch.

Note: This is not about pride or competition. We do not do what the other thousand cyber security companies do, and they do not and cannot do what we do. This is about collaboration and helping make the world a safer place. 

In summary, today's post was about helping the world understand that if you actually take a good, close look at what happened in the SolarWinds Breach, you'll find that the defining step that actually enabled the perpetrators to inflict substantial damage was their ability to compromise and misuse a single i.e. just one Active Directory privileged user account - without it, they could not even have gained restricted access, let alone unrestricted access, to anything and everything on-premises and in the Cloud.

I would like to end with this - while I may no longer be at Microsoft, I (have always strived to) represent Microsoft's finest, and today the impact of my work in cyber security equals that of many of the world's top cyber security companies.

Best wishes,

Sanjay

*Accuracy is paramount, because as evidenced by the SolarWinds breach, it only takes ONE AD privileged user account.

What's common between the Colonial Pipeline Hack and SolarWinds Breach?

Folks,

From the entire U.S. Government to the Fortune 1000, including almost all cyber security and cloud computing companies, today, at the foundation of IT, cyber security and privileged access at 85% of organizations worldwide lies Active Directory.

I'm Sanjay Tandon, formerly Program Manager for Active Directory Security on Microsoft Corporation's Windows Server Development Team, and today, CEO of Paramount Defenses.

Today, I'll share with you what is common between the Colonial Pipeline Hack and the SolarWinds Breach, and day after tomorrow onwards, I'll also provide sufficient technical details, but before I do so, I would like to share a few observations. 

 

Note - The only reason you may want to listen to what I have to say, is because, by virtue of my years at Microsoft and PD, I possess sufficient expertise, IP and capability to be able to help substantially enhance (and if requested, also demo how one could compromise) the foundational cyber security of any/every organization in the world.

Five Observations

I would like to share a few salient observations on the current(ly dismal) state of cyber security at organizations worldwide, because it is my professional opinion that until certain basic deficiencies are addressed, unfortunately, we will continue to witness many more such breaches - 

1. The Current State of Affairs

   It is really sad to see the current state of cyber security at organizations worldwide. Not a month seems to go by without there being yet another high-impact cyber security breach at some prominent organization or the other.

   That said, considering how inadequate the actual state of cyber security preparedness, defenses and proficiency are at most organizations, it is hardly surprising to see so many organizations get breached, ransomware'd etc.

   For instance, consider this - Active Directory (AD) is the very foundation of cyber security at organizations, and a Domain Controller (i.e. the machine on which AD is hosted) is technically the most valuable asset an organization has, yet, at most organizations, DCs remain vastly inadequately protected, and thus vulnerable to compromise.

   If this is the state of DC security at thousands of organizations worldwide, how can there be any security?

   Likewise, the compromise of a single Active Directory Privileged User account is tantamount to a complete Active Directory forest-wide breach, so such accounts must be minimal in number and highly protected. Yet, at most organizations, today there exist an excessively large and unknown number of Active Directory privileged accounts.

   If this is the state of AD privileged accounts at most organizations worldwide, how can there be any security? 

   
   




2. Three Fundamental Deficiencies

   It is my professional opinion that most organizations suffer from three key deficiencies, that ultimately result in inadequate cyber security defenses, leading to breaches - understanding, accountability and empowerment.     

   1. Understanding - Given a vast and dynamic attack surface, and sophisticated threats, it is imperative that all organizations possess a sufficient understanding of how to adequately protect themselves, yet most don't.

   2. Accountability - Security requires a clear chain of ownership and accountability:  Shareholders, customers, partners > CEO > CISO > Director(s) > Domain (and IT) Admins. Yet at most organizations, none exists.

   3. Empowerment - Organizational IT teams need to be adequately empowered to acquire and deploy security measures needed to adequately defend an organization, yet at most organizations, budgets are inadequate.

   
   

   For instance, IT personnel and Domain Admins from thousands of organizations have requested our help, found our unique products (e.g. 1, 2, 3) to be essential, yet so many end up conveying that they just do not have the budget.

   In reality, it is not that they do not have the budget; it is primarily that their executive management simply does not yet possess the required understanding i.e. Active Directory Security directly impacts foundational security and business continuity, and is thus paramount, and consequently their IT personnel are simply not empowered. 

   
   



3. The World is Mostly Reacting

   Sadly, at most organizations, cyber security is only taken sufficiently seriously after they have been breached, and in most instances, the response is similar - the breach is disclosed, then FireEye is called in to investigate, and ultimately, promises are made to enhance security. In the case of govts., broad directives/EOs may be issued.

   FireEye does a thorough investigation and in most cases, the findings are similar i.e. the perpetrators used the same set of well-known techniques and in almost every case, compromised and misused an Active Directory privileged user account to obtain Domain Admin level access, which was then used to achieve their objective. 

   Subsequent to FireEye's investigation, this is priority #1, budget is no longer a problem, a new CISO is hired, half a dozen new cyber security solutions are deployed, millions are spent etc. but the damage has already been done.

   
   



4. Lack of Specifics in Public Discourse

   After every breach, the CNNs and ABCs of the world will extensively cover it, you'll hear interviews from prominent Senators, Congressmen and cyber-security experts, all of whom will speak about the serious impact, the role in national security, the influence of a foreign power etc., yet not one of them mention one piece of specific detail.

   In the absence of details in the public discourse, the actual problem, and the solution that it requires, will largely remain unaddressed, and most cyber security companies out there will likely use this opportunity to convince organizations to deploy their latest cyber security solutions, whether or not they actually make a difference. 

   As a result, in all the noise, and due to the lack of focus on details, the actual specific deficiency/weakness that was exploited, and the attack vector that was used in a specific breach, will often likely continue to remain unaddressed at thousands of other organizations worldwide, paving the way for the next breach and the one after it, and so on.

   For instance, in virtually every major cyber security breach to date, the most damaging part of the breach was made possible by the perpetrator compromising and misusing a single Active Directory privileged user account to fulfill his/her objective, whether it be exfiltrating data, unleashing malware etc. and yet to date, at most organizations worldwide, no one has any idea as to exactly how many users have privileged access in Active Directory because the elephant in the room, i.e. "Active Directory", was not mentioned even once in the public discourse.

   
   



5. The Basics - Secure the Foundation and Deny them the Opportunity

   At its simplest, all security is fundamentally about access control. In order to compromise anything, perpetrators require access - if we reliably deny them the required access, we will have won half the cyber security battle.

   Most importantly, if perpetrators are unable to obtain privileged access, specifically Domain Admin equivalent access, they will almost never able to inflict colossal damage i.e. no widespread ransomware, data exfiltration, etc.

   Towards that end, the most important proactive measure organizations can take to adequately defend themselves is to adequately secure and defend their foundational Active Directory deployments, the two most important parts of which are to 1) secure all DCs (and admin workstations), and 2) accurately identify and minimize the number of accounts that possess privileged access in Active Directory, then fiercely protect every AD privileged account.

   Here's why - An attacker only needs to compromise one DC or one AD privileged user account. That's it. Just ONE.

   Real-world Evidence - If the perpetrators of the Colonial Pipeline attack had not been able to compromise a DC, they would likely not have been able to unleash ransomware. Likewise, if the perpetrators in the SolarWinds Breach had not been able to compromise an Active Directory privileged user account, they would not have been able to gain access to and exfiltrate vast amounts of data on-prem and in the Cloud, at thousands of organizations. 

• Note - If you find this to be high-level and light on technical details, it is so by intent, given its purpose. For those who may wish to judge my competency based on details - one, two, three, four, five, six, etc..
  
  I've also written an innocuous production-level ransomware example to show it could be AD deployed.

What is common between the Colonial Pipeline Hack and the Solar Winds Breach?

In the last few months, two major cyber security incidents, the SolarWinds Breach and the Colonial Pipeline Hack have had a notable impact on the world, the former having impacted the security of thousands of organizations worldwide, and the latter having caused a week long shutdown of the largest oil pipeline operator in the eastern United States.

The one thing that both these attacks had in common was that in each of these cyber security incidents, the perpetrators specifically targeted and successfully compromised the foundational Active Directory deployments of organizations.

Note - The compromise of a single Domain Controller and/or a single Active Directory privileged user account is tantamount to the compromise of an organization's entire foundational Active Directory deployment.

It can be stated with a high degree of certainty that had the perpetrators not been able to compromise the foundational Active Directory deployments of these organizations, in all likelihood, these attacks would not have been successful.

I'll share the relevant technical details of both of these attacks, on this blog, starting day after tomorrow, as stated below.  

Trillion $ Insights

Over the next few days, starting day after tomorrow, I'll share ten specific high-value details that have a direct bearing on the foundational cyber security of every organization operating on Active Directory today; you may wish to tune in.

Day after tomorrow, I'll share the details of what enabled the most impactful part of the SolarWinds Breach right here, and in days to come, I'll also share what enabled the most impactful part of the Colonial Pipeline Hack here.

Sincerely,
Sanjay.

Founder and CEO, 

Paramount Defenses

PS: I am often asked for advice on how to secure Active Directory. 

It being an ocean of a subject, here's the essence of it -

In the hierarchy of security measures, prevention is #1, avoidance is #2, detection is #3 and remediation is #4.

I. Prevention - The most effective measure is prevention; the most effective way of preventing an AD breach is as follows: 

1. Adequately secure and defend every single domain controller (and if used, privileged admin workstations (PAWs))
2. Accurately identify and minimize the number of privileged accounts in Active Directory, then protect all of them.
3. Always follow secure admin practices e.g. do NOT logon to any machine except PAWs using Domain Admin creds.

II. Detection - You may wish to consider using an AD Security Monitoring /Threat Intelligence solution to gain visibility and detect enactment of attacks. It is important to keep in mind that such solutions usually monitor replication so they provide quick but "after-the-fact" insights. In general, the efficacy of such solutions is a function of the timeliness of your response.

III. Remediation - You may wish to consider using an AD Backup and Restore solution, in the event of an incident. An AD restore is an extremely complicated and expensive operation, not to be taken lightly, and only to be used as a last resort.