At the HEART of the Colonial Pipeline Hack - Admin Access in Active Directory

Folks,

The Colonial Pipeline Hack may be one of the most high visible impact breaches the world has witnessed yet because it resulted in the shut down of one of America's largest gasoline pipelines for an entire week due to ransomware. 

The Colonial Pipeline attack has been extensively covered by the media (e.g. CNN.) It has also already been the subject of a substantial amount of discussion, including Congressional Hearings, so I am not cover the same high-level details here.

Instead, I am going to shed light on the most important and enabling step in the entire Colonial Pipeline Hack, which is the one that enabled its perpetrators to easily and automatically unleash ransomware enterprise-wide on all its systems.

 

Objective

The Colonial Pipeline Hack occurred between May 06 and May 12, 2021. Since then, there have been several blog posts written on it, notably those by FireEye and Arete, and they describe various aspects of this breach in great detail.

The objective of this post is to pinpoint the most salient (cardinal) part of the Colonial Pipeline Hack i.e. the technical part that actually enabled and empowered its purported perpetrators to easily deploy ransomware company-wide.  

Introduction

Today, from the U.S. Government to the Fortune 1000, including almost all cyber security and cloud computing companies, at the very foundation of IT, cyber security and privileged access at 85% of organizations worldwide lies Active Directory.

I'm Sanjay Tandon, formerly Program Manager for Active Directory Security on Microsoft Corporation's Windows Server Development Team, and today, CEO of Paramount Defenses. These are my observations on the Colonial Pipeline Hack -

Overview

The Colonial Pipeline Hack is the largest cyberattack yet on an oil infrastructure target in the history of the United States. 

In short, perpetrators gained entry in the networks of Colonial Pipeline through a virtual private network account (which allowed employees to remotely access the company's computer network) and they subsequently and ultimately deployed ransomware across the company's entire computer network, resulting in Colonial Pipeline having to shut down its pipeline for an entire week (, causing gas shortages nationwide,) and having to pay millions of dollars in ransom (via Bitcoin.) 

The Salient Step

The most important, enabling and salient (cardinal) step in the entire Colonial Pipeline hack was the following one -

The perpetrators first gained privileged access in Active Directory and then leveraged the ability to deploy group policies to domain-joined computers via Active Directory to automatically deploy their ransomware across Colonial Pipeline's network!

In fact, they seem to have used the exact technique I had warned about and described in sufficient technical detail last year  - How Attackers Could Unleash Ransomware on Thousands of Computers in an Organization using Active Directory

   

Evidence

Both FireEye and Arete seemed to have researched the Colonial Pipeline Hack and published detailed blog posts.

The evidence lies in this snippet from Arete's post Darkside Ransomware: Caviar Taste on your Big-Game Budget  -  

"We observed Darkside payload (e.g. azure_agent.exe.exe) staged on the domain controller in a network shareable folder (e.g. C:\Windows\IME\azure), followed by the establishment of a scheduled task (e.g. \Windows\SYSVOL\domain\Policies\{L0NGMGU1D}\User\Preferences\ScheduledTasks) set with Group Policy and instructing hosts to obtain and execute the payload. This resulted in a fully automated enterprise-wide deployment in less than 24 hours after data was exfiltrated."

There you have it! Its clear that the perpetrators first* gained privileged access in Colonial Pipeline's Active Directory and once they had done so, they used that gained privileged access to leverage Active Directory integrated group policy to automate the effortless deployment of ransomware domain-wide (, eerily similar to as described here over a year ago.)

*It should be clear to most that in order to perform the above, one requires privileged access in Active Directory.

Active Directory - The Heart of Privileged Access Worldwide

Today, from the entire United States Government to the global Fortune 1000, Active Directory is the very foundation of IT, bedrock of cyber security and heart of privileged access, at 85% of all government at business organizations worldwide.

Here's why -

1. The entirety of an organization's user accounts and their credentials reside in Active Directory


2. The entirety of an organization's computers are joined to and have a secure channel with Active Directory


3. The entirety of an organization's IT assets (files, folders etc.) are protected by Active Directory security groups


4. The entirety of an organization's end-point management and security policies are deployed from Active Directory


5. The credentials of the entirety of an organization's Active Directory accounts are synced with Azure AD in the Cloud

Further, to facilitate the management and protection of these organizational user and computer accounts, security groups and policies, OUs and containers, an ocean of privileged access is delegated and provisioned inside Active Directory.

Finally, the most powerful administrative (privileged) accounts and groups, i.e. all Domain Admin equivalent accounts and groups, that possess unrestricted organization-wide access, are all stored, managed and protected in Active Directory.

In other words, worldwide, not just the Keys to the Kingdom, the keys to every door in the kingdom lie in Active Directory.

(As such, in Windows based networks, Active Directory is the account/credential database used by Kerberos, the native authentication protocol in Windows, and every domain controller is a Kerberos Key Distribution Center (KDC). Based on this fact alone, Active Directory is also the foundation of cyber security in a Windows Server based IT infrastructure.)

Thus, factually speaking, an organization's Active Directory is the single most valuable IT and corporate asset, worthy of the highest protection, because it is the heart of privileged access and the foundation of an organization's cyber security.

No Mention of Active Directory in the Mainstream Media

To date, most major cyber security breaches in the last decade, including the Sony Hack, Target Breach, JP Morgan, Snowden, OPM Beach, UN Breach, SolarWinds Breach, and now the Colonial Pipeline Hack and others, all involved Active Directory and specifically involved the compromise and misuse of an Active Directory Privileged User Account.

In fact, as I have had also stated in our blog post on the SolarWinds Breach, the perpetrators in the SolarWinds Hack only targeted Active Directory environments, and here's proof based on additional research published by FireEye - 

 

"The backdoor also determines if the system is joined to an Active Directory (AD) domain, and if so, retrieves the domain name. Execution ceases if the system is not joined to an AD domain." 

This is how important, pervasive and mission-critical Active Directory is today at thousands of organizations worldwide.

Yet, there is virtually no mention of Active Directory in any coverage of cyber security breaches in the mainstream media!

Here are 10 prominent news items on the Colonial Pipeline Hack and even if you were to read each and every single one of them in their entirety, you won't find a single mention of the word Active Directory in them -

1. Bloomberg - Hackers Breached Colonial Pipeline Using Compromised Password


2. CNN - Cyberattack forces major US fuel pipeline to shut down


3. Wall Street Journal - U.S. Pipeline Cyberattack Forces Closure


4. New York Times - Cyberattack Forces a Shutdown of a Top U.S. Pipeline


5. Forbes - The Colonial Pipeline Attack Is A Major National Security Incident


6. FBI - FBI Statement on Compromise of Colonial Pipeline Networks


7. CrowdStrike - DarkSide Pipeline Attack Shakes Up the Ransomware-as-a-Service Landscape


8. FireEye - Shining a Light on DARKSIDE Ransomware Operations 


9. KrebsOnSecurity - A Closer Look at the DarkSide Ransomware Gang


10. ZDNET - Colonial Pipeline attack - Everything you need to know

 

Clearly, there is more that the world needs to know than they are currently being told by the media and others out there.

The reason this is SO very important is that unless organizations worldwide realize that it is their foundational Active Directory deployments, and specifically privileged access in Active Directory that is at the heart of virtually all breaches, the situation is NOT going to improve, because the ultimate enabler of all breaches will still be left inadequately protected.

Privileged Access in Active Directory

Speaking of privileged access in Active Directory, there exists an ocean of privileged access in every Active Directory.

Specifically, from the CEO's domain user account to the all-powerful Domain Admins security group, and from the domain computer account of every domain-joined computer to every domain security group that is used to protect millions of IT resources company-wide, literally everything in Active Directory is an object, protected by an ACL (access control list), within which reside hundreds of Active Directory security permissions, each one of which allows or denies one of over eighty different kinds of permissions to some user, service account, group, nested group, well-known security principal etc. etc., and it is together, i.e. collectively that millions of Active Directory security permissions in the ACLs of thousands of Active Directory objects, ultimately determine exactly who has what privileged access, where and how in Active Directory.

Avenues to Gaining Privileged Access in Active Directory

Obtaining privileged access in Active Directory is the new holy grail for perpetrators, and the #1 target today, because once such access is obtained, the perpetrator can obtain access to just about everything, on-premises, and in the Cloud.

It remains a less known fact that virtually all major recent cyber security breaches of the last decade, including JP Morgan, Sony Hack, Anthem Breach, the OPM Breach, Snowden, the United Nations Breach and now the SolarWinds Breach, involved the compromise and misuse of a single Active Directory privileged user account.

Traditional Techniques

Novice and intermediate perpetrators generally employ traditional techniques such as password guessing/brute-forcing, Kerberoasting and Pass-the-Hash (PtH) in their attempts to compromise Active Directory privileged user accounts. 

Fortunately for defenders, advances in protection measures have reduced the likelihood of success with such measures.

Advanced Techniques

Professional perpetrators seem to prefer employing advanced techniques that involve escalation of privilege based on the identification and exploitation of excessive access on privileged accounts, groups, and certain objects in Active Directory. 

Here are the Top-5 advanced techniques to gain privileged access in Active Directory -

1. Use Mimikatz DCSync to replicate secrets (i.e. password hashes) from an Active Directory domain 


2. Reset the password of any existing Active Directory Privileged User account e.g. the Administrator account


3. Change the membership of any existing Active Directory Privileged Group e.g. the Domain Admins group


4. Modify the ACL (access control list) protecting the special AdminSDHolder object in Active Directory


5. If Smartcards are in use, disable use of Smartcards on an AD Privileged User's account, then reset its password

The novelty of these five advanced privilege escalation techniques is that their use only requires the perpetrator to have sufficient Active Directory Effective Permissions to be able to enact these administrative tasks in a target Active Directory.

Specifically, the use of these advanced techniques does not require perpetrators to attempt a single move that could raise suspicion or be easily detected, such as moving laterally, compromising DCs, Kerberoasting, PTH etc. All a perpetrator needs to do is avail of the already gained Authenticated User level access to correctly analyze the ocean of security permissions that exists in Active Directory and identify privilege escalation paths leading to Domain Admin accounts.

Note: The risk posed by the use of these advanced techniques is adequately described in The Paramount Brief.  

These advanced techniques are already in use today, and often rely on the use of an inaccurate but freely available tool called Bloodhound. The only tools that can make such determinations accurately are Gold Finger and Gold Finger Mini.

I cannot emphasize this enough - "The compromise of a single Domain Controller or that of a single Active Directory Privileged User Account is tantamount to a complete Active Directory Forest-wide compromise."

 

Concluding Thoughts

The sole purpose of penning this blog post was to help organizations worldwide understand that in fact what enabled the perpetrators of the Colonial Pipeline Hack to be able to easily deploy ransomware system-wide (aka domain-wide) was their ability to compromise and then misuse a single Active Directory Privileged User account.

In the case of the Colonial Pipeline Hack, its perpetrator's intentions were to unleash ransomware for monetary gain. 

Likewise, a perpetrator could easily accomplish virtually any objective of choice, whether it be data exfiltration, automated asset destruction, tampering a highly sensitive asset (e.g. software source-code, blue-prints of a highly sensitive project, such as a Nuclear Reactor,) taking over the energy grid of a city/state, compromising a government agency (e.g. an embassy or a military deployment), stealing data (e.g. financial details, customer PII etc.) from a Fortune 100 company etc., if he/she could simply compromise ONE Active Directory Privileged User account. 

I cannot emphasize this enough, so I will say it once more, for the umpteenth time - the compromise of a single DC or a single Active Directory Privileged User account is tantamount to a complete, colossal, organization-wide breach, that can not only result in substantial damage, it can cost millions of dollars and weeks to recover from.

Securing DCs is easy for we know exactly how many we have; unfortunately, the same isn't true of privileged users in AD.

In that regard, it is my professional opinion as former Microsoft Program Manager for Active Directory Security that the accurate identification and subsequent reduction in the number of individuals that possess privileged access in Active Directory is the single most important step organizations can take to protect themselves from such colossal breaches.

I will also tell you that today, while there exist over a thousand cyber security companies in the world, including numerous prominent ones such as Palo Alto Networks (PANW), Palantir Technologies (PLTR), CyberArk (CYBR), FireEye (FEYE), CrowdStrike (CRWD), Check Point Software (CHKP), ZScaler (ZS), Splunk (SPLK), CloudFlare (NET), NortonLifeLock (NLOK), Sophos Group (SOPH), SolarWinds (SWI), Tenable (TENB), Varonis (VRNS), VMWare (VMW), Cisco (CSCO), IBM, Intel (INTC), Microsoft (MSFT) etc., today not one cyber security company in the world possesses the capability to help organizations accurately* identify and lockdown privileged access in their foundational Active Directory deployments.

Well, I shouldn't say not one, because there is one. The only company in the world that can do so is Paramount Defenses ; it can empower organizations to instantly and accurately identify privileged users/access, domain-wide, at a button's touch.

Note: This is not about pride or competition. We do not do what the other thousand cyber security companies do, and they do not and cannot do what we do. This is about collaboration and helping make the world a safer place. 

In summary, today's post was about helping the world understand that if you actually take a close (detailed) look at what happened in the Colonial Pipeline Hack, you'll find that the defining step that actually enabled the perpetrators to inflict substantial damage was their ability to compromise and misuse a single i.e. just one Active Directory privileged user account - without it, they could not have been able to unleash ransomware system-wide (i.e. domain-wide.)

By the way, if you liked this post, you may very likely also like my substantially detailed post on the SolarWinds Breach. 

Lastly, as I have adequately described by now, at the heart of both these breaches lay Active Directory. 

Best wishes,

Sanjay

What's common between the Colonial Pipeline Hack and SolarWinds Breach?

Folks,

From the entire U.S. Government to the Fortune 1000, including almost all cyber security and cloud computing companies, today, at the foundation of IT, cyber security and privileged access at 85% of organizations worldwide lies Active Directory.

I'm Sanjay Tandon, formerly Program Manager for Active Directory Security on Microsoft Corporation's Windows Server Development Team, and today, CEO of Paramount Defenses.

Today, I'll share with you what is common between the Colonial Pipeline Hack and the SolarWinds Breach, and day after tomorrow onwards, I'll also provide sufficient technical details, but before I do so, I would like to share a few observations. 

 

Note - The only reason you may want to listen to what I have to say, is because, by virtue of my years at Microsoft and PD, I possess sufficient expertise, IP and capability to be able to help substantially enhance (and if requested, also demo how one could compromise) the foundational cyber security of any/every organization in the world.

Five Observations

I would like to share a few salient observations on the current(ly dismal) state of cyber security at organizations worldwide, because it is my professional opinion that until certain basic deficiencies are addressed, unfortunately, we will continue to witness many more such breaches - 

1. The Current State of Affairs

   It is really sad to see the current state of cyber security at organizations worldwide. Not a month seems to go by without there being yet another high-impact cyber security breach at some prominent organization or the other.

   That said, considering how inadequate the actual state of cyber security preparedness, defenses and proficiency are at most organizations, it is hardly surprising to see so many organizations get breached, ransomware'd etc.

   For instance, consider this - Active Directory (AD) is the very foundation of cyber security at organizations, and a Domain Controller (i.e. the machine on which AD is hosted) is technically the most valuable asset an organization has, yet, at most organizations, DCs remain vastly inadequately protected, and thus vulnerable to compromise.

   If this is the state of DC security at thousands of organizations worldwide, how can there be any security?

   Likewise, the compromise of a single Active Directory Privileged User account is tantamount to a complete Active Directory forest-wide breach, so such accounts must be minimal in number and highly protected. Yet, at most organizations, today there exist an excessively large and unknown number of Active Directory privileged accounts.

   If this is the state of AD privileged accounts at most organizations worldwide, how can there be any security? 

   
   




2. Three Fundamental Deficiencies

   It is my professional opinion that most organizations suffer from three key deficiencies, that ultimately result in inadequate cyber security defenses, leading to breaches - understanding, accountability and empowerment.     

   1. Understanding - Given a vast and dynamic attack surface, and sophisticated threats, it is imperative that all organizations possess a sufficient understanding of how to adequately protect themselves, yet most don't.

   2. Accountability - Security requires a clear chain of ownership and accountability:  Shareholders, customers, partners > CEO > CISO > Director(s) > Domain (and IT) Admins. Yet at most organizations, none exists.

   3. Empowerment - Organizational IT teams need to be adequately empowered to acquire and deploy security measures needed to adequately defend an organization, yet at most organizations, budgets are inadequate.

   
   

   For instance, IT personnel and Domain Admins from thousands of organizations have requested our help, found our unique products (e.g. 1, 2, 3) to be essential, yet so many end up conveying that they just do not have the budget.

   In reality, it is not that they do not have the budget; it is primarily that their executive management simply does not yet possess the required understanding i.e. Active Directory Security directly impacts foundational security and business continuity, and is thus paramount, and consequently their IT personnel are simply not empowered. 

   
   



3. The World is Mostly Reacting

   Sadly, at most organizations, cyber security is only taken sufficiently seriously after they have been breached, and in most instances, the response is similar - the breach is disclosed, then FireEye is called in to investigate, and ultimately, promises are made to enhance security. In the case of govts., broad directives/EOs may be issued.

   FireEye does a thorough investigation and in most cases, the findings are similar i.e. the perpetrators used the same set of well-known techniques and in almost every case, compromised and misused an Active Directory privileged user account to obtain Domain Admin level access, which was then used to achieve their objective. 

   Subsequent to FireEye's investigation, this is priority #1, budget is no longer a problem, a new CISO is hired, half a dozen new cyber security solutions are deployed, millions are spent etc. but the damage has already been done.

   
   



4. Lack of Specifics in Public Discourse

   After every breach, the CNNs and ABCs of the world will extensively cover it, you'll hear interviews from prominent Senators, Congressmen and cyber-security experts, all of whom will speak about the serious impact, the role in national security, the influence of a foreign power etc., yet not one of them mention one piece of specific detail.

   In the absence of details in the public discourse, the actual problem, and the solution that it requires, will largely remain unaddressed, and most cyber security companies out there will likely use this opportunity to convince organizations to deploy their latest cyber security solutions, whether or not they actually make a difference. 

   As a result, in all the noise, and due to the lack of focus on details, the actual specific deficiency/weakness that was exploited, and the attack vector that was used in a specific breach, will often likely continue to remain unaddressed at thousands of other organizations worldwide, paving the way for the next breach and the one after it, and so on.

   For instance, in virtually every major cyber security breach to date, the most damaging part of the breach was made possible by the perpetrator compromising and misusing a single Active Directory privileged user account to fulfill his/her objective, whether it be exfiltrating data, unleashing malware etc. and yet to date, at most organizations worldwide, no one has any idea as to exactly how many users have privileged access in Active Directory because the elephant in the room, i.e. "Active Directory", was not mentioned even once in the public discourse.

   
   



5. The Basics - Secure the Foundation and Deny them the Opportunity

   At its simplest, all security is fundamentally about access control. In order to compromise anything, perpetrators require access - if we reliably deny them the required access, we will have won half the cyber security battle.

   Most importantly, if perpetrators are unable to obtain privileged access, specifically Domain Admin equivalent access, they will almost never able to inflict colossal damage i.e. no widespread ransomware, data exfiltration, etc.

   Towards that end, the most important proactive measure organizations can take to adequately defend themselves is to adequately secure and defend their foundational Active Directory deployments, the two most important parts of which are to 1) secure all DCs (and admin workstations), and 2) accurately identify and minimize the number of accounts that possess privileged access in Active Directory, then fiercely protect every AD privileged account.

   Here's why - An attacker only needs to compromise one DC or one AD privileged user account. That's it. Just ONE.

   Real-world Evidence - If the perpetrators of the Colonial Pipeline attack had not been able to compromise a DC, they would likely not have been able to unleash ransomware. Likewise, if the perpetrators in the SolarWinds Breach had not been able to compromise an Active Directory privileged user account, they would not have been able to gain access to and exfiltrate vast amounts of data on-prem and in the Cloud, at thousands of organizations. 

• Note - If you find this to be high-level and light on technical details, it is so by intent, given its purpose. For those who may wish to judge my competency based on details - one, two, three, four, five, six, etc..
  
  I've also written an innocuous production-level ransomware example to show it could be AD deployed.

What is common between the Colonial Pipeline Hack and the Solar Winds Breach?

In the last few months, two major cyber security incidents, the SolarWinds Breach and the Colonial Pipeline Hack have had a notable impact on the world, the former having impacted the security of thousands of organizations worldwide, and the latter having caused a week long shutdown of the largest oil pipeline operator in the eastern United States.

The one thing that both these attacks had in common was that in each of these cyber security incidents, the perpetrators specifically targeted and successfully compromised the foundational Active Directory deployments of organizations.

Note - The compromise of a single Domain Controller and/or a single Active Directory privileged user account is tantamount to the compromise of an organization's entire foundational Active Directory deployment.

It can be stated with a high degree of certainty that had the perpetrators not been able to compromise the foundational Active Directory deployments of these organizations, in all likelihood, these attacks would not have been successful.

I'll share the relevant technical details of both of these attacks, on this blog, starting day after tomorrow, as stated below.  

Trillion $ Insights

Over the next few days, starting day after tomorrow, I'll share ten specific high-value details that have a direct bearing on the foundational cyber security of every organization operating on Active Directory today; you may wish to tune in.

Day after tomorrow, I'll share the details of what enabled the most impactful part of the SolarWinds Breach right here, and in days to come, I'll also share what enabled the most impactful part of the Colonial Pipeline Hack here.

Sincerely,
Sanjay.

Founder and CEO, 

Paramount Defenses

PS: I am often asked for advice on how to secure Active Directory. 

It being an ocean of a subject, here's the essence of it -

In the hierarchy of security measures, prevention is #1, avoidance is #2, detection is #3 and remediation is #4.

I. Prevention - The most effective measure is prevention; the most effective way of preventing an AD breach is as follows: 

1. Adequately secure and defend every single domain controller (and if used, privileged admin workstations (PAWs))
2. Accurately identify and minimize the number of privileged accounts in Active Directory, then protect all of them.
3. Always follow secure admin practices e.g. do NOT logon to any machine except PAWs using Domain Admin creds.

II. Detection - You may wish to consider using an AD Security Monitoring /Threat Intelligence solution to gain visibility and detect enactment of attacks. It is important to keep in mind that such solutions usually monitor replication so they provide quick but "after-the-fact" insights. In general, the efficacy of such solutions is a function of the timeliness of your response.

III. Remediation - You may wish to consider using an AD Backup and Restore solution, in the event of an incident. An AD restore is an extremely complicated and expensive operation, not to be taken lightly, and only to be used as a last resort.