The Terrifying Takeaways from the Massive OAuth Breach

What Google, Salesforce, and the rest keep missing

In the last month, the Salesforce instances of lots of major companies have been hacked. The first that I heard about was Google. On August 5th, they announced their Salesforce instance was compromised and customer data exfiltrated. The root cause was a successful social engineering campaign that also hit Cisco, Workday, Farmers Insurance, Adidas, Allianz Life, Qantas, Louis Vuitton, Dior, Tiffany & Co., and others.

A short time later, they detected successful attacks against Google Workspace accounts that integrated with Salesloft Drift, and they determined that the OAuth tokens held by Salesloft had been stolen.

What is an OAuth token?

Let’s pause right there. What’s an OAuth token? When a user wants a service to act on their behalf, they need that service to be able to log in as them. But instead of giving that service their actual password, they generate a one-off password with credentials embedded into a string that can be shared. Then the trusted service can use that string to log in as the user and perform operations on their behalf.

Here’s the problem: a user’s login may have two-factor authentication or may use strong authentication like a passkey. More security mature applications might look to signals like where the user is logging in from versus normal patterns to block access when something looks off. But OAuth tokens bypass all of that. Anyone with access to the token can effectively log in as the user. It’s similar in that way to a browser’s session cookie, which, if stolen, gives an attacker a way to bypass all the usual authentication controls.

Or for an analogy, think of the king who uses his royal ring to impress a wax seal on all his letters. If he gives that ring to another person, that person is then empowered to act on his behalf. It’s a token that proves the trust and that allows them to seal envelopes or enter premises as if they were the king himself. OAuth tokens are the royal ring, but there can be zillions of them and they can be copied.

The danger of OAuth tokens

Here’s the other problem: few companies protect their OAuth tokens. This isn’t our first time talking about this. I wrote a blog post in 2017 on the Proliferation of Under-protected OAuth Tokens where we pointed out security companies getting hacked, coughing up poorly protected tokens and causing their customers to get hacked, too.

It would be far better if we had a mechanism that allowed delegation of time-limited passkeys where the calling server could make use of it, but others couldn’t, and where hardware-security modules could keep secrets from ever being revealed or exfiltrated. But, alas, what we have instead are these powerful authentication bypass tools that we generate and share unknowingly and often.

For example, if you use Zoom for meetings and you connect it to your work email and calendar so you can book meetings from within the Zoom app, then the process of connecting Zoom and, let’s say, Gmail, from the user standpoint just requires a login and clicking a button that allows the integration. But under the hood, Zoom just acquired an OAuth token giving it direct access to your Gmail account. Hopefully they do a good job protecting this data (note: Zoom has a history of lying about their security), but we have seen over and over again that many companies do not.

Salesloft OAuth token theft leads to hundreds of compromised companies (a timeline)

Which brings us back to Salesloft. They produce a suite of AI-powered marketing and sales tools that integrate into various systems like Salesforce, but also including Snowflake, Gmail, and others. Their Drift product de-anonymizes website visitors, scores them, automatically tries to engage with them, and so on. Each provided integration requires OAuth tokens and Salesloft holds a lot of them. Here’s a timeline of events:

August 7 – Salesloft announces a merger with Clari; the Clari CEO to become CEO of the joint company (noted because the timing is interesting and it begs the question: will the merger complete now?)
August 8 – Hackers infiltrate Salesloft and gain access to data including OAuth tokens
August 20 – Salesloft announces a “security issue in the Drift application” – no details on how the hack happened or what was compromised are shared, but they said, “out of an abundance of caution, we have proactively revoked connections between Drift and Salesforce.”
August 26 – Google’s Threat Intelligence Group (GTIG) announces they’ve detected “widespread data theft” due to the Salesloft Drift breach
August 26 – Salesforce posts an advisory warning of unusual activity from a “Third Party Connected App.” They went on to say, “Salesforce security teams detected unusual activity that may have resulted in unauthorized access to a small number of customers’ orgs data” and they said that all relevant Salesloft tokens were invalidated. They emphasized that this wasn’t a Salesforce problem: “this issue did not stem from a vulnerability within the core Salesforce platform.”
August 28 – Salesforce disabled the connection to the Drift app and removed it from the marketplace without explaining why invalidating tokens was insufficient
August 28 – Google disabled the integration between the Drift app and Google Workspace accounts
September 2 – Cloudflare, Palo Alto Networks, ZScaler, and proofpoint announced they were breached via stolen OAuth credentials taken from Salesloft; these are among the hundreds of organizations GTIC says were compromised
- Cloudflare was breached Aug 9-17; they received notifications from Salesforce and Salesloft on Aug 23
September 3 – Another major security company, Tenable, announced they had also been compromised.
September 4 – Elastic belatedly discovers and discloses their own compromise, but in their case, an email inbox was browsed by the hackers, and this inbox was presumably a support inbox since it contained some instances of customer credentials.

Okta also had their OAuth tokens stolen, but they had IP allowlists that limited where the tokens could be used and they say that prevented a breach.

Company reactions

Note: we’re calling out security companies that were hit in particular here because it’s always interesting to see how security companies respond. And for some of them, the response was pretty disappointing.

⭐️⭐️⭐️⭐️⭐️ Google and Cloudflare led the way with great investigations and detailed information, advice, and in Cloudflare’s case, even an apology taking responsibility for their choices in what tools they use and the consequences if those tools are compromised. 5/5 stars for these two companies.

⭐️⭐️⭐️⭐️ Okta wasn’t compromised but did give a thoughtful response on how to constrain the power of OAuth tokens. Zscaler did a good job of laying out the important stuff including the affected information, the response, and what customers can do. It’s not the best response, but at least they put together a response befitting of a security company.

⭐️⭐️⭐️½ Salesloft gets 3.5 stars for their response. It should have been much quicker with stronger action taken earlier, but at least they posted regular updates and gave guidance to customers including indicators of compromise, IPs to search for, etc. They also (belatedly) hired outside investigators to come in and help them. What they didn’t do is explain how they were compromised or what else might have leaked. But pro-active communication in an event like this with detailed recommendations is important enough to highlight what they’ve done right even as they reel from an attack that will likely cost them a lot of business.

⭐️⭐️ Elastic had a pretty terse statement, but was at least specific about what was accessed and who was contacted as a result.

⭐️½ Tenable, Palo Alto Networks, and proofpoint did the absolute bare minimum in their disclosures with terse statements pointing out they weren’t the only ones hacked, it wasn’t their fault, and it’s no big deal. They don’t say what data of their customers was accessed, what deeper steps they’re taking (beyond turning off the hacked app), etc.

⭐️ Even more disappointing, if possible, is the Salesforce response. The OAuth tokens led to their platform and the stolen information of these companies came out of their platform. Yet their public posting just says that they turned off the Salesloft integration and the whole thing isn’t their fault. There’s no guidance on what security teams should look for (indications of compromise), how to find it, ways to audit other OAuth tokens, or anything of any use to their customers. It basically just reads as, “hey, this isn’t our fault; we turned off the connection to the bad guys; the end.” They’re not a security company, but they’re big enough to know how to provide help and guidance to their customers in a difficult time. But they didn’t.

The two big, important takeaways

The takeaway should not be anything to do with Salesloft. It shouldn’t even be about how to gracefully and competently respond to a breach event. Nor should the takeaway be to add IP allowlists for specific OAuth tokens as Okta claims (because the hackers could have initiated all of the theft via Salesloft while they were in there and then Okta’s protections would not have worked). That can’t hurt, but that isn’t the important thing to remember. The two big takeaways here should be:

OAuth tokens are incredibly dangerous things that have proliferated across the SaaS ecosystems via innocuous-seeming integrations.
OAuth tokens should be protected at least as well as cryptographic keys or raw passwords; which is to say, they should never be readable by hackers who gain access to the underlying data store. They should not even be accessible to internal admins.

Pushing SaaS vendors to protect secrets

Here’s what I didn’t see any impacted company say: “we’re going to go to all of our vendors, find out what OAuth tokens they hold, and find out how they’re protecting them.” Not even Cloudflare said that.

What the hell? Are we all just doomed to keep repeating the same mistakes?

Cloudflare, Google, Palo Alto – everyone affected should not just be looking at Salesloft right now. That’s narrow, short-term thinking. They should be reviewing their criteria and guidelines for their SaaS vendors and adding criteria to require the use of application-layer encryption, preferably without any path for humans to decrypt those tokens. The roots of trust for protecting them should be in Hardware Security Modules (HSMs) and the tokens should not be exportable in plaintext at all. With that criteria set, these companies should go to every vendor, find out what OAuth tokens they hold, determine what level of risk that poses if they should be compromised, and then force their vendors to stop treating OAuth tokens like any other piece of data in their infrastructure.

We’ve been pointing out these problems and watching them bite people for years. Nothing talks like money and making renewals or sales conditional on good security hygiene – and by that, I don’t just mean SOC-2, but I mean a detailed identification of sensitive data and a plan to keep that data safe even in the face of a breach or a compromised insider.

Security breach fatigue has led to robotic handling; strategic responses are missing

When even security companies shrug when they get hacked and phone in their breach disclosures as simple CYA notices, you know we’re in a bad place. Astoundingly bad.

Breaches are almost common-place now and the responses have become robotic. There’s a tactical plan with a series of steps that are followed by a breached company. The playbook is shared around and makes responding to a breach a mostly brainless exercise.

The hard work, the strategic thinking about what preventative measures could have changed the outcome, that’s not happening. When did we decide our current security is good enough? That breaches are simply something for the lawyers to manage? This is wrong. We are not doomed to a groundhog day of breaches and CYA notices. We can make things better. We can think strategically about what layers can be implemented to reduce damage. We can push best practices forward and force our vendors to invest in better security. And if we can’t all do that, then at least the big companies with the large contracts can and should.

Because the rush to adoption of AI tools ahead of security for AI is just going to make things worse. And all of the buyers of Salesloft are in the AI adoption herd, though as far as we know it wasn’t AI that caused the tokens to leak. Or hell, maybe it was; Salesloft hasn’t said much other than they started with a compromise of a GitHub account.