Audit Your S3 Storage Immediately

UPDATES: more S3 leaks added to the bottom of the article.

In the past month, we’ve seen breach after breach hit huge organizations not because some clever hacker used a zero-day exploit, but because data was stored in Amazon’s S3 data storage and the permissions allowed anyone to read the data.

Some of these organizations worked with consultants or third-party vendors who misconfigured the data protections, but that doesn’t absolve them of responsibility. In fact, you can’t delegate responsibility (just authority), and so a breach of an organization’s data, even via a partner, is still ultimately the fault of the organization. Post breach lawsuits prove this time and again.

Security by Obscurity

The URLs that point to S3 buckets have a lot of crazy characters in them that frequently aren’t linked publicly. So it’s easy for someone to post something publicly, but expect that no one will ever find it. Unfortunately, this is almost never true. Obscurity can work in the short term, but pretty much never works in the long term. Hoping that no one discovers the magic URL that leads to all of your customer data is a very bad idea.

In the case of S3, there are numerous public tools, such as DigiNinja’s Bucket Finder, that scan Amazon for public S3 buckets with hidden data.

Here are some organizations who learned this lesson the hard way in the past couple of months:

• Republican National Committee—A vendor put the personal information and GOP analysis of more than 198 million voters on AWS S3 in a bucket shared with the public.
• Booz Allen Hamilton — The folks responsible for some of the biggest Government leaks in recent history, including the Snowden leak, got hit again when a trove of top secret data was found unprotected on S3. Over 60,000 files were publicly accessible.
• World Wrestling Entertainment (WWE) — Personal data on 3 million fans was stored unsecured on S3. Information included home addresses, educational background, earnings, and ethnicity.
• Verizon—The personal data of 6 million customers’ was posted by a partner on S3 and made public. The data included customer names, mobile numbers, account PINs, home addresses, emails, and account balances.
• Dow Jones — Data on between 2.2 million and 4 million customers (the correct number is a subject of dispute) was exposed to any logged-in AWS user. There are millions of such users. Data included names, addresses, and partial credit card numbers. Also exposed were databases of sensitive compliance information from partner financial institutions.

Amazon Warns

Amazon sent out a notice to S3 users with buckets of files that are publicly readable. That warning reads in part:

We’re writing to remind you that one or more of your Amazon S3 bucket access control lists (ACLs) are currently configured to allow access from any user on the internet.

If you maintain an Amazon S3 bucket, whether you received the notice or not, it behooves you to check the permissions on your buckets and the objects in those buckets as well. Instructions are below.

Auditing Your Buckets

It isn’t too hard to double check your bucket permissions using Amazon’s command line interface tool. Check out their site to figure out how best to install it, but on a Mac, brew install aws works nicely.

Next, configure the tool with your AWS S3 credentials by calling aws configure.

Finally, iterate over each bucket and print out the permissions using this little bash script you can copy into a terminal:

bash

IFS=$(echo -en "\\n\\b")  
for bucket in \`**aws s3 ls** | awk '{ print $3 }'\`   
  do echo "Bucket $bucket:"   
  **aws s3api get-bucket-acl** --bucket "$bucket"   
done

You should see output something like this:

JSON

Bucket blah-blah-blah:  
{  
    "Owner": {  
        "DisplayName": "someuser",  
        "ID": "8942fa787d23f..."  
    },  
    "Grants": \[  
        {  
            "Grantee": {  
                "Type": "CanonicalUser",  
                "DisplayName": "someuser",  
                "ID": "8942fa787d23f..."  
            },  
            "Permission": "FULL\_CONTROL"  
        }  
    \]  
}

Watch closely for grants to “AuthenticatedUsers” and “AllUsers” in the output and double check to be sure that you intend that access for those buckets. If you do, you should also double check what data is actually in those buckets to make sure sensitive info isn’t mixed in with the public info.

If you have a lot of buckets, use grep to find any shared with one of those groups.

Finally, if there’s any concern that you may have an object with public permissions inside a bucket that is otherwise private, then you’ll want to check the permissions on those object more comprehensively. If you have a lot of objects, this could take awhile.

Below is a quick and dirty bash hack to do this for you. It will print out the key for any object with permissions granted to the public or all authenticated users. You’ll need to specify a bucket ID before pasting into a terminal window. If you see any keys printed in the results between the progress dots, you’ll want to check those objects out carefully.

bash

BUCKET="\[YOUR BUCKET ID\]"

IFS=$(echo -en "\\n\\b")  
for key in \`**aws s3api list-objects** --bucket "$BUCKET" |grep '"Key"' | sed -e 's/^.\*"Key": "//' -e 's/",\\? \*$//'\`  
    do echo -n .  
    **aws s3api get-object-acl** --bucket "$BUCKET" --key "$key" | egrep '(AuthenticatedUsers|AllUsers)' && echo "Public: $key"  
done

Note: any interaction with AWS can trigger usage charges, so beware especially if you have a large number of files.

What About Encryption?

The fact that all these folks misconfigured AWS permissions and put sensitive data in public places is bad enough, but for sensitive customer data and even top secret data, simple permissions should not be considered “good enough.”

Sensitive data should always be encrypted so that even if a server is compromised or a rogue Amazon employee gains access, the data is still unreadable except to authorized users. This seems like basic common sense, but unfortunately, few organizations encrypt their data in this way. None of the recently embarrassed organizations did.

If cryptographically backed access controls (regardless of where data is stored) are something that could aid your company, please reach out to IronCore Labs to see if we can help.

UPDATE 8/17/17: ES&S, a voting machine supplier, left sensitive data on 1.8 million Americans in an unsecured AWS S3 bucket.

UPDATE 8/31/17: Groupize, a provider of backend software for the hospitality industry, left credit card details, contracts, internal financials and more exposed in an unsecured AWS S3 bucket.

UPDATE 9/1/17: Charter Communications exposed 600 gigabytes of data containing 4 million records of Time Warner Cable customers’ sensitive data via yet another public S3 bucket.

UPDATE 9/5/17: Tigerswan, a military contractor, and former partner TalentPen, leaked over 9,000 resumes of law enforcement, military, and government personnel via a public S3 bucket.

UPDATE 9/20/17: Viacom, the owner of Parmaount Pictures, Comedy Central, MTV and more media properties, left its data exposed on an unprotected S3 bucket. Among the things found in the data were the keys to the Viacom kingdom including their master AWS keys, GPG keys, and server credentials. A hacker could have used these to steal enormous amounts of data from Viacom. Viacom says there was “no material impact” from the exposed data.

UPDATE 9/22/17: Verizon Wireless posted confidential documents and internal user credentials to an unsecured S3 bucket.

UPDATE 9/22/17: SVR Tracking, an auto tracking company, leaked hundreds of thousands of records with customer data in an unsecured S3 bucket.

UPDATE 11/21/17: The Pentagon stored US CENTCOM and PACCOM intelligence data scraped from open Internet resources.