A Checklist to Quickly Evaluate SaaS Security

Large companies have security teams that scrutinize every partner and vendor they use. They put the vendor through the wringer with security reviews that usually involve long and tedious spreadsheets.

The result of those painful security reviews is generally an analysis on the relative level of risk that comes with the solution. Then the business stakeholder can choose to take the advice of security or overrule it (spending political capital in the process) to say that the benefits outweigh the risks.

The review process is painful for everyone and, frankly, despite the intentions of various startups and others, it seems to be getting worse, not better.

And we have another problem: only the biggest companies even get to evaluate the security of their vendors. The rest of us aren’t privvy to the information needed to do a full risk analysis and we may not be sophisticated enough to do it even if the vendor is willing.

Security Evaluation for the Rest of Us

It’s ludicrous to think that security only matters to large enterprises. Yet that’s how the industry operates. So the rest of us are left to scrape together the breadcrumbs of publicly available information as part of our purchasing process.

My Credentials

I’ve been on both ends of security evaluations, I’ve run engineering teams building large, complex enterprise cloud software, and I’m the CEO of a company that helps SaaS providers harden their apps. I’ve seen a lot of the skeletons hiding in the closets of software vendors and I’ve developed a nose for approximating a risk analysis based on publicly available information.

These are the things I look for and the steps I take when evaluating cloud software if the vendor might hold any data that I feel is private.

Seeing Through the B.S.

The hardest part of evaluating software and services for security maturity is the sheer amount of bullshit that gets spewed. Companies frequently have polished web pages that say things like, “we take your security seriously.” They then go on to talk up minor things as if they’re extremely important. For example, if someone tells you they’re using “military grade encryption”, run away. Seriously. It’s a big “B.S.” signal.

Step 1: Find the security page

*Note: for now, we’re only evaluating their basic security posture and how well they’re able to keep confidential data safe. We are not looking at integrity or availability.*

Most companies have a page (or sometimes a downloadable PDF) that gives a high-level marketing-oriented view of their security. We’re going to use that page to evaluate them, but the first trick is to find it. And that’s not always as easy as you might hope. Here’s what I do, in order:

1. Go to the website and look for a “Security” or “Trust Center” link in the footer. If you don’t find it there, it’s already a bit of a bad sign, but don’t give up yet.
2. I do a Duckduckgo (or Google or whatever) search that looks something like site:company-x.com security to see what comes up. I’m just looking at page titles here.
3. If that fails, it’s sometimes because this sort of information gets pushed into “resources” sections of a website as a download. That’s the third place I check.
4. Finally, if all that fails, I go to the support pages. Sometimes a company’s knowledge base isn’t indexed for search by the major search engines or is on a different domain. Find the knowledge base and search it for “security” and often you’ll find details about what the company does to be a trustworthy partner.

At this point, if you still haven’t found a page discussing how a company is going to protect your data, they get an F and you should disqualify them with extreme prejudice.

Step 2: Check the boxes

Next you have to wade through statements like, “we care about your security” and “we use state-of-the-art encryption for all data, whether at rest or in transit” and “plans include SSL encryption to keep your data safe” to find where they make specific security claims. For example, “we care about your security” is not a specific claim, but saying they use SSL or encrypt “at rest or in transit” are statements that you can map back to the checklist.

We’ve made a spreadsheet that has both the checklist and some of the phrases you’re looking for to figure out if each row should be checked or not. You can freely copy and modify the spreadsheet to suit your needs. It is licensed under the Creative Commons Attribution 4.0 International License, which just means we’d like you to give us credit if you redistribute it with or without modifications.

There are currently 23 checkboxes. This may evolve over time, but we’d like to keep the number low rather than comprehensive so it remains a lightweight way to evaluate a company’s seriousness about security.

Step 3: Score it

The spreadsheet will do this for you. For the moment, we score one point for basic checkboxes, three points for intermediate, and five points for advanced. We put a maturity level at the bottom where anything under 15 is considered “weak,” with “reasonable” and “exceptional” levels above that.

Example Scores

As of the time of writing, here’s what we gleaned from a few well-known services. Note: the specifics with notes are in tabs in the spreadsheet. We welcome corrections if we missed something.

Now you try. And let us know what you think.

👉 Free SaaS Security Evaluation Spreadsheet