Data Control Platform Guide

Preface

Intended Audience

The Data Control Guide can be used on multiple levels, but is primarily useful as a starting point to understand the core capabilities of the Data Control Platform via use-cases and examples. We have included a number of patterns here, along with sample source code to make things more concrete for developers and architects, but readers looking for a higher-level understanding should be able to read the text and skip over the code snippets fairly easily.

Data Control Platform in One Paragraph

The Data Control Platform is a suite of SDKs for managing sensitive data. These SDKs implement state-of-the-art privacy and security protocols, and they can help you meet internal and external compliance requirements. The protected data can be database fields, files, or any other storage format, and the security of and control over data can be maintained no matter where or how the data is stored. The Data Control Platform handles the hard problems of cryptographic algorithm selection and implementation as well as key management. Developers can use the platform to build zero-trust architectures, end-to-end encryption solutions, ways to “pull back” shared data, or enforcement of data policies. Depending on the needs of the application, Data Control can be integrated into endpoint applications in the browser, on mobile devices, or in server side applications.

Introduction: Trade Hoping for Knowing

What is the Data Control Platform?

Many of us have had that moment of hesitation when collecting data from a user. We think, “Are we just going to store that in the clear? What if ---?” Most developers have learned to ignore that moment of pause, and the result is untracked duplication, misuse, and leaks of stored data. When data is stored without built-in access controls enabled, there are no boundaries to where the data can be used, how it can be duplicated, or who might be able to access it.

You can maintain control over your data. The Data Control Platform can help. Data Control is a data-centric way of designing privacy and security features into your application by storing the access controls with the data.

Data-centric privacy and security by design changes the mindset of application development. By applying access controls at the data level, uncertainty is removed. Can my application access this data? It can if you can decrypt it! Can I copy the data somewhere else? Go ahead, it’s safe wherever you put it. A customer wants us to delete all his data!?! Remove access to that data and it has effectively been deleted -- even from cold backups. Our whole S3 bucket of customer data was just dumped by an unknown party! It’s okay… they can’t decrypt it.

Now you are starting to see the possibilities. This new way of thinking allows your application to send, store, retrieve, process, and delete data with much greater confidence. You don’t have to wonder where the data has migrated and who has access. Gone are the days of blindly hoping the data isn’t misused, or trying to cobble together layers of protection to prevent unauthorized access. Data Control allows applications to know with mathematical certainty who can access a piece of data by storing Cryptography Based Access Controls (CBAC) with the data itself.

Cryptography Based Access Control (CBAC)

Access control schemes, such as RBAC, aim to restrict access to a system to only authorized users. The problem with even the most sophisticated access control schemes is they can be bypassed. Think about it - if your application is checking to see who the current user is and what rights they have and is augmenting SQL queries to enforce those rights, then there’s a lot of room for error. Besides the complexity, your app itself likely has full access to all of the data via its database credentials. If the app is hacked or those credentials are leaked, the data is gone. Additionally, you still have the problem of other devs building new functionality or apps that omit the access control layers or don’t implement them correctly or curious database administrators peeking at data, or government subpoenas going to service providers, or hackers getting access to the infrastructure and therefore to the crown jewels, or… this list could go on and on. The problem with today’s access control schemes is that you can’t know if they’re actually being enforced, even with rigorous security teams, checklists, and policies.

Instead of protecting the soft, gooey nuggets of data at the heart of your application by hoping there are no weak links in your access control, your network, your applications, and your staff, what if the data itself was hardened? What if when an attacker got to the data and instead of finding a diamond, they found only a lump of coal -- capable of becoming a diamond, but only under the right conditions? CBAC allows you to know with mathematical certainty who or what can access a piece of data by baking the access control into the data itself.

Quickstart

This section of the guide is intended to help someone who wants to get started using the Data Control Platform as quickly as possible. In order to do that we'll generate the users and devices using the commandline. This is not something you would want to do in your production application, but it lets you play around and get started as quickly as possible. We can use ironoxide-cli to generate our test users and devices in just a few easy steps, which will let you start writing code without much hassle.

Sign up for an account

In order to create a device that IronCore will recognize, you'll first need to create an admin account. Signing up for an account with IronCore is easy. Visit IronCore admin console to sign up for an account. As part of the sign up process you'll create a project and segment. For now you can name them test-project and test-segment. These will be fine placeholder names for getting started.

If you already had an account and did not need to generate a new one, you'll instead just need to generate a new identity assertion key, which we'll cover in the next section.

Generate Identity Assertion Keys

Identity Assertion Keys are generated from the IronCore admin console. Select the project for which you want to generate keys and then click the "Add Identity Assertion Key" button. This will display a dialog that gives you the chance to download both the identity assertion key file as well as the configuration file. Save both these files, as you'll need them to generate users.

Generate a User and Device

In order to generate a user, you can use the ironoxide-cli utility that we provide. Binaries are provided for multiple architectures and platforms. If your architecture or platform isn't available you can also build and install the utility on your system by cloning the repo and running cargo install --path <path to ironoxide-cli folder>. To do user creation, you'll need a configuration file and the identity assertion key you downloaded above. The configuration file is assumed to be named config.json and the identity assertion key is named assertionKey.pem. If you saved them using different names, adjust the following command accordingly.

Once you have installed ironoxide-cli, you can create a user by running ironoxide-cli user-create my-user --iak assertionKey.pem --config config.json --password MyTotallySecurePasswordIDidntCopyFromTheWebsite.

You should see output like this:

Creating user "my-user"
Generating device for user "my-user"
Outputting device context to "my-user.json"

The my-user.json file is what we call a device context. This is what you need to initialize our SDKs and interact with it as my-user. Once you've initialized you'll be able to decrypt things encrypted to my-user, create groups, and perform all other actions as my-user.

Note that the device should not be shared between users (or instances of an application). They are ephemeral and should be generated as needed by your application. Using a separate device for each instance of the application allows you to revoke that device's access without shutting down all of them. It's easy to revoke access from a device.

Other patterns

Encrypt To User

Encrypt To Group

Initializing An SDK

FAQ

What do I do if I forget my password?

See Password Recovery Tokens.

Use Cases for the Data Control Platform

The following are some high-level use cases for the Data Control Platform. These are meant to spark your creativity and to help you understand what’s possible. If you have a new use case, please reach out to us. We’d love to hear from you!

Sharing Sensitive Files

Securely sharing files is very straightforward with the Data Control Platform. In fact, we have a free standalone tool available to do this on Linux, MacOS, and Windows called ironhide. The tool is an open-source, easy-to-use command line utility that lets you choose where to store the encrypted file. You can think of it as a souped-up gpg. What’s unique about ironhide vs a tool like gpg is that encrypting the data and deciding (or changing) who can decrypt the data are completely separable! We call this separation of encryption from cryptographic permissioning orthogonal access control.

Secure file sharing can also be built directly into your app using the Data Control SDKs. If your application stores file attachments, for example, maybe the files can remain opaque to backend services and be converted back to a usable format at the point of use, where an end user actually needs access. Or maybe a backend service only needs limited or one-time access to a file; for instance, maybe a file attachment is scanned for viruses once when it is uploaded to the server. With Data Control, you can implement a zero-trust end-state where your users and their collaborators can see those files, but the people who operate your systems and the attackers that penetrate your security cannot.

Shared Developer/CI Secrets

Most secret sharing systems involve some centralized store of secrets. But this can be inconvenient, particularly in a world of distributed version control systems. Using the Data Control Platform, secrets can be stored in their most natural place and shared with only those developers or systems who need access -- even if a repository is public.

Storing Encrypted Data In a Database

The Data Control Platform is independent of the backend storage. Controlled data can be stored in relational databases, key/value stores, document databases, or any storage mechanism that fits your application. Your application uses a Data Control SDK to encrypt the data and then stores the protected data.

By default, the Data Control SDKs return encrypted data as bytes. This means that the application’s data store must accept those raw bytes, or the application can use Base64 or a similar encoding scheme to turn the bytes into a string that the database will accept.

While the approach of storing opaque encrypted data suits many use cases, a data store’s search, query, sort, and indexing capabilities can not operate directly on the encrypted data. The Data Control Platform does provide encrypted search capabilities, and future versions will expand this capability.

Encrypting Data To Large Groups of Users

The Data Control Platform supports the use of Scalable Encryption Groups to protect your data, and these groups handle a large number of users in an extremely efficient manner. When you encrypt data, you decide which users and/or Scalable Encryption Groups to encrypt to, but the decision about which users can decrypt the data via the group is an independent decision. This is orthogonal access control at work.

You decide how to organize your users into groups, so that all the users that need access to a particular type of data are grouped together. There is no limit to the size of a group - maybe it’s a handful of people in a small department working on a sensitive project, or maybe it’s a group of all the company’s employees. Users can belong to multiple groups, so your decisions about group structure can be governed by your privacy and security needs, not by group size or other arbitrary limitation.

Either way, when you decide a piece of data needs to be accessed by a group, you encrypt that data to that group. This encryption operation takes the same amount of time regardless of the size of the group - a group of 100,000 users takes no longer than a group of 10 users. If you need to add people to or remove people from the group, that can be done without touching any data that was encrypted to the group. See Encrypting Data To Dynamic Groups of Users and Scalable Encryption Groups for more information.

Encrypting Data To Dynamic Groups of Users

Scalable Encryption Groups support not only large groups of users, but also groups with very dynamic membership. Your decisions about how to protect your data should be based on deciding what individuals, roles, or projects need access to that data. We allow you to create Scalable Encryption Groups to represent collections of users that have the same access requirements, and to then easily encrypt data to those groups. But the real world is a messy place, and few groups of users stay static. New employees come on board, old ones leave, roles change, and so do decisions about how much access a role requires. Once a person no longer needs access to a piece of data, their ability to decrypt the data should be revoked. If you use a traditional encryption scheme, that means that each time a change in group membership occurs, you need to re-encrypt all of that group’s data to the updated set of users, redistribute the new copy of the data to everyone that still needs access, and trust that any users who were removed delete their copies of the data. Or you could just give up and not revoke access from users that no longer need it.

With IronCore’s Data Control SDKs, you no longer need to make that concession. Because the management of group membership is independent from the encryption of data to a group (thanks to orthogonal access control!), you can change group membership whenever necessary, without touching any encrypted data. If your applications are designed to leave data encrypted when it is in storage and only decrypt it at the point of use, you can be confident that the only people that can get to the data are the users who should have access.

This use case is also applicable to give access to newly provisioned users in a self-sign-up or self-install type of environment. When a new user installs an application, a semi-automated flow can assist with getting the user into the correct groups to be able to decrypt the necessary data.

Auditing Data Access

Control of data isn’t just about determining who can access the data. You only really have control when you know who has accessed the data. Your confidence increases if that audit trail also provides the ability to detect anomalous behavior like a user who is decrypting unusual amounts of data.

Some audit trails only tell you when data has changed, but the Data Control Platform provides an unbypassable audit trail also shows when data is accessed. This is possible because a mathematical step, called a transform, must be performed by the platform for data to be decrypted on a device.

This level of audit trail helps companies meet compliance obligations. If your data is subject to regulations, you may be required to track how that data is accessed, when, and by whom, and be ready to share that information with regulators or auditors.

"Deleting" User Data By Removing Access (GDPR Right To Be Forgotten / Right To Erasure)

In a standard application architecture, if a user requests that their data be deleted, it can be difficult to locate and delete all copies of the data. By utilizing CBAC and the Data Control Platform, access to data can be revoked without locating or modifying the data using a technique called “crypto-shredding.”

Crypto-shredding is a way of deleting data by instead deleting the key that’s needed to decrypt it. If you can do this, then you can effectively delete all copies of the encrypted data even if they reside in offline backups. This can help with the right to erasure and right to be forgotten requirements of GDPR and other similar data privacy laws.

The Data Control Platform provides several ways to approach crypto-shredding that orient around revoking access to the data. This accomplishes the same thing by making sure that a company can no longer decrypt the data.

Protecting Personal Data that Is Used for Record Location

Suppose you have a front end application that communicates with a back end service you provide, and that your system deals with customer data. Some part of that data is almost certainly Personally Identifiable Information (PII), and you should definitely protect access to that data. You probably store each customer’s name, email address, mailing address, and maybe even some more sensitive information like birth date or social security number. You can use the IronCore SDKs to encrypt that data in the front end application and to only decrypt it at the point of use, in another instance of the application. If your back end service does not need access to this PII data, you can protect the data end to end and eliminate the concern that an attacker or a curious administrator might extract the information from the back end.

However, some parts of the customer data are probably needed by your application to look up a customer’s record, such as the person’s name, email address, and mailing address. This doesn’t eliminate your opportunity to encrypt the data; you can use IronCore’s encrypted search feature to index this data so that you can encrypt the data at the point of origin and store it safely, but you can still use the encrypted search features that the Data Control Platform provides to find the right records. Our concepts section has an overview of our encrypted search functionality.

Suppose your application needs to look up customers by name, email, and mailing address. You will probably get the most usability and performance by indexing each of those elements separately. Your application can create three separate indices, and each time a user enters or updates data about a customer, the application uses the IronCore SDK to also generate the index tokens to represent that customer record in each of these indices. Your application and back end service are responsible for storing the index tokens in a persistent store that will allow the back end to search for matches.

Suppose your application needs to search for a customer record given a name. It uses the IronCore SDK, specifying the customer name index and the search string entered by a user, to generate a list of index tokens for the search and retrieve all the matching records from the back end. The IronCore SDK can help your application to decrypt the sensitive data in those records and filter out records that don’t match the query (false positives).

The details on how to put all these pieces together are shown in the Encrypted Search Patterns.

Core Concepts

Basics: Users, Devices, and Scalable Encryption Groups

For the Basic Patterns to make sense it will be useful to introduce the main actors in the Data Control Platform. Users, Devices and Scalable Encryption Groups are each cryptographic entities with their own public/private key-pairs. It is essential to understand that with CBAC, the links between these entities are mathematical and not based on traditional permissioning concepts.

NOTE: The underlying technology that makes this possible is Transform Encryption. It is not necessary to understand the mathematics of Transform Encryption to utilize Users, Devices, and Scalable Encryption Groups.

CBAC Users

Users are the basic building block for CBAC. A CBAC “User” is a flexible concept that could be a user in an application, a kubernetes pod, a cloud function, or another actor that needs to perform a cryptographic operation. Each user is a unique identity that has its own public/private key-pair. Users can have many devices and can belong to many groups.

Devices

Devices are the only entity in the Data Control Platform that can decrypt data. A device is authorized using a user’s private key, and thus a device is tightly bound to a user. Since data is never encrypted directly to a device, devices can be considered ephemeral as there is no penalty for deleting a device and creating another one. Device authorizations can also be revoked, removing the ability to perform cryptographic operation, including decryption.

All SDK operations happen in the context of a particular device.

Scalable Encryption Groups

Scalable Encryption Groups, or simply Groups, are collections of users. A user in the context of a group is called a member. Groups can have any number of members. Group membership can be modified by users designated as group administrators. Adding a user to a group is a series of cryptographic operations involving the administrating user’s keys, the group’s keys, and the new member’s public key.

USERDEVICEGROUP1 User : N Devices N Groups : M Users

Now you know what Users, Devices, and Scalable Encryption Groups are, but what can they do? This is best demonstrated in the context of encryption and decryption operations.

Data can be encrypted to zero or more Users and zero or more Scalable Encryption Groups. Data can only be decrypted via an authorized Device

This “unbalanced” relationship between encryption and decryption might seem odd at first, but the power of this design is revocation. In CBAC, revocation occurs when the cryptographic link, called a Transform Key, between a User and a Device or Group and a User is severed. If you are interested in the underlying cryptographic operations, check out Transform Encryption. Each time data is transformed during the decryption process, there is the possibility for the Data Control Platform to be unable to perform the transformation if access has been revoked. This failure might occur if a User has been removed from a Group, or a Device has been revoked by User.

This is incredibly powerful, as revocation can occur without modifying the encrypted data! That is, changing the users who can decrypt data via a Scalable Encryption Group, or deleting a device attached to a user doesn’t require doing anything to encrypted data. This concept - separation between encrypting data and deciding who can decrypt it, is known as Orthogonal Access Control.

Basics: Audit Trails

Data Control Patterns

The following are common patterns we’ve helped architects and developers implement in their systems and applications. This library will continue to grow as we continue to explore the applications of Data Control and CBAC. If you have implemented a pattern not represented here, please let us know!

Adding Data Control To an App With Distinct Users

Use this pattern when:

  • You are integrating a Data Control Platform SDK into a new or existing app with distinct users
  • You are building a server-side app, but don’t want to maintain a DeviceContext secret

But don’t forget:

  • You will need to be able to generate a Data Control Platform JWT for applications with user logins
  • You can use ironoxide-cli to generate device keys for server-side apps or to just test things out

To use the Data Control Platform your app must have a Data Control DeviceContext. For most apps, devices should be generated dynamically via the CBAC User linked to a user login within your app. The process of linking your cryptographic identity (CBAC User) to your login identity is facilitated via a JWT (JSON Web Token) issued by your identity provider. This JWT is an assertion that your app has successfully authenticated the current user, and it securely provides that user’s identity in the form of a user ID. If you are using a third party identity provider (Auth0, Google, Microsoft, etc), they will have a way of issuing a JWT (sometimes called an ID token) for the currently logged in user. You will need to add some “custom claims” to the provider’s JWT.

If you are doing your own authentication, you will need to add an endpoint on your backend that can generate a JWT that asserts the identity of the current user. You should choose a library from jwt.io that supports the ES256 signing algorithm in the language of your choice.

Option 1: Data Control Platform SDK Managing the DeviceContext

Some Data Control Platform SDKs offer the option of letting the SDK manage the DeviceContext internally, freeing the app from dealing with any device secrets. In these cases, the SDK is taking advantage of platform-specific secure storage mechanisms, and the DeviceContext will be stored and retrieved as needed.

Here’s a basic flow of creating a CBAC User where the app wants to delegate DeviceContext storage to the Data Control Platform SDK.

Your AppSDKIdentity ProviderUser LoginjwtCallbackGET JWTGenerate JWT with DCP Custom ClaimsJWT for Logged In UserSDK.initialize(jwtCallback, passwdCallback)Create UserCreate DeviceStore Device in Platform's Secure StorageSDK handleYour AppSDKIdentity Provider

import {initialize, ErrorCodes, document} from "@ironcorelabs/ironweb";

/**
 * Request the endpoint on your server that we setup above. Since we setup the endpoint
 * above to return the token in plaintext, we need to parse the response as such.
 */
function getJWTForUser() {
    //The fetch() API returns a Promise which we can then chain to parse out
    //the plaintext content of the response, which will be a valid JWT.
    return fetch("/generateJWT").then((response) => response.text());
}

/**
 * Request the user's password. This function will only be called if the user
 * has not authorized this browser to access their data.
 */
function getUsersPassword(didUserExist) {
    //The `didUserExist` argument is a boolean parameter which gives you the opportunity
    //to alter the user experience for newly created/synced users.
    return new Promise((resolve) => {
        const password = window.prompt("Please provide your secure password.");
        resolve(password);
    });
}

/**
 * Initialize the IronWeb SDK with the two provided callbacks. The Promise will resolve
 * once successful at which point the document/group/user methods are available to be
 * invoked
 */
initialize(getJWTForUser, getUsersPassword)
    .then((initResult) => {
        //initResult.user.id is the initialized user
        return document.list();
    })
    .then((userDocs) => {
        //Show user a list of their documents
    })
    .catch((sdkError) => {
        if (sdkError.code === ErrorCode.USER_PASSCODE_INCORRECT) {
            alert("Wrong password provided! Please try again");
        }
    });

Option 2: Direct Management of the DeviceContext

Some platforms do not have a standard secure storage mechanism. If this is the case, or if you would rather handle the storage of the DeviceContext secret within your app, the Data Control Platform SDKs provide APIs to directly create users and devices.

A cryptographic identity (CBAC User) only needs to be linked to the logged in user once per account. Once the Data Control Platform knows about a user, the next step is to create a device that is securely associated with the user. If your app has a secure place to store the DeviceContext, device generation can be skipped on subsequent logins, but it is also possible for devices to be created as needed. Reusing a DeviceContext is preferred to reduce start-up/login time for your app.

Here’s the basic flow of creating a CBAC User with the Data Control Platform SDK where the caller wants to handle the storage of the DeviceContext. Future sessions would skip user and device creation and reuse the DeviceContext. If the user keys have already been created but a new device is needed, just the generateNewDevice portion would be needed.

Your AppSDKIdentity ProviderUser LoginGET JWTGenerate JWT with DCP Custom ClaimsJWT for Logged-in UserUser.create(jwt, password)User CreatedUser.generateNewDevice(jwt, password)DeviceContextSecurely store DeviceContextinitialize(deviceContext)SDK handleYour AppSDKIdentity Provider

use ironoxide::prelude::*;

const USER_PASSWORD: &str = "get password from user";

pub async fn init_sdk_with_config(config: &IronOxideConfig) -> Result<IronOxide, IronOxideErr> {
    IronOxide::user_create(&get_jwt(), USER_PASSWORD, &UserCreateOpts::default(), None).await?;
    let device = IronOxide::generate_new_device(
        &get_jwt(),
        USER_PASSWORD,
        &DeviceCreateOpts::default(),
        None,
    )
    .await?;
    ironoxide::initialize(&device.into(), config).await
}

#[tokio::main]
async fn main() -> Result<(), IronOxideErr> {
    let sdk = init_sdk_with_config(&IronOxideConfig::default()).await?;
    // use sdk
    Ok(())
}

fn get_jwt() -> String {
    // call out to identity provider or your back-end to get a DataControl JWT
    unimplemented!()
}

A Note About Passwords

In both flows a password is required. This password is used to protect the CBAC User’s private key. Unlike most passwords, this is not something that can be easily reset or recovered. If a lost password is a possibility, particularly anytime end-user input is required, we highly recommend using one of the Password Recovery Patterns.

Password Recovery Tokens

Password recovery tokens allow a user to securely recover from a forgotten password protecting their CBAC User’s private key.

Use this pattern when:

  • A user provided private key escrow password gates access to cryptographic data.

But don’t forget:

  • If the user forgets their password and loses their recovery token, they are locked out. Make sure this is made clear and make it easy for them to download or otherwise store the recovery token.
  • You will need to have a way of storing an encrypted_password associated with the user.
  • If the user changes their password, a new recovery token must be generated and given to the user. The user must understand that the old token has expired and the new one should be saved.

If end users are going to create and input a private key password it would be wise to have some way to handle the user losing or forgetting that password. If you don’t have an alternative and the user forgets their password, they would be unable to decrypt their data.

One way to handle this (lifted from two factor authentication flows) is a recovery token. The user is given a generated token displayed to them one time, usually when they first create their password. That token can be used as a fallback, letting them securely access their private key, which can be used to reset their password.

This token is actually a randomly generated key that has been used to encrypt a stored version of their password. The recovery process involves using this token to decrypt their password, which can then be used to log them in.

The basic flow to create a recovery token is:

  1. Start with a user’s password
  2. Generate a new crypto key (key) and random initialization vector, or IV (iv)
  3. Use key and iv to encrypt the password, producing encrypted_password
  4. Export the key as recovery_token
  5. Store the iv on the front of the encrypted_password bytes and persist it (e.g. save it to a database).
  6. Provide the user with the recovery_token

This token creation process must be repeated any time the user’s password changes.

The flow to use a recovery token is:

  1. Request the recovery_token from the user
  2. Retrieve the encrypted_password from the database
  3. Import/reconstitute the key from the recovery_token
  4. Split the iv from the front of the encrypted_password
  5. Use the iv and the key to decrypt the encrypted_password
  6. Log the user in with their decrypted password and make them change their password (creating a new token in the process)

Diagram

Your UIYour App LogicYour ServiceCapture user pwdCreate encrypted pwd & recovery tokenPersist encrypted pwdDisplay recovery token to userUser forgets pwdCapture user recovery tokenRequest encrypted pwdReturn encrypted pwdDecrypt pwd using recovery keyLog user in with decrypted pwdForce pwd change & recreate recovery tokenYour UIYour App LogicYour Service

Code Sample

const AES_ALGO = "AES-GCM";
const AES_KEYLEN = 256;
const KEY_FORMAT = "raw";
const KEY_PERM = ["encrypt", "decrypt"];
const IV_LEN = 12;

const toByteArray = (s) => new TextEncoder().encode(s);
const fromByteArray = (b) => new TextDecoder().decode(b);

const exportRecoveryToken = (key) =>
    window.crypto.subtle
    .exportKey(KEY_FORMAT, key)
    .then((rawKey) => new Uint8Array(rawKey));

const importRecoveryToken = (token) =>
    window.crypto.subtle.importKey(KEY_FORMAT, token, AES_ALGO, true, KEY_PERM);
/**
 * Encrypt a password to a random key and iv combination. The encrypted password should be
 * persisted and the combination of the key and iv will be provided to the user as a one
 * time use recovery token to unlock it in the future.
 */
const encryptPasswordWithRecoveryToken = (password) =>
    window.crypto.subtle.generateKey({name: AES_ALGO, length: AES_KEYLEN}, true, KEY_PERM)
    .then((key) => {
        const iv = window.crypto.getRandomValues(new Uint8Array(IV_LEN));
        const encryptPass = window.crypto.subtle.encrypt(
            {name: AES_ALGO, iv},
            key,
            toByteArray(password)
        );
        const exportToken = exportRecoveryToken(key);
        return Promise.all([encryptPass, exportToken, Promise.resolve(iv)])
            .then(([encryptedPassword, recoveryToken]) => ({
                recoveryToken: new Uint8Array(recoveryToken),
                encryptedPassword: new Uint8Array([
                    ...iv,
                    ...new Uint8Array(encryptedPassword)]
                ),
            }));
    });

/**
 * Decrypt a user's password using their recovery token. Once this is done you should force a
 * password change to generate and save a new recovery token.
 */
const decryptPasswordWithRecoveryToken = (recoveryToken, encryptedPassword) =>
    importRecoveryToken(recoveryToken)
        .then((key) => {
            const iv = encryptedPassword.slice(0, IV_LEN);
            const encryptedPass = encryptedPassword.slice(IV_LEN);
            return window.crypto.subtle.decrypt({name: AES_ALGO, iv}, key, encryptedPass);
        })
        .then((password) => fromByteArray(new Uint8Array(password)));

async function testRecovery() {
    const {encryptedPassword, recoveryToken} =
        await encryptPasswordWithRecoveryToken("supersecretpassword");
    console.assert(encryptedPassword !== undefined);
    console.assert(recoveryToken !== undefined);
    // We recommend serializing/deserializing with base64 for user interaction.
    console.log(`Recovery token: ${recoveryToken}\n Encrypted password: ${encryptedPassword}`);

    const password = await decryptPasswordWithRecoveryToken(recoveryToken, encryptedPassword);
    console.assert(password === "supersecretpassword", "decrypted password didn't match input");

    console.log(`Successfully encrypted then decrypted "${password}" using a recovery token.`);
}

Encrypt Directly To Users

Encrypting a piece of data directly to a user allows any Device generated by that user to decrypt the data. The IronCore SDKs allow data to be encrypted to a list of one or more users, using the provided method to encrypt a document.

Use this pattern when:

  • Sharing with a fixed, small number of users.
  • Decryption performance is much more important than encryption performance.
  • Revoking access needs to be more granular. Revoking access to a user does not affect anyone else.

But don’t forget:

  • Additional design is required to ensure data is recoverable in the event the user’s private key is lost.
  • Scaling to a large number of users is computationally expensive. The time it takes to share data scales linearly with the number of users.
  • Adding new people requires an encrypt call for each shared document. This scales poorly if the number of documents shared is large.
  • Granting access user by user requires an encrypt call for each user AND for each document. This can become very expensive with large numbers of documents
appSDKironcoreWebServiceEncrypt data to bobGet bob's public keyEncrypt dataStore encrypted AES keysReturn encrypted dataappSDKironcoreWebService

Example Code:

    let message = "This is my secret for a single user.";
    let encrypted_result = sdk
        .document_encrypt(
            message.as_bytes(),
            &DocumentEncryptOpts::with_explicit_grants(None, None, true, vec![user_id.into()]),
        )
        .await?;

Encrypting to Group

Encrypting to a Scalable Encryption Group allows any CBAC User that is a member of that Group to decrypt the data.

Use this pattern when:

  • The users that need to decrypt the data changes over time. Group membership can be changed at any time without affecting the encrypted data. Keep in mind that encrypting data to a group is completely decoupled from granting or revoking access for an individual CBAC User to decrypt data.
  • There are more than one or two users that need to decrypt the same data. Scalable Encryption Groups allow you to encrypt data once and let (up to) thousands of users decrypt the data.
  • There is a logical grouping of users that often need to have access to the same data.
  • You want to be able to revoke access but may not be able to edit all copies of the encrypted file.
  • The user you want to encrypt to doesn’t yet have keys.

But don't forget:

  • Decrypting data encrypted to a Scalable Encryption Group is slightly more expensive than decrypting data encrypted directly to a User since there’s one extra transform operation needed. Both are very fast, but this might be an issue when decrypting on power constrained devices.
  • Groups must be administered. It is strongly advised that at least three CBAC Users are designated as Group administrators to minimize the risk of all admins losing access to their private keys. Without a functioning Group Admin, a Scalable Encryption Group’s membership cannot be changed. Data can still be decrypted by current members, however.
appSDKironcoreWebServiceEncrypt data to groupGet group's public keyEncrypt dataStore encrypted AES keysReturn encrypted dataappSDKironcoreWebService

Example Code:

    let group_id = create_group(sdk).await?;
    let message = "This is my secret which a whole group should see.";
    let encrypted_result = sdk
        .document_encrypt(
            message.as_bytes(),
            &DocumentEncryptOpts::with_explicit_grants(None, None, true, vec![(&group_id).into()]),
        )
        .await?;

Encrypted Search Patterns

The following implementation patterns illustrate how you can use the IronCore SDKs in your application to do short substring searches of data that is end-to-end encrypted.

The current examples all use IronCore's IronOxide SDK and are implemented using the Rust programming language. However, the IronWeb SDK, which is intended for use in web applications written in Javascript, also supports all the encrypted search functions. The functionality is similar - you can find the details in the documentation on the search functions in IronWeb here.

Creating an Index

The first step in updating your application to support encrypted search is to create the index that will be used to protect the privacy of the index data. This index encapsulates the information necessary to generate index tokens from strings. The tokenization process extracts index terms from a string, and for each of those terms, it prepends an optional partition name that is provided by your application, then uses a secret salt value (which functions as the secret key) to generate a 32 bit integer index token. As long as that salt remains secret, an attacker cannot create a rainbow table of entries for a given partition name. Our SDKs simplify the process of generating and protecting this salt value, providing a create_blind_index method to handle the details.

There is some setup work that must be done before you can create the index. In order to use an index to process new or updated data or to search, you must be able to access the salt. We protect that salt value using our transform cryptography solution, allowing you to manage which people should have access easily. You just create a scalable encryption group that includes all the users that can enter protected data or need to search for protected records. The ID of this group must be provided when you create a new index. Because IronCore’s transform cryptography supports orthogonal access control, you don’t need to have all the users assigned to the group before you can use it to generate your search index.

The IronCore SDK's create_blind_index method takes the group ID as input, and it generates a random value for the salt, encrypts that value to the group whose ID you provided, and returns the encrypted salt. Your application is responsible for storing that encrypted salt and for providing it when initializing the blind index for future use.

ClientSDKICLBackendCreate Blind Index(groupId)Get public key(groupId)public key for groupgenerate random saltencrypt salt to groupencrypted saltsave encrypted saltOKClientSDKICLBackend

Using IronOxide

This example uses the IronOxide SDK's group_create and create_blind_index methods to set up the group that will be used to protect your blind index's salt and to create the index.

Note: the EncryptedBlindIndexSalt that is returned implements serialization using the serde package, which allows you to decide which of the serde-supported formats to use for serialized value. This example uses JSON for the at-rest representation of the encrypted salt.

This code assumes that there is an initialized instance of the IronOxide SDK available as the object sdk.

    create_group(&sdk, &salt_group_id, "PII Search").await?;
    let encrypted_salt = sdk.create_blind_index(&salt_group_id).await?;
    let encrypted_salt_str = serde_json::to_string(&encrypted_salt)?;
    save_encrypted_salt_to_app_server(encrypted_salt_str);

The group ID is created from a string by doing a call like this: rust let salt_group_id = GroupId::try_from("indexedSearchGroup")?;

and the function to create a group is

async fn create_group(
    sdk: &IronOxide,
    group_id: &GroupId,
    name: &str,
) -> Result<GroupCreateResult> {
    let opts = GroupCreateOpts::new(
        Some(group_id.to_owned()),                   // ID
        Some(GroupName::try_from(name.to_owned())?), // name
        true,                                        // add as admin
        true,                                        // add as user
        None,                                        // owner - defaults to caller
        vec![],                                      // additional admins
        vec![],                                      // additional users
        false,                                       // needs rotation
    );
    let group = sdk.group_create(&opts).await?;
    Ok(group)
}

Preparing an Index for Use

Once you have created a blind index and stored the encrypted salt, your application can start using the index to process new records, update existing records, or search for records. You first need to retrieve the EncryptedBlindIndexSalt that you initially created and serialized. Once you have retrieved the value and deserialized it, you can initialize the index for use, using the initialize_search method, which takes the encrypted salt and uses the SDK to decrypt it.

ClientBackendSDKICLfetch encrypted saltencrypted saltInitialize SDKOKInit Blind Index(encryptedSalt)transform(encrypted salt keys)transformed keysdecrypt keysdecrypt saltOKClientBackendSDKICL

Using IronOxide

This example uses the SDK's EncryptedBlindIndexSalt object and its initialize_search method to set up a blind_index object for further use in your application. It assumes that there is an initialized instance of the IronOxide SDK available as the object sdk.

    let encrypted_salt_str = get_encrypted_salt_from_app_server();
    let encrypted_salt: EncryptedBlindIndexSalt = serde_json::from_str(&encrypted_salt_str)?;
    let blind_index = encrypted_salt.initialize_search(&sdk).await?;

Indexing New Data

Assuming your application has executed the steps shown above in the pattern Preparing an Index for Use, and that the sdk and blind_index are available in your application at the point where you have a new record that has a sensitive data field, you can use code like the following to index the data using the tokenize_data method, then use the SDK to encrypt the field. Let’s assume that you have a customer struct that contains a field name, a string that contains PII, and that you want to make the customer data available to the customerService group. We’ll assume that group has already been created.

In this example, after the customer name has been encrypted, the resulting bytes are base64 encoded, and this string replaces the customer name in the struct. In order to simplify the back end service, a new field name_keys is assumed to be added to the customer record to hold the EDEKs that are needed to decrypt the encrypted name field. Once the index tokens are generated and the PII in the customer record is encrypted, the index tokens and the customer record can be sent together to the back end service for storage.

ClientSDKBackendtokenize data(blind index, PII)token setencrypt data(PII, users and groups)encrypted PII, EDEKsupdate record with encrypted PII, EDEKssave data(record, token set)save recordsave token set for recordOKClientSDKBackend

Using IronOxide

This example uses the IronOxide SDK's tokenize_data and document_encrypt_unmanaged methods to prepare a customer record to be saved. It assumes that there is an initialized instance of the IronOxide SDK available as the object sdk.

    let name_tokens = blind_index
        .tokenize_data(&customer.name, None)?
        .into_iter()
        .collect::<Vec<u32>>();
    let encrypt_opts = DocumentEncryptOpts::with_explicit_grants(
        None,                  // document ID - create unique
        None,                  // document name
        false,                 // don't encrypt to self
        vec![group_id.into()], // users and groups to which to grant access
    );
    let enc_name = sdk
        .document_encrypt_unmanaged(customer.name.as_bytes(), &encrypt_opts)
        .await?;
    // Replace name with encoded encrypted version. Also need to store EDEKs to decrypt name.
    customer.name = base64::encode(enc_name.encrypted_data());
    customer.name_keys = base64::encode(enc_name.encrypted_deks());
    save_customer(customer.clone(), &name_tokens, &[]);

Updating an Indexed Field

If you have implemented indexing of the sensitive fields in new records before you encrypt them, as described in Indexing New Data, you will probably need to handle the case where a record is being updated, and one of the sensitive fields that has been indexed is changed. On the front end, handling this follows much the same process as you used for a new record: generate the index tokens for the new value of the field, encrypt the new value, then send the updated record and the new tokens to the back end for storage. However, on the back end, you do need to perform an additional step with the index tokens; before you save the new index tokens associated with the record, you should delete the old set of index tokens for that record. Once the old tokens are deleted, save the new set of tokens the same way you would for a new record, then update the actual record data with the new values.

ClientSDKBackendtokenize data(blind index, new PII)token setencrypt data(new PII, users and groups)encrypted PII, EDEKsupdate record with encrypted PII, EDEKsupdate data(record, token set)delete old tokens for recordupdate recordsave token set for recordOKClientSDKBackend

Searching Using an Index

Once your application has started indexing the values in sensitive fields then encrypting them, you can add the capability to search on the contents of those fields. For our example, if a user of your application wants to search for a customer by name, or some part of the name, your application just extracts a set of index tokens from the search string and using the tokenize_query method, then it uses those tokens to find matching records in your back end service. Given the set of index tokens generated by a search query, any record whose set of index tokens is a superset of the search tokens (i.e. it contains every one of the tokens) is a possible match.

Once your back end has found potential matches and returned the records to the front end, your application needs to make sure that some of the matches are not false positives. To do that, it must use the IronCore SDK to decrypt the field in each record, then confirm that the field does actually match the search query. To do this, the application should apply the transliterate_string method to the decrypted field and to the search query. It should take the transliterated search query, split it into words on white space, and check the transliterated string to ensure that it contains each of the words from the query within it somewhere. For example, suppose your search query was "bei foo". The following strings would match: "北亰 football" and "bei jing egg foo yung". Other returned records whose transliterated field does not contain "bei" and "foo" as substrings are false positives.

ClientUserSDKBackendget search queryquery stringtokenize query(blind index, query)token setget matching records(token set)matching recordstransliterate string(query)transliterated stringdecrypt data(record PII)decrypted PIItransliterate string(PII)transliterated PIIfilter transliterated PII(transliterated query)record(if match)loop[ for each matching record ]search completeClientUserSDKBackend

Using IronOxide

This example uses the IronOxide SDK's tokenize_query, transliterate_string, and document_decrypt_unmanaged methods to process a user query, fetch matching records, and filter them to display the matching records for the user. It assumes that there is an initialized instance of the IronOxide SDK available as the object sdk and an instance of the BlindSearchIndex called blind_index that was initialized using the encrypted salt from the previous examples.

This function uses the sdk to take a search query, extract index tokens, retrieve customer records from the server that match those index tokens, and display the returned records that actually match the query string.

async fn display_matching_customers(
    sdk: &IronOxide,
    name_index: &BlindIndexSearch,
    query_str: &str,
) -> Result<()> {
    let query_tokens = name_index
        .tokenize_query(query_str, None)?
        .into_iter()
        .collect();
    let customer_recs = search_customers(&query_tokens);
    let trans_query = ironoxide::search::transliterate_string(&query_str);
    let name_parts: Vec<&str> = trans_query.split_whitespace().collect();
    for cust in customer_recs.iter() {
        let result = filter_customer(&sdk, &cust, &name_parts).await?;
        match result {
            Some(decrypted_name) => println!("{} {} matched query", cust.id, decrypted_name),
            None => println!("{} did not match query", cust.id),
        }
    }
    Ok(())
}

This is the function that actually checks an individual customer record, using the sdk to decrypt the name field and compare a transliterated verison of it against the transliterated query string to make sure they match.

async fn filter_customer(
    sdk: &IronOxide,
    cust: &Customer,
    name_parts: &Vec<&str>,
) -> Result<Option<String>> {
    let cust_enc_name = base64::decode(&cust.name)?;
    let cust_name_keys = base64::decode(&cust.name_keys)?;
    let dec_result = sdk
        .document_decrypt_unmanaged(&cust_enc_name, &cust_name_keys)
        .await?;
    let dec_name = std::str::from_utf8(&dec_result.decrypted_data()).unwrap();
    let dec_name_trans = ironoxide::search::transliterate_string(&dec_name);
    if name_parts
        .iter()
        .all(|name_part| dec_name_trans.contains(name_part))
    {
        Ok(Some(dec_name.to_string()))
    } else {
        Ok(None)
    }
}

Displaying a list of customers that match a query just uses those functions, like this.

    let query_str = get_search_query();
    display_matching_customers(&sdk, &blind_index, &query_str).await?;

Using Multiple Indices

So far, our encrypted search patterns have considered a customer record that had a single field of PII, the customer name. Suppose you decide you need to protect a second field - the customer's email address, for example. You could use the same blind index to index and search both strings, but search performance will be better if you create a second blind index for the email address. You can just invoke the create_blind_index function twice, once to create an index for the names, and again to create an index for the emails. Since the indices use different salt values, knowledge about any of the contents of the set of index tokens created by one blind index won't provide any insight into the contents of the data protected by the second blind index.

Once you have created the two blind indices, you can use them to index data in your customer record before you encrypt it. Use one blind index to create the tokens for the name field and the other to create the tokens for the email field. We recommend that you maintain these two sets of index tokens separately - you could store them in the back end persistent store in two separate tables or key-value stores, or in the same store with a type discriminator.

You have some options for the actual encryption. You can encrypt each of the fields separately, producing separate encrypted data and EDEKs, or you could put the values into a single structure that you can serialize to a byte stream and encrypt as a single element. This could be a JSON object or a structure that is serialized using protobuf. In either case, you will need to store the encrypted data and the EDEKs, plus the two sets of index tokens.

One consideration of maintaining two separate indices is that you will need to understand the context when a user enters a search query - if the user enters a string that is a potential name, you should generate the index tokens using the name index and search for matches to those tokens in the store of name tokens. Likewise, if the user enters a string that is a potential email address, you would use the email index and search the store of email tokens. If the context of the search query is difficult to determine in your app, you do have the option of generating the index tokens for the query string using each of the blind indices, then searching both of the token stores for matches. This will likely generate additional false positives, but your client-side filtering should be able to handle that.

We do recommend that you use a single scalable encryption group to protect the salt for each of the blind indices. This will simplify the administration of the groups necessary to allow access to everyone who can generate or search data.

Using IronOxide

This example is similar to the one in the pattern Creating an Index, using the IronOxide SDK's create_blind_index method to create two different indices. It assumes that there is an initialized instance of the IronOxide SDK available as the object sdk.

Creating the second index is straightforward. We assume we still have several objects that were created in the previous patterns.

    let encrypted_salt2 = sdk.create_blind_index(&salt_group_id).await?;
    let blind_index2 = encrypted_salt2.initialize_search(&sdk).await?;

Now suppose you have an instance of the IronOxide SDK available as the object sdk, and the two indices in name_index and email_index. It is easy to use the two blind indices to generate index tokens for the fields before encrypting. Like the pattern for Indexing New Data, this code uses the SDK's tokenize_data and document_encrypt_unmanaged methods to prepare a customer record to be saved.

    // Generate the index tokens for the customer name and email address, then encrypt them
    let name_tokens = blind_index
        .tokenize_data(&customer2.name, None)?
        .into_iter()
        .collect::<Vec<u32>>();
    let email_tokens = blind_index2
        .tokenize_data(&customer2.email, None)?
        .into_iter()
        .collect::<Vec<u32>>();
    let enc_name = sdk
        .document_encrypt_unmanaged(&customer2.name.as_bytes(), &encrypt_opts)
        .await?;
    let enc_email = sdk
        .document_encrypt_unmanaged(&customer2.email.as_bytes(), &encrypt_opts)
        .await?;
    // Replace name and email with encoded encrypted versions. Also need to store EDEKs to decrypt both.
    customer2.name = base64::encode(enc_name.encrypted_data());
    customer2.name_keys = base64::encode(enc_name.encrypted_deks());
    customer2.email = base64::encode(enc_email.encrypted_data());
    customer2.email_keys = base64::encode(enc_email.encrypted_deks());
    save_customer(customer, &name_tokens, &email_tokens);

Support in other languages

Remember, although the code samples in these patterns were all written in Rust using IronCore's IronOxide, the search functionality is available in our other SDKs as well. In particular, Javascript-based web applications that use the IronWeb SDK can access all the encrypted search functions. The details in the documentation on the search functions in IronWeb are available here.

Features

We Are For

Trust Center

Contact Us

Follow Us