Changelog

See our buildlog for relationships between specific docker hashes and version tags. The most recent releases are at the bottom of the file. There will only ever be multiple hashes for a single version tag if the underlying image was rebuilt to fix a security vulnerability.

v6.0.0

Breaking Changes

This release replaces the transport layer between the Tenant Security Proxy and Tenant Security Logdriver, adding batching and (optional) encryption. This release doesn’t break the TSC/Alloy -> TSP contract, it does break the TSP -> TSL contract. Zero downtime migration is possible by deploying new TSPs and TSLs together next to existing ones before tearing down the old ones.

If you don’t use Logdriver, no changes are required.

Before you deploy

You need to generate a certificate and key, shared by every Logdriver in a pool. This is self-signed and needs no SAN or CA.

bash

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout tsl.key -out tsl.crt -subj "/CN=logdriver" \
  -addext "basicConstraints=critical,CA:FALSE"
openssl x509 -in tsl.crt -outform DER | openssl dgst -sha256 -binary | base64

The printed digest is what the TSP will be configured with, the cert and key are for TSL configuration.

Configuration Changes

TSP:

TSL:

Removed: LOGDRIVER_EVENT_PRODUCER_URL, TSP_EVENT_BIND_ADDRESS, TSP_SEND_HIGH_WATER_MARK, TSP_ENABLE_LOGDRIVER_INTEGRATION. Logdriver ignores the first silently. TSP logs the other three at startup with what replaces each, and treats no TSP_LOGDRIVER_URL with one of the old TSL variables as fatal on the assumption you want events but haven’t correctly configured the changes.

Sizing Pools

See sizing the pool.

Deploying

Old and new pairs coexist. An old TSP keeps delivering to its old Logdriver over ZeroMQ, and a new TSP delivers to a new Logdriver over HTTPS; nothing crosses.

1. Stand up the new Logdrivers with the certificate, key and token. Each needs its own volume for the durable store, exactly as before. Wait for /ready.
2. Deploy the new TSPs pointed at the new Logdriver address. They can be deployed at the same time, but a TSP that starts first will queue events and retry until its Logdriver answers.
3. Verify with the counters below before going further.
4. Retire the old TSPs, then the old Logdrivers. Give each old Logdriver time to drain its durable store to the sinks after its producers stop.

Grace period. A TSP on this release drains its queued events to Logdriver on SIGTERM, for up to 15 seconds, and logs whether that drain finished. Allow at least that much: Kubernetes’ default terminationGracePeriodSeconds of 30 is enough. A shorter grace period means SIGKILL arrives mid-drain, discarding whatever is still queued with nothing logged, since the process is gone before it can report.

Verifying

The two services now count the same events from both ends, so whether everything TSP produced reached Logdriver’s durable store is a question with an answer. TSP serves its counters at /metrics on TSP_HTTP_BIND_ADDRESS (default port 7777), Logdriver at /metrics on its health port, 9001.

tsp_security_events_produced_total            # events TSP generated
tsp_security_events_delivered_total           # events Logdriver acknowledged
tsp_security_events_undeliverable_total       # events Logdriver rejected as malformed
tsp_real_time_security_event_failures_total   # events TSP could not queue or encode
tsp_security_event_queue_depth                # events waiting to be delivered
logdriver_events_received_total               # events Logdriver decoded
logdriver_events_committed_total              # events Logdriver made durable
logdriver_events_dropped_total                # events lost inside Logdriver, by stage

Two sums close on a healthy pair, allowing for events in flight at the moment you scrape:

• produced = delivered + undeliverable + failures + queue_depth
• received = committed + dropped

Across the pair, delivered and received should track each other. received running ahead is duplicates: a producer that resends a batch after an uncertain outcome has those events counted once by TSP and twice by Logdriver.

Read against those:

The two to alert on during the migration are tsp_security_events_undeliverable_total and logdriver_batches_undecodable_total. Both mean a producer and a consumer disagree about the format, and events counted by either are gone rather than retried.

Do not alert on logdriver_events_dropped_total directly. Its stage label means no series exists until something is dropped, so on a healthy Logdriver it is absent rather than zero and the alert silently never fires. Alert on logdriver_events_received_total - logdriver_events_committed_total instead.

Rolling back

Redeploy the previous images and the previous configuration, on both sides together. There is no single variable that reverts one service, and a new TSP cannot talk to an old Logdriver.

Events already committed to a Logdriver’s durable store are delivered by that Logdriver. Roll back the consumer before it has drained and those events stay in its store until it runs again with a compatible configuration.

v5.3.0

• Optimize linking.
• Dependency updates.

v5.2.3

• Add 30 second timeout to all http calls from the TSP. This timeout already existed for calls to decrypt configurations, but now it also applies to calls to the KMS as well as to the Configuration Broker.
• Add logging to indicate the randomly assigned TSP instance id. This log message is at info level and will appear on startup. This can help when reporting issues to IronCore.
• Dependency updates.

v5.2.2

• Switch to using MiMalloc instead of SnMalloc internally.
• Allow config refresh loops to occur up to 20% faster. This is randomly determined each loop and should allow for large fleets of TSPs to better distribute load on the IronCore Configuration Broker.
• Dependency updates.

v5.2.1

• Treat network-level KMS config decrypt errors as transient.
• Defer tenant primary config updates so other TSP instances have time to learn about new configs before they’re used for wraps.
• Update tenant secrets cache correctly to save on calls to the config broker.
• Dependency updates.

v5.2.0

• Add endpoint for clients to report cached encryption/decryption operations.

v5.1.3

• Dependency updates.
• Log TSP version on startup to assist with debugging.

v5.1.2

• Added DEBUG log messages for all AWS errors.
• Unknown AWS errors now return the underlying error message to the client.
• Fixed a bug preventing AWS leased keys from being revoked when their associated IAM users lose access to them.

v5.1.1

• Dependency updates
• Added log messages for AWS unknown errors. These are logged at level debug.

v5.1.0

• Dependency updates
• Switch to the official Azure client

v5.0.4

• Dependency updates
• Fixed a potential issue where a newly assigned kms config with leasing enabled could wrap keys or secrets with unleased keys during the first refresh of the TSP.

v5.0.3

• Dependency updates

v5.0.2

• Fixed Prometheus compatibility issues introduced by v5.0.0
• Add two new metrics: tsp_secret_refresh_failures_total and tsp_last_secret_refresh_timestamp_seconds. See here for more information.

v5.0.1

• Fixes a deadlock issue that non-deterministically occurred when many tenant assignments were frequently updated.

v4.17.1

• Fixes a deadlock issue that non-deterministically occurred when many tenant assignments were frequently updated.

  Note: v5+ of the TSP has been released and is an easy upgrade, a metric was removed. Please upgrade if possible.

v5.0.0

Breaking changes:

• Remove the tsp_requests_in_flight metric. This metric was not functioning correctly for requests involving leased keys and was often misleading.
  • This is a breaking change specifically because we previously recommended configuring a HorizontalPodAutoscaler based on this metric. Ensure you remove this before upgrading.

Other changes:

• Switch to the official Rust Prometheus dependency. This library is better maintained and comes with some performance improvements.

v4.17.0

• Improved multi-threaded performance, TSP now scales vertically at around 12-15k requests per second per CPU depending on clock speed.

v4.16.1

• Improved logging performance over v4.16.0
• Fixed a performance regression from v4.15.1 at high throughput.

v4.16.0

• Switched logging output to include some context about where the log message came from. This also reduces log memory usage.
• Removed TSP_SEND_LOGGING_TIMEOUT_MS. Switched to dropping log events immediately if both buffers (TSP_SEND_HIGH_WATER_MARK and TSP_EVENT_LOG_CHANNEL_SIZE, default 50000 messages combined) are full. This results in significant TSP performance improvements w/TSL when under high load.

v4.15.1

• Fix issue that could rarely result in TSP deadlock.
• Clarify log messages related to KmsConfigurationDisabled error.
• Include http_code label for all tsp_request_duration_seconds timeseries.
  • For batch calls, this will be 200 if there are no errors, 207 if there are mix of successes and errors, and the highest of the response status codes when all responses are errors.

v4.15.0

• Added a new environment variable TSP_STARTUP_READINESS that defaults to "relaxed". When set to "strict_configs", the TSP will not report ready if any of the KMS configurations failed to be decrypted with a transient error. By default, the TSP reports ready as long as any of the KMS configurations successfully decrypted.
• Upgraded ironoxide to v4.0.0, consuming a change to reuse a single request client when decrypting KMS configurations.

v4.14.1

• Fixed an issue that could cause KMS configs that always had key leasing enabled to produce unleased documents in the first several minutes after creation.
• Dependency updates.

Known Issue: The fix for the above issue increased the window for an existing slight chance of unwrap failures in the first 3 minutes after a new KMS config with key leasing enabled is created (or toggled on). We’re working on a future patch fix for this, but are releasing the fix as is to treat the more commonly encountered problem.

v4.14.0

• Added feature which allows KMS configurations to be used as they’re processed instead of waiting for all of them to be processed. This allows for less impact when there is a configuration error.
• Fixed noisy logging surrounding push metrics.
• Updated the algorithm for leasing new keys to allow more seamless distribution to TSP deployments with many TSPs.
• Dependency updates.

v4.13.0

• Added support for AWS_ENDPOINT_URL to allow development deployments to use LocalStack.
• Added support for a new error code (KMS_ACCOUNT_ISSUE), which helps differentiate account issues from other issues when interacting with KMS. Check compatibility in your Tenant Security Client.

v4.12.2

• Pre-emptively fix a bug that could result in leased keys staying cached for longer than intended
• Pre-emptively fix a bug that could result in leased keys being cached after key leasing was disabled
• Add more clear WARN logging when an encrypted leased key edge case is encountered

v4.12.1

• Update dependencies

v4.12.0

Note: This is the minimum version required to support Cloaked AI.

• Add Vector secret type.
• Add tsc_version_request_total metric, which counts the number of requests from each TSC language/version. Support for this metric will be added to TSCs over time.

v4.11.1

• Add deterministic encryption feature. See the documentation site and press releases for more detail on how this new feature can be used.
• Added code to work around EDEKs encrypted with leased keys that don’t match the KMS config ID in their header.

v3.3.7

Note: all v3 TSPs are currently EOL. This is a courtesy release for anyone that hasn’t upgraded to v4+ TSP and is affected by this bug.

• Fixed a bug that inconsistently caused EDEKs to be encrypted with non-primary leased keys. This bug exists in TSPs until v4.3.0
• Backport code to work around EDEKs encrypted with leased keys that don’t match the KMS config ID in their header.

v4.10.0

• Dependency updates.
• Update base image to be built on scratch instead of alpine.

v4.9.0

• Add support for numeric tenant secret ids.
• Dependency updates.

v4.8.0

• Update Alpine to 3.17.
• Switch to the official AWS SDK for internal KMS calls.
• Update dependencies and rust version.

v4.7.0

• Added support for Thales CipherTrust Manager (including Community Edition). Initial implementation and testing was done with both Community Edition and managed services (in cooperation with Complior).
• Earlier TSP versions will report a non-terminal error about encountering a config of unknown type “THALES”. If you see that error, you need to upgrade to 4.7.0+ to support current tenant configurations.

v4.6.1

• Added search secret migration. If you’re using Cloaked Search integrated with TSP to manage tenant keys, upgrade to this version so that tenant search secrets can be automatically migrated when tenants switch to new primary KMS configurations.

v4.6.0

• Support additional key derivation type based on HMAC SHA-256.

v4.5.0

• Added new optional environment variable API_KEY_2. If present, this API key is an alternative that clients can use for authorization. Intended to be used when rotating API keys.

v4.4.2

• Added multi arch builds. amd64 and arm64 docker containers are both published to gcr.

v4.4.1

• Added error codes for KMS throttling.

v4.3.0

• Added support for KMS configs assigned to multiple organizations.

v4.2.0

• Added environment variables TSP_BIND_ADDRESS, TSP_HTTP_BIND_ADDRESS, TSP_HEALTH_BIND_ADDRESS, and TSP_EVENT_BIND_ADDRESS to allow customization of bind addresses for the TSP.
• Dependency updates.

v4.1.2

• Fixed a bug that could cause leased keys to stay in memory longer than intended.

v4.1.0

• Initialize requests in flight metrics to 0 on startup.
• Dependency updates.

v4.0.1

• Fixed a bug that could cause leased keys to stay in memory longer than intended.

v4.0.0

• Incremental configuration refreshes, which improves performance and memory use when the TSP has many tenants.
• Removed deprecated metadata interface.
• Separated tenant-security-logdriver from tenant-security-proxy, see documentation for configuration information.

Compatibility

• tenant-security-logdriver must be version 4.0.0+
• TSC-node must be version 2.0.0+
• TSC-java must be version 3.0.0+

v3.3.6

• Improve performance of TSP startup when many leased keys need to be generated.
• Reduce per-tenant memory footprint of TSP by sharing HTTP clients.
• Remove tenant_id label from published metrics to reduce the number of metrics dimensions for cases with many tenants.
• Dependency updates.

v3.3.5

• Improve performance of TSP startup.
• Use RustTLS instead of OpenSSL TLS connections.

v3.3.4

• Logging service performance improvements.

v3.3.3

• Dependency updates. No code changes.

v3.3.1

• Fixed a bug leading to dropped security events in cases with high tenant activity for prolonged periods.

v3.3.0

• Added rekey endpoint and functionality.

v3.2.0

• Added backwards compatible wrap/unwrap/batch interfaces.

Compatibility

• TSC-node must be version 1.0.0+
• TSC-java must be version 2.0.0+

In the future we’ll always produce at least a minor migration version when making breaking changes to the TSP/TSC to prevent possible downtime.

v3.1.1

• Fixed a bug that prevented TSP from reporting ready if no logging configs were present.

v3.1.0

• Added prometheus metrics for TSP container.
• Improved resiliency of reading stored logging events.
• Added checks for misconfigured logdriver volume.

v3.0.0

• Added Security Event endpoint and functionality.
• Fixed a bug that may prevent the TSP from starting when corrupt events are in its DB.
• Improved TSP stdout log consistency.
• Added TSP_SEND_LOGGING_TIMEOUT_MS, TSP_SEND_HIGH_WATER_MARK, and TSP_EVENT_LOG_CHANNEL_SIZE optional configurations.

Compatibility

• TSC-node must be version 2.0.2+
• TSC-java must be version 3.0.1+

v2.3.0

• Standardized JSON logging across the proxy and logdriver.
• Added limiting functionality to the logdriver tenant buffers. If defaults don’t work for you contact IronCore for settings to tweak.
• Improve healthchecks.

v2.2.0

• Adds logging of TSP activity to a tenant’s logging system (GCP Stackdriver and Splunk are currently supported). This feature must be configured on a per-tenant basis. If unsent log-events need to survive TSP restarts, a persistent volume needs to be added to the TSP container.

v2.1.0

• Enables key leasing feature within the TSP. Key leasing must be enabled on a per-KMS configuration basis for leased keys to be used. By default, upgrading to this version of the TSP will have no effect until a KMS configuration is updated to allow for key leasing.

v2.0.2

• Fixes unwrap of certain Azure keys which do not contain an embedded version header.

v2.0.1

• Fix bug that caused KMS config request interval to fail if the Config Broker couldn’t be reached. Now an error message will be logged but the TSP will retry the request to the Config Broker on the next planned interval.
• Fixes behavior of TSP if the configuration/keys of a running container are revoked within the Config Broker. In this case the TSP will fully exit as it is no longer in a valid state.

v2.0.0

• Add batch unwrap and wrap endpoints to the service.
• Logging improvements.
• Rewrite of the TSP in Rust for performance, stability, and binary size improvements

Note

Key leasing has been pushed to a later release once event logging is completed, 2.0.0 is production ready.

v2.0.0-beta.1

• Add key leasing.

Warning

Don’t use in production until audit logging for leased keys is introduced, as tenant KMS logs won’t reflect how the keys are actually being used.

v2.0.0-beta.0

• Add batch unwrap and wrap endpoints to the service.

v1.4.5

• Add retries on KMS configuration decrypts to cut down on intermittent issues impacting customers.

v1.4.4

• Add extra logging traces for configuration decrypt calls that fail.

v1.4.3

• Fixed issue with Azure versioned keys.
• NOTICE on upgrading to this version, any Azure EDEKs should be batch decrypted and re-encrypted to avoid future issues with Azure key versioning.

v1.4.2

• Improved error handling for some classes of Azure KMS authentication errors.

v1.4.1

• Fixed a replay security vulnerability with API calls to the Config Broker.
• Dropped base image from Alpine 3.10 to Alpine 3.9 now that it is vulnerability free and since 3.10 was sometimes causing segfault problems.

v1.4.0

• Added additional error codes which provide better granularity about why requests to the tenant’s KMS failed to succeed. These new error codes are covered in more detail within the Tenant Security Client changelog.
• Added a single level of retry for when a KMS cannot be reached. If the network is down or some other networking problem occurs, the Proxy will automatically attempt a single retry of the request in case the network was only temporarily unreachable.
• Dropped base image from Alpine 3.10 to Alpine 3.9 now that it is vulnerability free and since 3.10 was sometimes causing segfault problems.

v1.3.0

• Add caching of KMS SDK clients to prevent authorization rate limiting errors. Clients credentials will be refreshed every time configurations are pulled from the Config Broker.
• Dropped base image from Alpine 3.10 to Alpine 3.9 now that it is vulnerability free and since 3.10 was sometimes causing segfault problems.

v1.2.0 (Unsupported)

• Renamed container to tenant-security-proxy.

Update

Support for this version was dropped due to a vulnerability. See the 2019-10-30 entry in our security advisories list.

v1.1.0

• Changed permissions for and moved PM2 to run within app directory.

v1.0.0

Initial release.

Versioning Policy

See our container versioning policy documentation.