Docs
SaaS Shield
Suite
Tenant Security Logdriver
Changelog

Changelog

See our buildlog for relationships between specific docker hashes and version tags. The most recent releases are at the bottom of the file. There will only ever be multiple hashes for a single version tag if the underlying image was rebuilt to fix a security vulnerability.

v6.0.0

Breaking Changes

This release replaces the transport layer between the Tenant Security Proxy and Tenant Security Logdriver, adding batching and (optional) encryption. This release doesn’t break the TSC/Alloy -> TSP contract, it does break the TSP -> TSL contract. Zero downtime migration is possible by deploying new TSPs and TSLs together next to existing ones before tearing down the old ones.

Before you deploy

You need to generate a certificate and key, shared by every Logdriver in a pool. This is self-signed and needs no SAN or CA.

bash

openssl req -x509 -newkey rsa:2048 -keyout tsl.key -out tsl.crt -days 365 -nodes -subj "/CN=logdriver"
openssl x509 -in tsl.crt -outform DER | openssl dgst -sha256 -binary | base64

The printed digest is what the TSP will be configured with, the cert and key are for TSL configuration.

Configuration Changes

TSP:

variable	notes
`TSP_LOGDRIVER_URL`	unset disables event delivery, set to TSL(s) Service VIP, load balancer, or DNS hostname
`TSP_LOGDRIVER_CERT_FINGERPRINT`	required when the URL is `https://`
`TSP_LOGDRIVER_AUTH_TOKEN`	required when the URL is `https://`, shared secret with TSL, can be generated with `openssl rand -base64 32`
`TSP_LOGDRIVER_COMPRESS`	optional, default off, only suggested on if you’re paying egress TSP -> TSL

TSL:

variable	notes
`LOGDRIVER_HTTP_BIND_ADDRESS`	required; startup fails without it, address to receive events at
`LOGDRIVER_HTTP_TLS_CERT` / `_KEY`	cert PEM paths for TLS; set both or neither
`LOGDRIVER_INGEST_AUTH_TOKEN`	required whenever TLS is configured, shared secret with TSP, can be generated with `openssl rand -base64 32`

Removed: LOGDRIVER_EVENT_PRODUCER_URL, TSP_EVENT_BIND_ADDRESS, TSP_SEND_HIGH_WATER_MARK, TSP_ENABLE_LOGDRIVER_INTEGRATION. Logdriver ignores the first silently. TSP logs the other three at startup with what replaces each, and treats no TSP_LOGDRIVER_URL with one of the old TSL variables as fatal on the assumption you want events but haven’t correctly configured the changes.

Sizing Pools

See sizing the pool.

Deploying

Old and new pairs coexist. An old TSP keeps delivering to its old Logdriver over ZeroMQ, and a new TSP delivers to a new Logdriver over HTTPS; nothing crosses.

Stand up the new Logdrivers with the certificate, key and token. Each needs its own volume for the durable store, exactly as before. Wait for /ready.
Deploy the new TSPs pointed at the new Logdriver address. They can be deployed at the same time, but a TSP that starts first will queue events and retry until its Logdriver answers.
Verify with the counters below before going further.
Retire the old TSPs, then the old Logdrivers. Give each old Logdriver time to drain its durable store to the sinks after its producers stop.

Grace period. A TSP on this release drains its queued events to Logdriver on SIGTERM, for up to 15 seconds, and logs whether that drain finished. Allow at least that much: Kubernetes’ default terminationGracePeriodSeconds of 30 is enough. A shorter grace period means SIGKILL arrives mid-drain, discarding whatever is still queued with nothing logged, since the process is gone before it can report.

Verifying

The two services now count the same events from both ends, so whether everything TSP produced reached Logdriver’s durable store is a question with an answer. TSP serves its counters at /metrics on TSP_HTTP_BIND_ADDRESS (default port 7777), Logdriver at /metrics on its health port, 9001.

tsp_security_events_produced_total            # events TSP generated
tsp_security_events_delivered_total           # events Logdriver acknowledged
tsp_security_events_undeliverable_total       # events Logdriver rejected as malformed
tsp_real_time_security_event_failures_total   # events TSP could not queue or encode
tsp_security_event_queue_depth                # events waiting to be delivered
logdriver_events_received_total               # events Logdriver decoded
logdriver_events_committed_total              # events Logdriver made durable
logdriver_events_dropped_total                # events lost inside Logdriver, by stage

Two sums close on a healthy pair, allowing for events in flight at the moment you scrape:

produced = delivered + undeliverable + failures + queue_depth
received = committed + dropped

Across the pair, delivered and received should track each other. received running ahead is duplicates: a producer that resends a batch after an uncertain outcome has those events counted once by TSP and twice by Logdriver.

Read against those:

counter	meaning
`tsp_security_event_delivery_retries_total`	consumers behind, or a config fault. Non-zero with rising `tsp_security_event_queue_depth` is the early warning
`tsp_security_events_undeliverable_total`	Logdriver answered `400`. These events are discarded rather than retried
`logdriver_batches_shed_total`	Logdriver returning `429`. Expected under load; the producer retries elsewhere
`logdriver_batches_undecodable_total`	a batch arrived whole and would not parse: a version-skewed producer or a corrupted body, not load
`logdriver_batches_duplicate_total`	a producer resent after an uncertain outcome. Expected at low rates; delivery is at-least-once

The two to alert on during the migration are tsp_security_events_undeliverable_total and logdriver_batches_undecodable_total. Both mean a producer and a consumer disagree about the format, and events counted by either are gone rather than retried.

Do not alert on logdriver_events_dropped_total directly. Its stage label means no series exists until something is dropped, so on a healthy Logdriver it is absent rather than zero and the alert silently never fires. Alert on logdriver_events_received_total - logdriver_events_committed_total instead.

Rolling back

Redeploy the previous images and the previous configuration, on both sides together. There is no single variable that reverts one service, and a new TSP cannot talk to an old Logdriver.

Events already committed to a Logdriver’s durable store are delivered by that Logdriver. Roll back the consumer before it has drained and those events stay in its store until it runs again with a compatible configuration.

v5.1.0

Optimize linking.
Dependency updates.

v5.0.2

Dependency updates.
Log Logdriver version on startup to assist with debugging.

v5.0.1

Dependency updates

v5.0.0

BREAKING CHANGE: Swapped to a more performant event persistence system. No changes are needed to upgrade to 5.0.0, but when upgrading you should allow for a graceful shutdown rollout, to keep events from being stranded in the old DB file.
Improved multi-threaded performance by using an alternative to the standard musl allocator
Further improved throughput, with one 2 CPU TSL now able to keep up with a fully loaded 2 CPU TSP. A 2 CPU TSL can now handle events for 50k+ requests per second through the TSP without dropping any events.

Compatibility

tenant-security-proxy must be version 4.0.0+

v4.4.0

Switched logging output to include some context about where the log message came from. This also reduces log memory usage.
Improved single tenant throughput. Previously single tenant speed was by default limited to 450 rps, now the TSL will more efficiently use resources while there’s one very active tenant, while still being fair.
Dependency updates.