Tenant Security Logdriver Deployment

The Tenant Security Logdriver (TSL) is deployed as a pool that one or more Tenant Security Proxies (TSPs) deliver security events to. The service includes health check endpoints, and it can be further configured using optional environment variables to tune performance.

Event Ingest

Each TSP connects out to the TSL and POSTs batches of events, so a single address can front a whole pool of TSLs: a Kubernetes Service, a load balancer, or a DNS name resolving to several. Every TSL in a pool is interchangeable, and producers rebalance across the pool on their own as instances are added, drained, or shed load.

Required Configuration

• LOGDRIVER_HTTP_BIND_ADDRESS. host:port that the event ingest listener binds. Startup fails without it, since a TSL that cannot ingest is not useful. There is no default; the examples below use 5555.

If the listener is serving TLS, which it should be anywhere the network between your TSPs and your TSLs is not fully trusted, three more variables come into play.

• LOGDRIVER_HTTP_TLS_CERT and LOGDRIVER_HTTP_TLS_KEY. Default: unset, serving plaintext. PEM paths enabling TLS on the ingest listener. Set both or neither. Producers pin the certificate by digest rather than by name, so it can be self-signed with no SAN and no CA. Every TSL in a pool serves the same certificate and key.
• LOGDRIVER_INGEST_AUTH_TOKEN. Default: unset, which accepts events from anything that can reach the port. Shared secret every producer must present as an Authorization: Bearer <token> header, and the identical string goes in each TSP’s TSP_LOGDRIVER_AUTH_TOKEN. Required whenever TLS is configured, and optional without it so a local development setup needs no secret at all. You choose the value; openssl rand -base64 32 produces a suitable one. Every TSL in a pool carries the same token, since a producer is balanced across them, and only one token is accepted at a time, so rotating means changing producers and consumers together. If deployed side-by-side behind the same addresses that may lead to some 401s as old containers try to reach new ones, but those will be retried. Bring up new TSLs -> new TSPs, then bring down old TSPs -> old TSLs, to minimize security event loss.

Generate the certificate, key, and digest with:

bash

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout tsl.key -out tsl.crt -subj "/CN=logdriver" \
  -addext "basicConstraints=critical,CA:FALSE"
openssl x509 -in tsl.crt -outform DER | openssl dgst -sha256 -binary | base64

The certificate and key go on the TSLs. The printed digest is what each TSP is configured with, as TSP_LOGDRIVER_CERT_FINGERPRINT.

Batches that arrive with Content-Encoding: gzip are inflated on the request path. No TSL setting enables or disables that; whether a producer compresses is its own decision, made with the TSP’s TSP_LOGDRIVER_COMPRESS.

Optional Configuration

Outside of the configuration mentioned in the startup section of the overview, there are several optional environment variables that allow for tuning. In general it is recommended that you don’t specify these (which will cause the container to use the default values) unless you are instructed to adjust them to resolve an issue.

• LOGDRIVER_CHANNEL_CAPACITY. Default: 1000. Controls the number of messages that can be held in buffers between logdriver pipeline stages. Increasing this will have a memory impact.
• LOGDRIVER_SINK_BATCH_SIZE. Default: 1000. Maximum number of events that can be bundled into a single batch call to a tenant’s logging system. Increasing this may slow down individual network calls to cloud logging sinks but will allow for faster draining of high volume tenants’ buffers. Increasing this should be your first go-to for improving TSL event throughput.
• LOGDRIVER_BUFFER_POLL_INTERVAL. Default: 2000. Maximum age (in milliseconds) of a less-than-full batch for a tenant before it is sent out. Lengthening this increases the resources available to high throughput tenants at the cost of tenants with less-than-full batches waiting longer for events to arrive at their log sink. Shortening it has the inverse effect.
• LOGDRIVER_CONFIG_REFRESH_INTERVAL. Default: 600. Interval (in seconds) between each logdriver configuration cache refresh.
• LOGDRIVER_CHANNEL_TIMEOUT. Default: 250. Time (in milliseconds) that pipeline channel sends are allowed before they are abandoned.

Health and Liveness Checks

The Docker container also exposes endpoints for checking liveness and health of the container. The checks are implemented based on the Kubernetes lifecycle concepts. The exposed URLs and their meaning are

• /health: Returns a 200 status code when the container is ready to accept requests. Returns a 500 status code when the server is shutting down or is still initializing.
• /live: Returns a 200 status code when the container is not shutting down. Returns a 500 status code when the server is shutting down.
• /ready: Returns a 200 status code when the container is ready to accept requests. Returns a 500 status code when the server is not ready to accept requests.

The container will not report as being “ready” until it has retrieved and decrypted the initial set of tenant logging configurations from the Configuration Broker. If the TSL is overloaded it will also report 500 NOT_READY until it is able to work through some of its logging backlog.

Each of these health endpoints is running on port 9001 within the Docker image, as is the /metrics endpoint. That port is separate from LOGDRIVER_HTTP_BIND_ADDRESS and is not configurable, so don’t expose it to producers.

Metrics

Each TSL container provides the following Prometheus metrics on a /metrics endpoint, served on the health port 9001.

logdriver_events_received_total (counter)

• Events that entered the pipeline. The denominator the other counters here are read against.
• Labels: None

logdriver_events_committed_total (counter)

• Events written to the durable store. This is the commit point for the at-least-once guarantee: past it, an event survives a restart.
• Labels: None

logdriver_events_dropped_total (counter)

• Events lost inside the TSL, by the pipeline stage that dropped them.
• Labels:
  • stage - pipeline stage that dropped the events

logdriver_batches_shed_total (counter)

• Batches refused with a 429 because the pipeline was saturated. The producer retries these elsewhere in the pool, so they are not lost and this is deliberately not counted as a drop. Sustained increases mean consumers are behind. It counts batches rather than events because a refused body is never decoded, so its event count is unknown.
• Labels: None

logdriver_batches_undecodable_total (counter)

• Batches that arrived whole and would not parse, and so were never delivered to a sink. This means a version-skewed producer or a corrupted body, not load. The producer counts these as delivered, accurately, since they arrived and were accepted; this is where they stop. Worth alerting on.
• Labels: None

logdriver_batches_duplicate_total (counter)

• Batches this instance had already seen, which a producer resent because the outcome of its first attempt was unknown. Delivery is at-least-once, so these are expected at low rates rather than a fault, and their events are processed again rather than discarded. This is why logdriver_events_received_total can run ahead of a producer’s tsp_security_events_produced_total. Only duplicates that reach the same instance are visible, so a producer that moves between instances mid-retry is counted by neither.
• Labels: None

Allowing for events in flight at the moment you scrape, logdriver_events_received_total equals logdriver_events_committed_total plus logdriver_events_dropped_total. Across a TSP and the pool it delivers to, tsp_security_events_delivered_total and logdriver_events_received_total should track each other; received running ahead is duplicates. See the TSP metrics for the producer half of that ledger.

Do not alert on logdriver_events_dropped_total directly. Its stage label means no series exists until something is dropped, so on a healthy TSL it is absent rather than zero and the alert silently never fires. Alert on logdriver_events_received_total - logdriver_events_committed_total instead.

Performance

Logging performance can be measured across related dimensions: per-tenant and global.

Per-Tenant Performance

The TSL attempts to provide fairness to all tenants by limiting the maximum age of a batch for any tenant to LOGDRIVER_BUFFER_POLL_INTERVAL. High volume tenants that are reaching full batches before that time period will receive resources any time they hit a full batch, but have the same priority as any low volume tenant whose less-than-full batch is past the max age. This results in efficient resource usage while one or more tenants are under heavy load while still allowing low load tenants to send out their events in a timely manner.

Global Performance

Measured on 2 vCPU VMs, one service to a VM, over HTTPS with compression off:

Burst performance can be higher, but throughput is to some extent dependant on downstream log sink throughputs. If the TSP’s tsp_real_time_security_event_failures_total is increasing, all upstream buffers are saturated and you should add TSL instances and/or increase LOGDRIVER_SINK_BATCH_SIZE.

Two things these numbers do not cover. They count events through the TSL’s durable store, and delivery onward to Splunk or Stackdriver depends on your sink. And they are compression off, which is the default; the TSP’s TSP_LOGDRIVER_COMPRESS buys bandwidth at the cost of TSL throughput, so size below these figures if you enable it.

Sizing the Pool

How many TSPs one TSL carries depends on what your TSPs do:

3,500 events/sec per TSP is events emitted as a side effect of ordinary wrap and unwrap traffic, and is the figure to size an ordinary fleet from. A tenant calling the security event API directly is the other end, and at that rate one or two producers saturate a TSL.

A TSL sheds batches well below its ceiling and producers retry, so logdriver_batches_shed_total climbing is not by itself a sign of an undersized pool. Ingest latency is what to watch alongside it: roughly 7ms with a single producer and 57ms with 24.

Resource Usage

Memory

There are two major factors related to memory usage of the container.

There is a flat amount of memory, around 500B per-tenant, used to store information about them needed to make logging calls.

The TSL will also buffer a maximum of 350,000 events across all tenants in memory before it goes into a NOT_READY state and drops events that it receives. Standard events take roughly 120B - 2KB of memory each, so this section of the memory use is capped at around 700MB. If you’re sending custom security events that are significantly larger, increase memory to compensate based on worst case usage.

We recommend 1256MB as a default that will cover most use cases.

CPU

After startup, CPU will primarily be used to marshal events through to their log sinks, mostly batching, making asynchronous HTTP calls, and writing to stdout. The TSL also needs CPU to decrypt logging configurations, which it pulls on a 10 minute schedule, and for database compaction. In order to allow events to be processed and sent out while CPU bound background tasks are taking place, a minimum of 2 CPUs should be given to the container.

We recommend 2 CPUs as a default for most use cases.

Failure Modes

Failure to Deliver Log Messages

A TSL whose pipeline is saturated refuses batches with a 429 and increments logdriver_batches_shed_total. The producer retries that batch, landing on another instance if the address fronts a pool, so shedding on its own loses nothing and is expected under load. This also happens well below the throughput ceiling, so it is not by itself a sign of an undersized pool.

Sustained load in excess of the global guidelines is different: with every instance shedding there is nowhere for a retry to land, the TSP’s own queue fills, and the TSP starts dropping events. Warnings will appear in the TSP and TSL logs, and the TSP’s tsp_real_time_security_event_failures_total will climb. The TSP will continue to serve wrap and unwrap requests, but the dropped events will not be delivered to the tenant’s logging system. Very large bursts of activity can trigger the same thing.

A saturated TSL’s readiness check (/ready) also starts returning 500 once its in-memory buffers fill, which takes it out of the pool if a Service or load balancer is watching readiness. That is the intended response, since producers rebalance onto the instances that are still keeping up. Take care that your orchestration does not act on a TSL’s readiness by removing TSPs from client-facing rotation, since a TSP with no working TSL still serves encrypt and decrypt correctly.

Rejected Batches

The TSL answers 401 when a producer presents a token that does not match LOGDRIVER_INGEST_AUTH_TOKEN, or none at all. Producers log the mismatch, retry with backoff and keep queueing, so a brief disagreement costs nothing while a sustained one fills the producer’s queue and then drops events. Because only one token is accepted at a time, rotating it means changing every TSL and every TSP together.

The TSL answers 400 and increments logdriver_batches_undecodable_total when a batch arrives whole and will not parse. These events are discarded rather than retried, and the producer counts them in tsp_security_events_undeliverable_total. This means a version mismatch between producer and consumer, or a corrupted body, and never load.

Troubleshooting

File Descriptor Limits

File descriptor limit errors may show up if you have high concurrent traffic flowing through a single TSL instance. If you notice these errors in the logs you should either increase the file descriptor limit associated with TSL or add TSL instances to the pool.

Example Deployments

Example Docker Compose

We don’t recommend running a simple docker compose like this in production, but it is useful to see the basics of what is needed to run the Tenant Security Proxy (TSP) and Tenant Security Logdriver (TSL) together. If you need a more robust production example, see the kubernetes example.

YAML

version: "3.3"
services:
  tenant-security-proxy:
    environment:
      - TSP_LOGDRIVER_URL=http://tenant-security-logdriver:5555
    env_file:
      - ./config-broker-config.conf
    ports:
      - "7777:7777"
      - "9000:9000"
    image: tenant-security-proxy
    links:
      - tenant-security-logdriver
  tenant-security-logdriver:
    environment:
      - LOGDRIVER_HTTP_BIND_ADDRESS=0.0.0.0:5555
    env_file:
      - ./config-broker-config.conf
    ports:
      - "9001:9001"
    image: tenant-security-logdriver
    volumes:
      - type: bind
        source: /tmp
        target: /logdriver

This ingests over plaintext with no token, which is why it is a local-only example. TSP_LOGDRIVER_URL needs no fingerprint or token while it is http://, and both become required as soon as it is https://.

Example Kubernetes Deployment

The TSP and the TSL are separate workloads that scale independently. The TSP holds nothing on disk, so it is a Deployment. Each TSL needs its own persistent disk to store events so nothing is lost across restarts, so the pool is a StatefulSet with a volumeClaimTemplate.

Three pieces connect them: a Secret holding the certificate, key, digest, and token; a Service in front of the TSL pool that TSPs POST to; and TSP_LOGDRIVER_URL on the TSP pointing at that Service. Kubernetes keeps not-ready pods out of a Service’s endpoints, so a TSL that takes itself out of rotation under load stops receiving batches without any further configuration.

First the shared secret material, generated as described in required configuration.

YAML

apiVersion: v1
kind: Secret
metadata:
  name: tsl-ingest
type: Opaque
stringData:
  # The self-signed certificate and key every TSL in the pool serves.
  tsl.crt: |
    -----BEGIN CERTIFICATE-----
    ...
  tsl.key: |
    -----BEGIN PRIVATE KEY-----
    ...
  # Base64 SHA-256 digest of the DER form of tsl.crt, which the TSPs pin.
  TSP_LOGDRIVER_CERT_FINGERPRINT: <digest>
  # One shared token. Both keys hold the same value.
  LOGDRIVER_INGEST_AUTH_TOKEN: <token>
  TSP_LOGDRIVER_AUTH_TOKEN: <token>

Then the TSL pool, with the Service its producers reach it at and the headless service the StatefulSet requires.

YAML

# This is the address TSPs POST events to. It fronts every ready TSL in the pool.
apiVersion: v1
kind: Service
metadata:
  name: tenant-security-logdriver
  labels:
    app: tenant-security-logdriver
spec:
  ports:
  - port: 5555
    targetPort: ingest
    name: ingest
  selector:
    app: tenant-security-logdriver

---

# This is the headless service used by the StatefulSet to keep track of its replicas.
apiVersion: v1
kind: Service
metadata:
  name: tenant-security-logdriver-sts
spec:
  ports:
    - port: 5555
      name: ingest
  clusterIP: None
  selector:
    app: tenant-security-logdriver

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tenant-security-logdriver
spec:
  # Size this from the TSP fleet's event rate. See #sizing-the-pool.
  replicas: 2
  selector:
    matchLabels:
      app: tenant-security-logdriver
  serviceName: tenant-security-logdriver-sts
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: tenant-security-logdriver
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9001'
    spec:
      securityContext:
        runAsUser: 2 # Any non-root user will do.
        runAsGroup: 2
        fsGroup: 2
        runAsNonRoot: true
      containers:
      - name: logdriver
        image: us-docker.pkg.dev/ironcore-images/gcr.io/tenant-security-logdriver:{CHOSEN_TAG}
        resources:
          requests:
            cpu: 2
            memory: 1256MB
          limits:
            cpu: 2
            memory: 1256MB
        envFrom:
        - secretRef:
            # See https://ironcorelabs.com/docs/saas-shield/tenant-security-logdriver/overview/#startup
            name: tsl-secrets
        env:
        - name: RUST_LOG
          value: info # Values are trace, debug, info, warn, error
        - name: LOGDRIVER_HTTP_BIND_ADDRESS
          value: 0.0.0.0:5555
        - name: LOGDRIVER_HTTP_TLS_CERT
          value: /tls/tsl.crt
        - name: LOGDRIVER_HTTP_TLS_KEY
          value: /tls/tsl.key
        - name: LOGDRIVER_INGEST_AUTH_TOKEN
          valueFrom:
            secretKeyRef:
              name: tsl-ingest
              key: LOGDRIVER_INGEST_AUTH_TOKEN
        ports:
        - containerPort: 9001
          name: health
        - containerPort: 5555
          name: ingest
        livenessProbe:
          httpGet:
            path: /live
            port: health
        readinessProbe:
          httpGet:
            path: /ready
            port: health
        securityContext:
          allowPrivilegeEscalation: false
        volumeMounts:
          - mountPath: /logdriver
            name: logdriver
          - mountPath: /tls
            name: tls
            readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: tsl-ingest
            items:
              - key: tsl.crt
                path: tsl.crt
              - key: tsl.key
                path: tsl.key
  volumeClaimTemplates:
    - metadata:
        name: logdriver
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1GB

Then the TSP, which is the TSP-only Deployment with delivery configured. Only the pieces that differ from that example are shown.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-security-proxy
spec:
  selector:
    matchLabels:
      app: tenant-security-proxy
  template:
    metadata:
      labels:
        app: tenant-security-proxy
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '7777'
    spec:
      # The TSP drains queued events for up to 15s on SIGTERM. The default of 30
      # is enough; anything under 15 discards whatever is still queued.
      terminationGracePeriodSeconds: 30
      containers:
      - name: tenant-security-proxy
        image: us-docker.pkg.dev/ironcore-images/gcr.io/tenant-security-proxy:{CHOSEN_TAG}
        envFrom:
        - secretRef:
            # See https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/overview/#startup
            name: tsp-secrets
        env:
        - name: TSP_LOGDRIVER_URL
          value: https://tenant-security-logdriver:5555
        - name: TSP_LOGDRIVER_CERT_FINGERPRINT
          valueFrom:
            secretKeyRef:
              name: tsl-ingest
              key: TSP_LOGDRIVER_CERT_FINGERPRINT
        - name: TSP_LOGDRIVER_AUTH_TOKEN
          valueFrom:
            secretKeyRef:
              name: tsl-ingest
              key: TSP_LOGDRIVER_AUTH_TOKEN

A TSP that starts before any TSL is ready queues its events and retries until one answers, so ordering between the two workloads does not matter.

Autoscaling

The TSP is autoscaled on CPU exactly as in the TSP-only deployment.

The TSL pool is sized from event throughput rather than its own CPU, since it sheds and recovers well below saturation and CPU is a poor proxy for how close it is to its ceiling. Set replicas from the pool sizing guidance and revisit it when the TSP fleet’s event rate changes materially. If you do want it autoscaled, scale on logdriver_events_received_total per pod rather than CPU utilization.