1. Docs
  2. SaaS Shield
  3. Suite
  4. Tenant Security Logdriver
  5. Deployment
  1. Docs
  2. SaaS Shield
  3. Suite
  4. Tenant Security Logdriver
  5. Deployment

Logdriver Deployment

The Logdriver is deployed alongside the TSP and includes health check endpoints and optional configuration by environment variables to tune performance.

Configuration

Outside of the configuration mentioned in the startup section of the overview, there are several optional environment variables that allow for tuning. In general it is recommended that you don’t specify these (which will cause the container to use the default values) unless you are instructed to adjust them to resolve an issue.

  • LOGDRIVER_CHANNEL_CAPACITY. Default: 1000. Controls the number of messages that can be held in buffers between logdriver pipeline stages. Increasing this will have a memory impact.
  • LOGDRIVER_SINK_BATCH_SIZE. Default: 1000. Maximum number of events that can be bundled into a single batch call to a tenant’s logging system. Increasing this may slow down network calls to cloud logging sinks, but may allow for faster draining of high volume tenants’ buffers.
  • LOGDRIVER_BUFFER_POLL_INTERVAL. Default: 2000. Interval (in milliseconds) between each reaping pass of tenant buffers. Decreasing this will increase data rate but may result in a buildup of uncompleted network calls that can eventually use up the container’s network resources.
  • LOGDRIVER_CONFIG_REFRESH_INTERVAL. Default: 600. Interval (in seconds) between each logdriver configuration cache refresh.
  • LOGDRIVER_CHANNEL_TIMEOUT. Default: 250. Time (in milliseconds) that pipeline channel sends are allowed before they are abandoned.
  • LOGDRIVER_EVENT_PRODUCER_URL. Default: “tcp://localhost:5555”. Target from which to pull events.

Health and Liveness Checks

The Docker container also exposes endpoints for checking liveness and health of the container. The checks are implemented based on the Kubernetes lifecycle concepts. The exposed URLs and their meaning are

  • /health: Returns a 200 status code when the container is ready to accept requests. Returns a 500 status code when the server is shutting down or is still initializing.
  • /live: Returns a 200 status code when the container is not shutting down. Returns a 500 status code when the server is shutting down.
  • /ready: Returns a 200 status code when the container is ready to accept requests. Returns a 500 status code when the server is not ready to accept requests.

The container will not report as being “ready” until it has retrieved and decrypted the initial set of tenant logging configurations from the Configuration Broker. If the Logdriver is overloaded it will also report 500 NOT_READY until it is able to work through some of its logging backlog.

Each of these health endpoints are running on port 9001 within the Docker image.

Performance

Logging performance can be measured across related dimensions: per-tenant and global.

Per-Tenant Performance

Logdriver defines a maximal rate a single tenant can sustain. By default, this is 500 log events per second per tenant, although this is tunable via configuration. This limit is in place, primarily, to provide fairness between tenants and to avoid overwhelming any particular log sink.

Global Performance

Expected sustained performance should be 5000-7000 operations per second, depending on computing and network resources. Burst performance can be higher. Keep the per tenant fairness in mind when testing the Logdriver with a low number of tenants, because total throughput will be higher when a given test rate accounts for the 500 log events per second per tenant.

Resource Usage

Memory

There are two major factors related to memory usage of the container.

There is a flat amount of memory, around 500B per-tenant, used to store information about them needed to make logging calls.

Logdriver will also buffer a maximum of 350,000 events across all tenants in memory before it goes into a NOT_READY state and drops events that it receives. Events take roughly 2KB of memory each, so this section of the memory use is capped at around 700MB.

We recommend 1256MiB as a default that will cover most use cases.

CPU

After startup, CPU will primarily be used to to marshal events through to their log sinks, mostly batching and making asynchronous HTTP calls. Logdriver also needs CPU to decrypt logging configurations, which it pulls on a 10 minute schedule. In order to allow events to be processed and sent out while decryption is taking place, a minimum of 2 CPUs should be given to the container.

We recommend 2 CPUs as a default for most use cases.

Failure Modes

Failure to Deliver Log Messages

Sustained load in excess of the per-tenant or global guidelines will eventually cause the Logdriver service to start rejecting messages. Warnings will appear in the TSP and Logdriver’s logs. The TSP will continue to function, but the log messages will not be delivered to the tenant’s logging system.

This behavior can also be triggered by very large bursts of activity.

In this case, the Logdriver container’s readiness check (/ready) will start to return 500. Traffic should be directed to another TSP+Logdriver set until /ready returns a 200. Ensure that whatever orchestration you’re using doesn’t remove all TSPs from rotation this way, since the TSP does still function in this state with reduced logging reliability and slower response times.

Troubleshooting

File Descriptor Limits

File descriptor limit errors may show up if you have too much concurrent traffic flowing through a single Logdriver instance. If you notice these errors in the logs you should either increase the file descriptor limit on that Logdriver or horizontally scale to provide more Logdrivers to the TSP the overloaded one is serving.

Example Deployments

Example Docker Compose

We don’t recommend running a simple docker compose like this in production, but it is useful to see the basics of what is needed to run the Tenant Security Proxy (TSP) and Logdriver (LD) together. If you need a more robust production example, see the kubernetes example.

YAML
version: "3.3" services: tenant-security-proxy: env_file: - ./config-broker-config.conf ports: - "7777:7777" - "9000:9000" image: tenant-security-proxy links: - tenant-security-logdriver tenant-security-logdriver: environment: - LOGDRIVER_EVENT_PRODUCER_URL=tcp://tenant-security-proxy:5555 env_file: - ./config-broker-config.conf ports: - "9001:9001" image: tenant-security-logdriver volumes: - type: bind source: /tmp target: /logdriver

Example Kubernetes Deployment

When running the TSP and LD together, LD needs a persistent disk to store log messages, to ensure nothing gets lost. To give each LD its own disk, we use a StatefulSet with a volumeClaimTemplate. The StatefulSet is a bit more complicated than the Deployment we used for TSP alone.

Here’s the StatefulSet, with both its required headless service (tenant-security-proxy-sts) and the service that will be used by TSP clients (tenant-security-proxy).

YAML
# This is the client-facing service. apiVersion: v1 kind: Service metadata: name: tenant-security-proxy labels: app: tenant-security-proxy spec: type: NodePort ports: - port: 7777 targetPort: 7777 name: http selector: app: tenant-security-proxy --- # This is the headless service used by the StatefulSet to keep track of its replicas. apiVersion: v1 kind: Service metadata: name: tenant-security-proxy-sts spec: ports: - port: 7777 name: http clusterIP: None selector: app: tenant-security-proxy --- apiVersion: apps/v1 kind: StatefulSet metadata: name: tenant-security-proxy spec: # We're not setting replicas here because that's controlled by a HorizontalPodAutoscaler. selector: matchLabels: app: tenant-security-proxy serviceName: tenant-security-proxy-sts podManagementPolicy: Parallel template: metadata: labels: app: tenant-security-proxy annotations: prometheus.io/scrape: 'true' prometheus.io/port: '7777' spec: securityContext: runAsUser: 2 # Any non-root user will do. runAsGroup: 2 fsGroup: 2 runAsNonRoot: true containers: - name: tenant-security-proxy image: gcr.io/ironcore-images/tenant-security-proxy:{CHOSEN_TAG} resources: requests: cpu: 2 memory: 1256Mi limits: cpu: 2 memory: 1256Mi envFrom: - secretRef: # See https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/overview/#startup name: tsp-secrets env: - name: RUST_LOG value: info # Values are trace, debug, info, warn, error ports: - containerPort: 9000 name: health - containerPort: 7777 name: http livenessProbe: httpGet: path: /live port: health readinessProbe: httpGet: path: /ready port: health securityContext: allowPrivilegeEscalation: false - name: logdriver image: gcr.io/ironcore-images/tenant-security-logdriver:{CHOSEN_TAG} resources: requests: cpu: 2 memory: 1256Mi limits: cpu: 2 memory: 1256Mi envFrom: - secretRef: # See https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/overview/#startup name: tsp-secrets env: - name: RUST_LOG value: info ports: - containerPort: 9001 name: health livenessProbe: httpGet: path: /live port: health readinessProbe: httpGet: path: /ready port: health securityContext: allowPrivilegeEscalation: false volumeMounts: - mountPath: /logdriver name: logdriver volumeClaimTemplates: - metadata: name: logdriver spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi

Autoscaling

The autoscaling configuration for this combined StatefulSet is nearly identical to the one for the simpler TSP Deployment.

YAML
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: tenant-security-proxy spec: maxReplicas: 10 minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: StatefulSet name: tenant-security-proxy metrics: - pods: metric: name: tsp_requests_in_flight target: averageValue: 4.5 type: AverageValue type: Pods # If you're using Kubernetes 1.20 or later, change this to a ContainerResource. - resource: name: cpu target: averageUtilization: 80 type: Utilization type: Resource