Skip to content
Kubernetes

Kubernetes
Since v1.0.0

Introduction

The Kubernetes provider scales Deployments and StatefulSets to zero replicas when their session expires, and back up when a request needs them. It can also hibernate CloudNativePG Clusters and pause OT-CONTAINER-KIT Redis instances. Sablier runs as a pod in the cluster and calls the Kubernetes API with its service account.

Install Sablier

Install the Helm chart in its own namespace:

helm repo add sablier https://sablierapp.github.io/helm-charts
helm install sablier sablier/sablier --namespace sablier --create-namespace -f values.yaml

The chart selects the Kubernetes provider and creates the service account and RBAC Sablier needs. Set Sablier’s configuration under config in your values file:

config:
  provider:
    kubernetes:
      # "_" (the default) is kept for backward compatibility; "." or "/"
      # reads better in instance names such as deployment.default.whoami.1
      delimiter: "."
  sessions:
    default-duration: 5m

To manage CloudNativePG Clusters or Redis instances, grant the extra permissions:

rbac:
  cnpg: false     # true to manage CloudNativePG Clusters
  otkRedis: false # true to manage OT-CONTAINER-KIT Redis instances

Reverse proxy plugins reach Sablier at http://sablier-sablier.sablier:10000: the chart names the Service sablier-<release>.

The CloudNativePG and Redis permissions are optional. Without them, or without their CRDs, Sablier skips those integrations.

get and update on deployments/scale and statefulsets/scale are deprecated. Sablier scales with a patch on the scale subresource. If your RBAC only grants get and update there, as it did for Sablier 1.19 and earlier, Sablier falls back to them and logs a warning once per kind. The fallback can fail when the workload is written concurrently, and it will be removed in the next major release. Grant patch on both subresources.

Usage example

Opt a Deployment in with the sablierapp.dev/enable label, and give it a group:

# A Deployment Sablier scales to zero when idle and back up on demand, and the
# Service your reverse proxy routes to.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  labels:
    app: whoami
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: sablierapp/mimic:v0.3.3
          args: ["-healthy=true"]
          ports:
            - containerPort: 80
          # Sablier reports the Deployment ready once its pods pass this probe.
          readinessProbe:
            exec:
              command: ["/mimic", "healthcheck"]
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  selector:
    app: whoami
  ports:
    - port: 80

Then configure your reverse proxy to call Sablier for the whoami group in front of the whoami Service. On the first request, Sablier scales the Deployment to one replica and holds the request, or shows a waiting page, until the pod is ready. When the session expires, Sablier scales it back to zero.

Sablier scales opted-in workloads to zero when it starts. A workload you deploy afterwards keeps running until its first session expires, unless you stop externally-started instances.

Provider reference

Configuration

OptionDefaultDescription
--provider.kubernetes.delimiter_The separator in instance names. Prefer . or /; _ is kept for backward compatibility.
--provider.kubernetes.ready-on-first-replicafalseReport a workload ready as soon as one replica is ready, instead of all of them. Since v1.16.0.
--provider.kubernetes.readinessreplicasendpoints also waits until the workload’s Services route to its pods. See readiness.
--provider.kubernetes.qps5Client-side limit of Kubernetes API requests per second.
--provider.kubernetes.burst10Requests allowed above that limit in a burst.

Every --provider.* option that is not provider-specific also applies; see the CLI reference.

The Kubernetes client waits on the client side whenever Sablier sends more than qps requests per second for longer than burst allows. Sablier discovers and indexes workloads with a few list and watch requests, but some work still costs about one request per opted-in workload: scaling running workloads to zero when Sablier starts, warming workloads that were started outside Sablier, and checking workloads that have running hours. With --provider.kubernetes.readiness=endpoints, each check of a waiting request also costs three list requests once the workload’s replicas are ready, until its Services route to it. With many opted-in workloads, this can delay requests that arrive at the same time. If you manage more than a few dozen workloads, raise both values, for example --provider.kubernetes.qps=50 --provider.kubernetes.burst=100. The API server’s own API Priority and Fairness still protects it.

Workloads

Deployments and StatefulSets

Sablier scales Deployments and StatefulSets through their scale subresource:

metadata:
  name: whoami-db
  labels:
    app: whoami-db
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami

CloudNativePG Clusters
Since v1.13.0

Sablier starts and stops CloudNativePG Cluster custom resources with the operator’s declarative hibernation:

  • Stop sets cnpg.io/hibernation: "on". The operator removes the cluster’s pods and keeps its PVCs.
  • Start sets cnpg.io/hibernation: "off", and the operator resumes the cluster.
# Requires the CloudNativePG operator, and rbac.cnpg=true in the Helm chart.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: whoami-pg
  labels:
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  instances: 3
  storage:
    size: 1Gi

Put the application, its database and any other dependency in the same group, so one request wakes the whole stack and inactivity hibernates all of it.

Resuming a hibernated cluster takes longer than a scale-up: the PVCs are reattached and PostgreSQL recovers. Make sure your reverse proxy timeouts allow for it.

OT-CONTAINER-KIT Redis instances
Since v1.14.0

The OT-CONTAINER-KIT redis-operator reconciles its StatefulSets back to their replica count, which would undo a scale to zero. Sablier pauses the operator around each scale:

  • Stop sets the redis.opstreelabs.in/skip-reconcile: "true" annotation on the Redis resource, then scales its StatefulSet to 0.
  • Start scales the StatefulSet back to 1, then removes the annotation so the operator resumes.

If the scale fails, Sablier clears the annotation immediately, so the operator is never left paused with pods running.

Label the Redis resource. The operator copies the labels to the StatefulSet it manages:

# Requires the OT-CONTAINER-KIT redis-operator, and rbac.otkRedis=true in the
# Helm chart. The operator copies these labels to the StatefulSet it manages.
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: Redis
metadata:
  name: whoami-redis
  labels:
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  kubernetesConfig:
    image: quay.io/opstree/redis:v8.10.1

Instance names

A reverse proxy can target one workload by name instead of a group. The whoami Deployment of the usage example is named:

deployment_default_whoami_1

The name has four parts, separated by the delimiter (_ by default):

  • Kind: deployment, statefulset or cnpgcluster. Target a Redis instance through its StatefulSet.
  • Namespace of the workload.
  • Name of the workload.
  • Replicas: the replica count Sablier scales to on start, since v1.15.0.

Prefer groups: a name ties your proxy configuration to the workload’s kind and namespace.

Labels
Next release

On Kubernetes, Sablier keys start with sablierapp.dev/ instead of sablier.:

metadata:
  name: team-api
  labels:
    app: team-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: "team-a,team-b" # comma: must be an annotation
  • enable must be a label. Sablier finds workloads with a label selector, and selectors can’t read annotations.
  • Values with commas or colons must be annotations. Kubernetes doesn’t allow them in labels, so a list of groups or a time window goes in an annotation.
  • Any other key can be either. If a key is set as both, the annotation wins.

The label reference lists every key.

Legacy sablier.* keys

Earlier releases only read sablier.* keys. Sablier still reads them, but they are deprecated on Kubernetes: keys without a prefix belong to the user and can clash with your own. If a workload has both forms, the sablierapp.dev/ key wins.

Workload settings

Each excerpt below is the metadata of a complete manifest in examples/kubernetes.

Enable
Since v1.4.0

sablierapp.dev/enable: "true" opts the workload in. It is required, must be a label, and any other value is ignored. See the usage example.

Group
Since v1.4.0

sablierapp.dev/group puts the workload in one or more groups. A session for any of them starts it. Without it, the workload is in the default group.

A list of groups contains commas, so set it as an annotation:

metadata:
  name: team-api
  labels:
    app: team-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: "team-a,team-b" # comma: must be an annotation

Learn more about groups.

Ready after
Since v1.13.0

sablierapp.dev/ready-after keeps reporting the workload as not ready for a settling delay after its pods first become ready. Use it when a pod passes its readiness probe before it can serve traffic.

metadata:
  name: slow-starter
  labels:
    app: slow-starter
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: slow-starter
    sablierapp.dev/ready-after: "30s"

Learn more about delaying readiness.

Ready on start
Since v1.15.0

sablierapp.dev/ready-on-start: "true" reports the workload ready as soon as Sablier sends the scale-up, without waiting for its pods.

metadata:
  name: fast-starter
  labels:
    app: fast-starter
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: fast-starter
    sablierapp.dev/ready-on-start: "true"

Learn more about marking ready on start.

Running hours
Since v1.13.0

  • sablierapp.dev/running-hours keeps the workload running during a daily window.
  • sablierapp.dev/running-days limits that window to some weekdays, since v1.15.0.

Both values contain colons or commas, so set them as annotations:

metadata:
  name: office-app
  labels:
    app: office-app
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: office-app
    sablierapp.dev/running-hours: "09:00-18:00" # colon: must be an annotation
    sablierapp.dev/running-days: "Mon,Tue,Wed,Thu,Fri"

The window uses Sablier’s local time. With Helm, set the time zone with the TZ environment variable:

# sablierapp.dev/running-hours windows use Sablier's local time.
extraEnv:
  - name: TZ
    value: Europe/Paris

Learn more about running hours.

Anti-affinity
Since v1.15.0

sablierapp.dev/anti-affinity lists groups the workload backs off from: Sablier keeps it idle while any of them has an active session. Set a list of groups as an annotation.

metadata:
  name: batch-worker
  labels:
    app: batch-worker
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: batch
    sablierapp.dev/anti-affinity: "streaming"

Learn more about anti-affinity.

Scale instead of stop
Since v1.13.0

Keep the workload running when idle, optionally throttled, instead of scaling it to zero:

  • idle.replicas set to 1 or more keeps that many replicas when idle.
  • idle.cpu and idle.memory set the CPU and memory limits while idle.
  • active.replicas, active.cpu and active.memory restore them on the next session.

CPU and memory use resource quantities, such as 100m, 2, 64Mi or 1Gi.

metadata:
  name: warm-api
  labels:
    app: warm-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: warm-api
    sablierapp.dev/idle.replicas: "1"
    sablierapp.dev/idle.cpu: "100m"
    sablierapp.dev/idle.memory: "64Mi"
    sablierapp.dev/active.cpu: "2"
    sablierapp.dev/active.memory: "512Mi"
Changing limits triggers a rolling update: new pods replace the old ones, so the service stays available. Sablier only changes the limits of the pod template’s first container, never its requests, so make sure your nodes can schedule the active limits.

Learn more about scaling instead of stopping.

Readiness

Sablier reports a Deployment or StatefulSet:

  • ready when it has at least one desired replica and all of them are ready. With --provider.kubernetes.ready-on-first-replica, one ready replica is enough.
  • stopped when it is scaled to 0, even while pods are still terminating.
  • starting otherwise.

A replica is ready when its pod passes its readiness probe, so give your containers one.

A CloudNativePG Cluster is stopped when its cnpg.io/hibernation annotation is "on", ready when status.readyInstances reaches spec.instances, and starting otherwise.

Wait for the Service to route
Next release

A pod turns ready before its Service routes to it. The EndpointSlice controller first marks the pod’s endpoint ready, then kube-proxy and your reverse proxy apply the change. A request that Sablier releases in between fails with a 502 or a 503.

With --provider.kubernetes.readiness=endpoints, a Deployment or StatefulSet whose replicas are ready stays starting until enough of its pods are routed: all desired replicas, or one with --provider.kubernetes.ready-on-first-replica. A pod is routed when it is ready and every Service that selects it lists it as a ready endpoint. A per-replica Service, such as one that selects statefulset.kubernetes.io/pod-name, counts too. The instance’s message names the Service it waits for. A workload that no Service selects is reported from its replicas only. CloudNativePG Clusters are not affected.

Sablier only checks this while requests wait for the workload. Other checks, such as running hours or the expiration of a session, report the workload from its replicas.

This mode needs list on services, pods and endpointslices (discovery.k8s.io): uncomment the last rules of the manifests. The Helm chart does not grant them yet: bind an extra ClusterRole to the sablier-<release> service account. If Sablier cannot list them, it logs a warning once and reports workloads from their replicas. If a list fails for another reason, such as a timeout, the workload stays starting and Sablier checks again on the next request.

The reverse proxy can still lag a little behind the EndpointSlice: kube-proxy programs a ClusterIP Service shortly after, and Traefik applies provider changes at most every providersThrottleDuration (2s by default). A retry on connection failure in the proxy, or a short ready after delay, covers the rest.

Limitations

  • Sablier must run as a pod in the cluster it manages: it only uses in-cluster credentials.
  • Block I/O throttling is not available on Kubernetes.
  • CPU and memory throttling applies to the first container of the pod template only, and restarts the pods.
  • Without the Redis permissions, Sablier logs a warning and scales the StatefulSet anyway; the operator then scales it back up.

Examples