Kubernetes

Kubernetes
Since v1.0.0

Introduction

The Kubernetes provider scales Deployments and StatefulSets to zero replicas when their session expires, and back up when a request needs them. It can also hibernate CloudNativePG Clusters and pause OT-CONTAINER-KIT Redis instances. Sablier runs as a pod in the cluster and calls the Kubernetes API with its service account.

Install Sablier

Install the Helm chart in its own namespace:

helm repo add sablier https://sablierapp.github.io/helm-charts
helm install sablier sablier/sablier --namespace sablier --create-namespace -f values.yaml

The chart selects the Kubernetes provider and creates the service account and RBAC Sablier needs. Set Sablier’s configuration under config in your values file:

examples/kubernetes/values.yaml

config:
  provider:
    kubernetes:
      # "_" (the default) is kept for backward compatibility; "." or "/"
      # reads better in instance names such as deployment.default.whoami.1
      delimiter: "."
  sessions:
    default-duration: 5m

To manage CloudNativePG Clusters or Redis instances, grant the extra permissions:

examples/kubernetes/values.yaml

rbac:
  cnpg: false     # true to manage CloudNativePG Clusters
  otkRedis: false # true to manage OT-CONTAINER-KIT Redis instances

Reverse proxy plugins reach Sablier at http://sablier-sablier.sablier:10000: the chart names the Service sablier-<release>.

Create the namespace, then apply the service account, RBAC, Deployment and Service:

kubectl create namespace sablier
kubectl apply -f sablier.yaml

examples/kubernetes/sablier.yaml

# Sablier without Helm: kubectl create namespace sablier && kubectl apply -f sablier.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sablier
  namespace: sablier
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sablier
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list", "watch", "patch"] # patch sets CPU/memory limits in scale mode
  - apiGroups: ["apps"]
    resources: ["deployments/scale", "statefulsets/scale"]
    verbs: ["get", "update"] # scale up and down
  # Only to manage CloudNativePG Clusters.
  - apiGroups: ["postgresql.cnpg.io"]
    resources: ["clusters"]
    verbs: ["get", "list", "watch", "patch"] # patch toggles hibernation
  # Only to manage OT-CONTAINER-KIT Redis instances.
  - apiGroups: ["redis.redis.opstreelabs.in"]
    resources: ["redis"]
    verbs: ["get", "patch"] # patch toggles skip-reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sablier
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sablier
subjects:
  - kind: ServiceAccount
    name: sablier
    namespace: sablier
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sablier
  namespace: sablier
spec:
  selector:
    matchLabels:
      app: sablier
  template:
    metadata:
      labels:
        app: sablier
    spec:
      serviceAccountName: sablier
      containers:
        - name: sablier
          image: sablierapp/sablier:1.19.0 # x-release-please-version
          args:
            - start
            - --provider.name=kubernetes
          ports:
            - containerPort: 10000
          readinessProbe:
            httpGet:
              path: /health
              port: 10000
---
apiVersion: v1
kind: Service
metadata:
  name: sablier
  namespace: sablier
spec:
  selector:
    app: sablier
  ports:
    - port: 10000

Reverse proxy plugins reach Sablier at http://sablier.sablier:10000.

The CloudNativePG and Redis permissions are optional. Without them, or without their CRDs, Sablier skips those integrations.

Usage example

Opt a Deployment in with the sablierapp.dev/enable label, and give it a group:

examples/kubernetes/whoami.yaml

# A Deployment Sablier scales to zero when idle and back up on demand, and the
# Service your reverse proxy routes to.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  labels:
    app: whoami
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: sablierapp/mimic:v0.3.3
          args: ["-healthy=true"]
          ports:
            - containerPort: 80
          # Sablier reports the Deployment ready once its pods pass this probe.
          readinessProbe:
            exec:
              command: ["/mimic", "healthcheck"]
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  selector:
    app: whoami
  ports:
    - port: 80

Then configure your reverse proxy to call Sablier for the whoami group in front of the whoami Service. On the first request, Sablier scales the Deployment to one replica and holds the request, or shows a waiting page, until the pod is ready. When the session expires, Sablier scales it back to zero.

Sablier scales opted-in workloads to zero when it starts. A workload you deploy afterwards keeps running until its first session expires, unless you stop externally-started instances.

Provider reference

Configuration

Option	Default	Description
`--provider.kubernetes.delimiter`	`_`	The separator in instance names. Prefer `.` or `/`; `_` is kept for backward compatibility.
`--provider.kubernetes.ready-on-first-replica`	`false`	Report a workload ready as soon as one replica is ready, instead of all of them. Since v1.16.0.
`--provider.kubernetes.qps`	`5`	Client-side limit of Kubernetes API requests per second.
`--provider.kubernetes.burst`	`10`	Requests allowed above that limit in a burst.

Every --provider.* option that is not provider-specific also applies; see the CLI reference.

Workloads

Deployments and StatefulSets

Sablier scales Deployments and StatefulSets through their scale subresource:

Stop scales to 0.
Start scales to active.replicas when set. Otherwise, it uses the replica count of the instance name when above 1, or 1.

examples/kubernetes/statefulset.yaml

metadata:
  name: whoami-db
  labels:
    app: whoami-db
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami

CloudNativePG Clusters
Since v1.13.0

Sablier starts and stops CloudNativePG Cluster custom resources with the operator’s declarative hibernation:

Stop sets cnpg.io/hibernation: "on". The operator removes the cluster’s pods and keeps its PVCs.
Start sets cnpg.io/hibernation: "off", and the operator resumes the cluster.

examples/kubernetes/cnpg-cluster.yaml

# Requires the CloudNativePG operator, and rbac.cnpg=true in the Helm chart.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: whoami-pg
  labels:
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  instances: 3
  storage:
    size: 1Gi

Put the application, its database and any other dependency in the same group, so one request wakes the whole stack and inactivity hibernates all of it.

Resuming a hibernated cluster takes longer than a scale-up: the PVCs are reattached and PostgreSQL recovers. Make sure your reverse proxy timeouts allow for it.

OT-CONTAINER-KIT Redis instances
Since v1.14.0

The OT-CONTAINER-KIT redis-operator reconciles its StatefulSets back to their replica count, which would undo a scale to zero. Sablier pauses the operator around each scale:

Stop sets the redis.opstreelabs.in/skip-reconcile: "true" annotation on the Redis resource, then scales its StatefulSet to 0.
Start scales the StatefulSet back to 1, then removes the annotation so the operator resumes.

If the scale fails, Sablier clears the annotation immediately, so the operator is never left paused with pods running.

Label the Redis resource. The operator copies the labels to the StatefulSet it manages:

examples/kubernetes/redis.yaml

# Requires the OT-CONTAINER-KIT redis-operator, and rbac.otkRedis=true in the
# Helm chart. The operator copies these labels to the StatefulSet it manages.
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: Redis
metadata:
  name: whoami-redis
  labels:
    sablierapp.dev/enable: "true"
    sablierapp.dev/group: whoami
spec:
  kubernetesConfig:
    image: quay.io/opstree/redis:v8.10.1

Instance names

A reverse proxy can target one workload by name instead of a group. The whoami Deployment of the usage example is named:

deployment_default_whoami_1

The name has four parts, separated by the delimiter (_ by default):

Kind: deployment, statefulset or cnpgcluster. Target a Redis instance through its StatefulSet.
Namespace of the workload.
Name of the workload.
Replicas: the replica count Sablier scales to on start, since v1.15.0.

Prefer groups: a name ties your proxy configuration to the workload’s kind and namespace.

Labels
Next release

On Kubernetes, Sablier keys start with sablierapp.dev/ instead of sablier.:

examples/kubernetes/group.yaml

metadata:
  name: team-api
  labels:
    app: team-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: "team-a,team-b" # comma: must be an annotation

enable must be a label. Sablier finds workloads with a label selector, and selectors can’t read annotations.
Values with commas or colons must be annotations. Kubernetes doesn’t allow them in labels, so a list of groups or a time window goes in an annotation.
Any other key can be either. If a key is set as both, the annotation wins.

The label reference lists every key.

Legacy sablier.* keys

Earlier releases only read sablier.* keys. Sablier still reads them, but they are deprecated on Kubernetes: keys without a prefix belong to the user and can clash with your own. If a workload has both forms, the sablierapp.dev/ key wins.

Workload settings

Each excerpt below is the metadata of a complete manifest in examples/kubernetes.

Enable
Since v1.4.0

sablierapp.dev/enable: "true" opts the workload in. It is required, must be a label, and any other value is ignored. See the usage example.

Group
Since v1.4.0

sablierapp.dev/group puts the workload in one or more groups. A session for any of them starts it. Without it, the workload is in the default group.

A list of groups contains commas, so set it as an annotation:

examples/kubernetes/group.yaml

metadata:
  name: team-api
  labels:
    app: team-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: "team-a,team-b" # comma: must be an annotation

Learn more about groups.

Ready after
Since v1.13.0

sablierapp.dev/ready-after keeps reporting the workload as not ready for a settling delay after its pods first become ready. Use it when a pod passes its readiness probe before it can serve traffic.

examples/kubernetes/ready-after.yaml

metadata:
  name: slow-starter
  labels:
    app: slow-starter
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: slow-starter
    sablierapp.dev/ready-after: "30s"

Learn more about delaying readiness.

Ready on start
Since v1.15.0

sablierapp.dev/ready-on-start: "true" reports the workload ready as soon as Sablier sends the scale-up, without waiting for its pods.

examples/kubernetes/ready-on-start.yaml

metadata:
  name: fast-starter
  labels:
    app: fast-starter
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: fast-starter
    sablierapp.dev/ready-on-start: "true"

Learn more about marking ready on start.

Running hours
Since v1.13.0

sablierapp.dev/running-hours keeps the workload running during a daily window.
sablierapp.dev/running-days limits that window to some weekdays, since v1.15.0.

Both values contain colons or commas, so set them as annotations:

examples/kubernetes/running-hours.yaml

metadata:
  name: office-app
  labels:
    app: office-app
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: office-app
    sablierapp.dev/running-hours: "09:00-18:00" # colon: must be an annotation
    sablierapp.dev/running-days: "Mon,Tue,Wed,Thu,Fri"

The window uses Sablier’s local time. With Helm, set the time zone with the TZ environment variable:

examples/kubernetes/values.yaml

# sablierapp.dev/running-hours windows use Sablier's local time.
extraEnv:
  - name: TZ
    value: Europe/Paris

Learn more about running hours.

Anti-affinity
Since v1.15.0

sablierapp.dev/anti-affinity lists groups the workload backs off from: Sablier keeps it idle while any of them has an active session. Set a list of groups as an annotation.

examples/kubernetes/anti-affinity.yaml

metadata:
  name: batch-worker
  labels:
    app: batch-worker
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: batch
    sablierapp.dev/anti-affinity: "streaming"

Learn more about anti-affinity.

Scale instead of stop
Since v1.13.0

Keep the workload running when idle, optionally throttled, instead of scaling it to zero:

idle.replicas set to 1 or more keeps that many replicas when idle.
idle.cpu and idle.memory set the CPU and memory limits while idle.
active.replicas, active.cpu and active.memory restore them on the next session.

CPU and memory use resource quantities, such as 100m, 2, 64Mi or 1Gi.

examples/kubernetes/scale-mode.yaml

metadata:
  name: warm-api
  labels:
    app: warm-api
    sablierapp.dev/enable: "true" # must be a label: discovery uses a label selector
  annotations:
    sablierapp.dev/group: warm-api
    sablierapp.dev/idle.replicas: "1"
    sablierapp.dev/idle.cpu: "100m"
    sablierapp.dev/idle.memory: "64Mi"
    sablierapp.dev/active.cpu: "2"
    sablierapp.dev/active.memory: "512Mi"

Changing limits triggers a rolling update: new pods replace the old ones, so the service stays available. Sablier only changes the limits of the pod template’s first container, never its requests, so make sure your nodes can schedule the active limits.

Learn more about scaling instead of stopping.

Readiness

Sablier reports a Deployment or StatefulSet:

ready when it has at least one desired replica and all of them are ready. With --provider.kubernetes.ready-on-first-replica, one ready replica is enough.
stopped when it is scaled to 0, even while pods are still terminating.
starting otherwise.

A replica is ready when its pod passes its readiness probe, so give your containers one.

A CloudNativePG Cluster is stopped when its cnpg.io/hibernation annotation is "on", ready when status.readyInstances reaches spec.instances, and starting otherwise.

Limitations

Sablier must run as a pod in the cluster it manages: it only uses in-cluster credentials.
Block I/O throttling is not available on Kubernetes.
CPU and memory throttling applies to the first container of the pod template only, and restarts the pods.
Without the Redis permissions, Sablier logs a warning and scales the StatefulSet anyway; the operator then scales it back up.

Examples

examples/kubernetes: every manifest on this page, with instructions to run them.
Code-Server + Traefik + Kubernetes Ingress: a complete setup behind Traefik.

Docker Swarm Podman

KubernetesSince v1.0.0

Introduction

Install Sablier

Usage example

Provider reference

Configuration

Workloads

Deployments and StatefulSets

CloudNativePG ClustersSince v1.13.0

OT-CONTAINER-KIT Redis instancesSince v1.14.0

Instance names

LabelsNext release

Workload settings

EnableSince v1.4.0

GroupSince v1.4.0

Ready afterSince v1.13.0

Ready on startSince v1.15.0

Running hoursSince v1.13.0

Anti-affinitySince v1.15.0

Scale instead of stopSince v1.13.0

Readiness

Limitations

Examples

Kubernetes
Since v1.0.0

CloudNativePG Clusters
Since v1.13.0

OT-CONTAINER-KIT Redis instances
Since v1.14.0

Labels
Next release

Enable
Since v1.4.0

Group
Since v1.4.0

Ready after
Since v1.13.0

Ready on start
Since v1.15.0

Running hours
Since v1.13.0

Anti-affinity
Since v1.15.0

Scale instead of stop
Since v1.13.0