Nomad

Next release

Introduction

The Nomad provider scales task groups to a count of zero when their session expires, and back up when a request needs them. Each task group of a job is a separate instance. Sablier calls the Nomad HTTP API with an ACL token, and follows job changes on its event stream.

Install Sablier

Sablier runs wherever it can reach the Nomad API: as a Nomad job, or next to the cluster.

When your cluster has ACLs enabled, give Sablier a token first. Write an ACL policy for the namespace of your jobs:

# ACL policy for the Sablier token:
#   nomad acl policy apply -description "Sablier" sablier sablier-policy.hcl
#   nomad acl token create -name sablier -policy sablier -type client
namespace "default" {
  # list-jobs and read-job: discover task groups, read allocations, watch job events.
  # scale-job: change the count of a task group.
  # submit-job: register the job with a new count while a deployment blocks scaling.
  capabilities = ["list-jobs", "read-job", "scale-job", "submit-job"]
}

Apply it and create a client token:

nomad acl policy apply -description "Sablier" sablier sablier-policy.hcl
nomad acl token create -name sablier -policy sablier -type client

Sablier uses the Secret ID of the token.

nomad var put nomad/jobs/sablier token=<secret ID>
nomad job run sablier.nomad.hcl

# Sablier as a Nomad job, for a Nomad cluster with the Docker driver and ACLs.
# Store the ACL token in a Nomad variable first:
#   nomad var put nomad/jobs/sablier token=<secret ID>
#   nomad job run sablier.nomad.hcl
job "sablier" {
  group "sablier" {
    network {
      port "http" {
        static = 10000
      }
    }

    service {
      name     = "sablier"
      port     = "http"
      provider = "nomad"

      check {
        type     = "http"
        path     = "/health"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "sablier" {
      driver = "docker"

      config {
        image = "sablierapp/sablier:1.19.0" # x-release-please-version
        args  = ["start", "--provider.name=nomad"]
        ports = ["http"]
      }

      env {
        NOMAD_ADDR      = "http://${attr.unique.network.ip-address}:4646"
        NOMAD_NAMESPACE = "default"
        # sablier.running-hours windows use this time zone.
        TZ = "Europe/Paris"
      }

      # The task reads its own variable, so the token never appears in the job.
      template {
        data        = "NOMAD_TOKEN={{ with nomadVar \"nomad/jobs/sablier\" }}{{ .token }}{{ end }}"
        destination = "secrets/env"
        env         = true
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}

    command:
      - start
      - --provider.name=nomad
      - --provider.nomad.address=http://nomad:4646
      # The development agent has no ACLs. With ACLs, add the token:
      # - --provider.nomad.token=<secret ID>
      - --logging.level=debug
      - --sessions.default-duration=1m
      - --sessions.expiration-interval=10s
      - --strategy.blocking.default-timeout=2m

Usage example

Opt a task group in with sablier.enable in its meta block, and register it with a count of zero:

job "whoami" {
  datacenters = ["dc1"]
  type        = "service"

  group "web" {
    # Sablier scales the task group up on request and back to 0 on expiry.
    count = 0

    # A map, because HCL does not accept quoted names in a meta block.
    meta = {
      "sablier.enable" = "true"
      "sablier.group"  = "demo"
    }

    network {
      port "http" {
        static = 8080
      }
    }

    # Sablier reports the instance as ready when the deployment marks the allocation healthy.
    update {
      health_check     = "checks"
      min_healthy_time = "5s"
    }

    service {
      name     = "whoami"
      port     = "http"
      provider = "nomad"

      check {
        type     = "http"
        path     = "/"
        interval = "2s"
        timeout  = "1s"
      }
    }

    task "server" {
      # The busybox web server is part of the Nomad image.
      driver = "raw_exec"

      config {
        command = "/bin/busybox"
        args    = ["httpd", "-f", "-p", "${NOMAD_PORT_http}", "-h", "${NOMAD_TASK_DIR}/www"]
      }

      template {
        destination = "local/www/index.html"
        data        = <<-EOT
          Hello from the Nomad task group {{ env "NOMAD_JOB_NAME" }}/{{ env "NOMAD_GROUP_NAME" }}.
          Allocation {{ env "NOMAD_ALLOC_ID" }} was started on demand by Sablier.
        EOT
      }

      resources {
        cpu    = 50
        memory = 32
      }
    }
  }
}

Then configure your reverse proxy to call Sablier for the demo group in front of the whoami service. On the first request, Sablier scales the task group to one allocation and holds the request, or shows a waiting page, until the allocation is healthy. When the session expires, Sablier scales it back to zero.

The demo runs the busybox web server of the Nomad image with the raw_exec driver, so it works on a development agent. Use your own driver and image.

With Traefik

A task group at zero has no allocation, so Nomad removes its service. Traefik then drops its router, and requests never reach the Sablier middleware. Keep a route:

• With the Traefik Nomad provider, set allowEmptyServices to true. Traefik then keeps the router of a service without instances.
• With Consul Catalog or Consul Connect, add a fallback router. Give it the lowest priority, in the file provider:

  routers:
    whoami-wake:
      rule: Host(`whoami.example.com`)
      priority: 1 # lower than the router of the running task group
      service: noop@internal
      middlewares:
        - sablier-whoami
      entryPoints:
        - https
      tls: {}

While the task group runs, its own router has a higher priority. Add the Sablier middleware to it too, so requests renew the session.

Set sablier.ready-after to at least the refresh interval of the Traefik provider, 15 seconds by default. Otherwise the waiting page can send the user on before Traefik knows the new allocation.

Provider reference

Configuration

An empty option falls back to the environment variable of the nomad CLI. A Sablier that runs with NOMAD_ADDR and NOMAD_TOKEN needs no option.

With tracing enabled, every Nomad API call is a child span. The event stream is one span for the whole connection.

Every --provider.* option that is not provider-specific also applies; see the CLI reference.

Workloads

Task groups

Sablier scales each task group of a job on its own, with the Scale Task Group API:

• Stop sets the count to 0, or to idle.replicas when set.
• Start sets the count to active.replicas when set, or to 1.

Start never restores the count the task group had before. Set sablier.active.replicas to run more than one allocation.

• Sablier never starts a stopped job. A job stopped with nomad job stop stays stopped, and requests for it get an error. Run nomad job start to start it again.
• An active deployment blocks scaling. Nomad rejects a scale request while a deployment of the job is active. Sablier then registers the job again with the new count, which needs the submit-job capability.

Instance names

A reverse proxy can target one task group by name instead of a group. The web task group of the whoami job in the usage example is named:

whoami/web

The name has two parts, separated by /:

• Job ID of the job.
• Name of the task group.

The namespace is not part of the name: one Sablier manages one namespace. Prefer groups: a name ties your proxy configuration to the structure of the job.

Labels

On Nomad, Sablier settings are keys of a meta block, on the job or on a task group. In this job, the job meta enables both task groups and puts them in the shop group:

  meta = {
    "sablier.enable" = "true"
    "sablier.group"  = "shop"
  }

The meta of the admin task group moves it to the back-office group:

    meta = {
      "sablier.group" = "back-office"
    }

• Use the map syntax meta = { ... }. HCL accepts keys with dots only as quoted map keys, see meta keys with dots.
• Job meta applies to every task group of the job. Each task group is still a separate instance.
• Task group meta wins over job meta. Sablier merges them key by key, the way Nomad does for tasks.
• Task meta is ignored. Sablier reads the job and the task group only.

The keys have the same names as in the label reference, which lists every key.

Workload settings

Each excerpt below is the meta of a task group in a complete job in examples/nomad.

Enable

Next release

"sablier.enable" = "true" opts the task group in. It is required, and any other value is ignored. Set it in the job meta to enable every task group of the job. See the usage example.

Group

Next release

sablier.group puts the task group in one or more groups, separated by commas. A session for any of them starts it. Without it, the task group is in the default group.

    meta = {
      "sablier.enable" = "true"
      "sablier.group"  = "team-a,team-b"
    }

Learn more about groups.

Ready after

Next release

sablier.ready-after keeps reporting the task group as not ready for a settling delay after its allocations first become healthy. Use it when a service check passes before the task can serve traffic.

    meta = {
      "sablier.enable"      = "true"
      "sablier.group"       = "slow-starter"
      "sablier.ready-after" = "30s"
    }

Learn more about delaying readiness.

Ready on start

Next release

"sablier.ready-on-start" = "true" reports the task group ready as soon as Sablier sends the scale request, without waiting for its allocations.

    meta = {
      "sablier.enable"         = "true"
      "sablier.group"          = "fast-starter"
      "sablier.ready-on-start" = "true"
    }

Learn more about marking ready on start.

Running hours

Next release

• sablier.running-hours keeps the task group running during a daily window.
• sablier.running-days limits that window to some weekdays.

    meta = {
      "sablier.enable"        = "true"
      "sablier.group"         = "office-app"
      "sablier.running-hours" = "09:00-18:00"
      "sablier.running-days"  = "Mon,Tue,Wed,Thu,Fri"
    }

The window uses Sablier’s local time. In the Sablier job, set the time zone with the TZ environment variable:

      env {
        NOMAD_ADDR      = "http://${attr.unique.network.ip-address}:4646"
        NOMAD_NAMESPACE = "default"
        # sablier.running-hours windows use this time zone.
        TZ = "Europe/Paris"
      }

Learn more about running hours.

Anti-affinity

Next release

sablier.anti-affinity lists groups the task group backs off from: Sablier keeps it idle while any of them has an active session.

    meta = {
      "sablier.enable"        = "true"
      "sablier.group"         = "batch"
      "sablier.anti-affinity" = "streaming"
    }

Learn more about anti-affinity.

Scale instead of stop

Next release

Keep the task group running when idle instead of scaling it to zero:

• sablier.idle.replicas set to 1 or more is the count when the session expires.
• sablier.active.replicas is the count on the next session, 1 by default.

    meta = {
      "sablier.enable"          = "true"
      "sablier.group"           = "warm-api"
      "sablier.idle.replicas"   = "1"
      "sablier.active.replicas" = "2"
    }

Nomad supports the counts only. CPU and memory limits belong to the task resources block, so Sablier ignores idle.cpu and idle.memory, and their active counterparts.

Learn more about scaling instead of stopping.

Readiness

Sablier reports a task group:

• stopped when its count is 0.
• ready when as many allocations as the count are running and healthy.
• starting while an allocation is pending, or running but not healthy yet.
• completed when every allocation of a batch job task group has completed.
• error when only failed or lost allocations are left, or when the job was stopped with nomad job stop.

An allocation is healthy when its deployment marks it healthy. With the default health_check, its tasks must run and its service checks must pass for min_healthy_time. Give your services a check.

An allocation without deployment health, for example in a batch job, is ready as soon as it runs.

Limitations

• One Sablier manages one Nomad namespace.
• Sablier can’t scale system and sysbatch jobs: they have no count.
• Sablier skips periodic and parameterized parent jobs, which never run themselves.
• A task group with a scaling block must allow min = 0. Otherwise Nomad rejects the scale to zero.
• CPU and memory throttling are not available on Nomad: Sablier only changes the count.
• Block I/O throttling is not available on Nomad.

Examples

• examples/nomad: every job on this page, with a Nomad development agent and Sablier in Docker Compose, and instructions to run them.