Proxmox LXC

Since v1.12.0

Introduction

The Proxmox LXC provider starts and stops LXC containers on a Proxmox VE node or cluster. You opt a container in with tags. Sablier runs outside Proxmox VE and calls the Proxmox VE API with an API token.

Install Sablier

Sablier needs an API token with two privileges on the containers it manages:

• VM.Audit to list the containers and read their status, configuration and IP addresses.
• VM.PowerMgmt to start and stop them.

Create a role with them, a dedicated user and its token. Run pveum as root on any node:

# A role with the only two privileges Sablier needs.
pveum role add Sablier --privs "VM.Audit VM.PowerMgmt"

# A dedicated user with that role on every guest.
pveum user add sablier@pve --comment "Sablier"
pveum acl modify /vms --users sablier@pve --roles Sablier

# The token sablier@pve!sablier, with the same privileges as its user.
pveum user token add sablier@pve sablier --privsep 0

pveum user token add prints the token secret once: store it. Sablier only sees the containers where the token has VM.Audit. To limit it to some containers, grant the role on /vms/<vmid> or on a pool instead of /vms; see objects and paths.

Then configure Sablier with the API URL of any node and the token ID:

provider:
  name: proxmox_lxc
  proxmox-lxc:
    url: "https://pve1.example.com:8006/api2/json"
    token-id: "sablier@pve!sablier"

Pass the secret in the SABLIER_PROVIDER_PROXMOX_LXC_TOKEN_SECRET environment variable, so it stays out of the file.

services:
  sablier:
    image: sablierapp/sablier:1.19.0 # x-release-please-version
    command:
      - start
      - --configFile=/etc/sablier/sablier.yaml
    environment:
      SABLIER_PROVIDER_PROXMOX_LXC_TOKEN_SECRET: ${PROXMOX_TOKEN_SECRET:?set PROXMOX_TOKEN_SECRET to the API token secret}
      # The image ships no CA certificates: trust the cluster CA.
      SSL_CERT_FILE: /etc/sablier/pve-root-ca.pem
    volumes:
      - ./sablier.yaml:/etc/sablier/sablier.yaml:ro
      # Copied from /etc/pve/pve-root-ca.pem on any node.
      - ./pve-root-ca.pem:/etc/sablier/pve-root-ca.pem:ro
    ports:
      - "10000:10000"

scp root@pve1.example.com:/etc/pve/pve-root-ca.pem .
PROXMOX_TOKEN_SECRET=<token secret> docker compose up -d

export SABLIER_PROVIDER_PROXMOX_LXC_TOKEN_SECRET=<token secret>
sablier start --configFile sablier.yaml

Sablier calls the API at startup and exits if it can’t connect. Reverse proxy plugins reach Sablier at http://<sablier host>:10000, or at http://sablier:10000 from the same Compose project.

Usage example

Opt a container in with the sablier tag, and give it a group with a sablier-group-<name> tag. This pct create creates the whoami container, stopped:

pct create "$VMID" "$TEMPLATE_STORAGE:vztmpl/$template" \
  --hostname whoami \
  --tags "sablier;sablier-group-whoami" \
  --net0 "name=eth0,bridge=$BRIDGE,ip=dhcp" \
  --rootfs "$ROOTFS_STORAGE:4" \
  --unprivileged 1

Then configure your reverse proxy to call Sablier for the whoami group in front of the container’s address. On the first request, Sablier starts the container and holds the request, or shows a waiting page, until the container has an IP address. When the session expires, Sablier stops it.

Provider reference

Configuration

Every --provider.* option that is not provider-specific also applies; see the CLI reference. With tracing on, each Proxmox VE API call is a child span, with or without tls-insecure.

Workloads

Sablier manages LXC containers on every node of the cluster:

• Start calls the API’s status/start, like pct start, and follows the start task until it ends.
• Stop calls the API’s status/stop, like pct stop, and waits for the task.

A cluster needs no extra setting. Sablier lists the nodes through the node in url, and pveproxy forwards each call to the node that hosts the container. If a node doesn’t answer, Sablier logs a warning and skips its containers.

Instance names

A reverse proxy can target one container by name instead of a group. The container of the usage example is named:

whoami

The name is the container’s hostname, set with --hostname.

• Keep hostnames unique across the cluster. If two tagged containers share a hostname, Sablier logs a warning and ignores that hostname until you rename one; request them by ID meanwhile. A tagged container always wins over an untagged one with the same hostname.
• The CT ID works too. A request for 200 or pve1/200 resolves to the container’s hostname and shares its session.

Prefer groups: a name ties your proxy configuration to one container.

Labels

Proxmox VE has no key/value labels, so Sablier reads the container’s tags:

pvesh set "/nodes/$node/lxc/$vmid/config" --tags "sablier;sablier-group-whoami;sablier-group-tools"

• sablier opts the container in. It stands for sablier.enable=true.
• sablier-group-<name> adds the container to a group. It stands for sablier.group, with one tag per group.
• Tags must match exactly. Sablier ignores Sablier, sablier=true or sablier.group=whoami.
• No other setting has a tag. See the limitations.

Set tags in the web interface with the pencil icon next to the container’s tags, or with pct set or pvesh. Sablier rereads them every 30 seconds, without a restart.

Workload settings

Each excerpt below comes from a script in examples/proxmox-lxc.

Enable

Since v1.12.0

The sablier tag opts the container in. It is required. See the usage example for a new container. For an existing one, append the tag to its current tags:

# pct set --tags replaces the whole list, so append to the current tags.
tags="$(pct config "$vmid" | awk '$1 == "tags:" {print $2}')"
pct set "$vmid" --tags "${tags:+$tags;}sablier"

--tags replaces the whole tag list, so keep the tags the container already has.

Group

Since v1.12.0

A sablier-group-<name> tag puts the container in the <name> group. A session for any of its groups starts it. Without one, the container is in the default group.

Add one tag per group, since v1.13.0. Earlier releases only read the first one.

pvesh set "/nodes/$node/lxc/$vmid/config" --tags "sablier;sablier-group-whoami;sablier-group-tools"

Remove a group tag, or the sablier tag, and Sablier drops the container from that group at its next scan.

Learn more about groups.

Readiness

Sablier reports a container:

• ready when Proxmox VE reports it running and one of its interfaces other than lo has an IPv4 or IPv6 address.
• starting while the start task runs, or while the container runs without an IP address.
• stopped when Proxmox VE reports it stopped.
• error for 30 seconds after a failed start task, with the task’s exit status. On the next check after that, Sablier starts the container again.

Give each container a network interface that gets an address: without one, it never reports ready. If Sablier can’t read the interfaces, it reports the container ready.

Sablier doesn’t check the service inside the container. The waiting page can clear before your application accepts connections.

Limitations

• Sablier manages LXC containers only, not virtual machines.
• Tags can’t carry values, so these settings are not available: Ready after, Ready on start, Running hours, Anti-affinity, Scale instead of stop, and CPU, memory or block I/O throttling.
• Proxmox VE has no event stream, so Sablier polls the API every 10 seconds. It notices a container started or stopped outside Sablier within that delay, and stops an externally-started one when it notices it.

Examples

• examples/proxmox-lxc: every script and file on this page, with instructions to run them.