AWS ECS

AWS ECS
Next release

Introduction

The AWS ECS provider sets the desired count of ECS services to zero when their session expires, and back up when a request needs them. ECS then stops or starts the tasks. Sablier manages the services of one cluster and calls the ECS API with the credentials of the AWS SDK.

Install Sablier

Run Sablier as an ECS service in the cluster it manages. Create the cluster with a default Service Connect namespace, then deploy the CloudFormation template:

aws ecs create-cluster --cluster-name sablier --service-connect-defaults namespace=sablier
aws cloudformation deploy --stack-name sablier --template-file sablier.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides Subnets=subnet-aaaa,subnet-bbbb SecurityGroup=sg-cccc

The template runs Sablier on Fargate. Its task definition selects the ECS provider and the cluster:

examples/ecs/sablier.yaml

          Command:
            - start
            - --provider.name=ecs
            - !Sub --provider.ecs.cluster=${Cluster}
            - !Sub --provider.ecs.region=${AWS::Region}

Sablier gets its credentials from the task role. The role’s IAM policy allows the four ECS API actions Sablier calls:

examples/ecs/sablier.yaml

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: sablier
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              # Check the cluster on startup.
              - Effect: Allow
                Action: ecs:DescribeClusters
                Resource: !Sub arn:${AWS::Partition}:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${Cluster}
              # ListServices has no resource type: scope it with a condition.
              - Effect: Allow
                Action: ecs:ListServices
                Resource: "*"
                Condition:
                  ArnEquals:
                    ecs:cluster: !Sub arn:${AWS::Partition}:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${Cluster}
              # Read the services and their tags.
              - Effect: Allow
                Action: ecs:DescribeServices
                Resource: !Sub arn:${AWS::Partition}:ecs:${AWS::Region}:${AWS::AccountId}:service/${Cluster}/*
              # Change the desired count of opted-in services only.
              - Effect: Allow
                Action: ecs:UpdateService
                Resource: !Sub arn:${AWS::Partition}:ecs:${AWS::Region}:${AWS::AccountId}:service/${Cluster}/*
                Condition:
                  StringEquals:
                    aws:ResourceTag/sablier.enable: "true"

ecs:ListServices needs "Resource": "*". The action has no resource type, so scope it to the cluster with the ecs:cluster condition key.
ecs:UpdateService can require the sablier.enable tag. The aws:ResourceTag condition stops Sablier from scaling services that did not opt in.

Reverse proxy plugins reach Sablier at http://sablier:10000. The service registers that name in the Service Connect namespace, so run your reverse proxy as an ECS service in the same namespace:

examples/ecs/sablier.yaml

      # Reverse proxies in the namespace reach Sablier at http://sablier:10000.
      ServiceConnectConfiguration:
        Enabled: true
        Services:
          - PortName: http
            DiscoveryName: sablier
            ClientAliases:
              - DnsName: sablier
                Port: 10000

Run Sablier outside ECS

Sablier reads credentials the way every AWS SDK does, and uses the first source it finds:

The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
A web identity token, such as EKS IAM roles for service accounts.
The shared config and credentials files, with AWS_PROFILE.
The container credentials of an ECS task role or EKS Pod Identity.
The instance role of an EC2 instance.

Grant those credentials the policy above. Set the region with --provider.ecs.region or AWS_REGION. Sablier does not read it from the EC2 instance metadata.

Usage example

Opt an ECS service in with the sablier.enable tag, and give it a group:

examples/ecs/whoami.yaml

# An ECS service Sablier scales to zero when idle and back up on demand.
# Reverse proxies in the Service Connect namespace reach it at http://whoami.
AWSTemplateFormatVersion: "2010-09-09"
Description: whoami, a service managed by Sablier

Parameters:
  Cluster:
    Type: String
    Default: sablier
  Subnets:
    Type: List<AWS::EC2::Subnet::Id>
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Must allow inbound TCP 80 from your reverse proxy

Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: whoami
      RequiresCompatibilities: [FARGATE]
      NetworkMode: awsvpc
      Cpu: "256"
      Memory: "512"
      ContainerDefinitions:
        - Name: whoami
          Image: sablierapp/mimic:v0.3.3
          Command: ["-healthy=true"]
          Essential: true
          PortMappings:
            - Name: http
              ContainerPort: 80
              AppProtocol: http
          HealthCheck:
            Command: ["CMD", "/mimic", "healthcheck"]
            Interval: 5
            Timeout: 2
            Retries: 3

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: whoami
      Cluster: !Ref Cluster
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: 0
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref Subnets
          SecurityGroups: [!Ref SecurityGroup]
          AssignPublicIp: ENABLED
      ServiceConnectConfiguration:
        Enabled: true
        Services:
          - PortName: http
            DiscoveryName: whoami
            ClientAliases:
              - DnsName: whoami
                Port: 80
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: whoami

aws cloudformation deploy --stack-name whoami --template-file whoami.yaml \
  --parameter-overrides Subnets=subnet-aaaa,subnet-bbbb SecurityGroup=sg-cccc

Then configure your reverse proxy to call Sablier for the whoami group in front of http://whoami. On the first request, Sablier sets the desired count to 1 and holds the request, or shows a waiting page, until the task is running. When the session expires, Sablier sets it back to 0.

Sablier sets the desired count of opted-in services to zero when it starts. A service you deploy afterwards keeps running until its first session expires, unless you stop externally-started instances.

Provider reference

Configuration

Option	Default	Description
`--provider.ecs.cluster`	`default`	The name or ARN of the cluster that holds the services. Sablier fails to start if the cluster is missing or not `ACTIVE`.
`--provider.ecs.region`		The AWS region of the cluster. When empty, Sablier reads `AWS_REGION` or the shared config profile.
`--provider.ecs.endpoint`		Replaces the ECS API endpoint URL, for example to target an emulator such as LocalStack.

Every --provider.* option that is not provider-specific also applies; see the CLI reference.

Workloads

Sablier scales ECS services that use the REPLICA scheduling strategy. It calls UpdateService to change the desired count:

Stop sets the desired count to 0, or to idle.replicas when set.
Start sets it to active.replicas when set, or 1.

ECS then starts or stops tasks from the service’s task definition. Sablier never changes the task definition.

Instance names

A reverse proxy can target one service by name instead of a group. The whoami service of the usage example is named:

whoami

The instance name is the service name, without the cluster. Service names are only unique within a cluster, so one Sablier manages one cluster. A request can also name the service by its ARN: it resolves to the service name and shares its session.

Prefer groups: a name ties your proxy configuration to the service.

Labels

On ECS, Sablier settings are resource tags on the service. The keys and values are those of the label reference:

examples/ecs/group.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: team-api
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: team-a team-b # space separated: tag values cannot contain commas

Tag the service. Sablier reads the tags of the service only, not those of its task definition or tasks.
Separate list values with spaces. Tag values can’t contain commas, so lists of groups and weekdays are space-separated.
Keys and values are case-sensitive. sablier.enable must be exactly true.

To tag a service you did not create with CloudFormation, use the AWS CLI:

aws ecs tag-resource --resource-arn arn:aws:ecs:eu-west-1:123456789012:service/sablier/whoami \
  --tags key=sablier.enable,value=true key=sablier.group,value=whoami

Workload settings

Each excerpt below is the service of a complete template in examples/ecs.

Enable
Next release

The sablier.enable tag set to true opts the service in. It is required, and any other value is ignored. See the usage example.

Group
Next release

sablier.group puts the service in one or more groups. A session for any of them starts it. Without it, the service is in the default group.

Separate the groups with spaces, because a tag value can’t contain a comma:

examples/ecs/group.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: team-api
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: team-a team-b # space separated: tag values cannot contain commas

Learn more about groups.

Ready after
Next release

sablier.ready-after keeps reporting the service as not ready for a settling delay after its tasks first reach the RUNNING state. Use it when your application needs time to serve traffic after its task starts.

examples/ecs/ready-after.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: slow-starter
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: slow-starter
        - Key: sablier.ready-after
          Value: 30s

Learn more about delaying readiness.

Ready on start
Next release

sablier.ready-on-start set to true reports the service ready as soon as Sablier raises its desired count, without waiting for its tasks.

examples/ecs/ready-on-start.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: fast-starter
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: fast-starter
        - Key: sablier.ready-on-start
          Value: "true"

Learn more about marking ready on start.

Running hours
Next release

sablier.running-hours keeps the service running during a daily window.
sablier.running-days limits that window to some weekdays. Separate the days with spaces.

examples/ecs/running-hours.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: office-app
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: office-app
        - Key: sablier.running-hours
          Value: "09:00-18:00"
        - Key: sablier.running-days
          Value: Mon Tue Wed Thu Fri # space separated

The window uses Sablier’s local time. Set the time zone with the TZ environment variable of Sablier’s container:

examples/ecs/sablier.yaml

          # sablier.running-hours windows use Sablier's local time.
          Environment:
            - Name: TZ
              Value: Europe/Paris

Learn more about running hours.

Anti-affinity
Next release

sablier.anti-affinity lists groups the service backs off from: Sablier keeps it idle while any of them has an active session. Separate the groups with spaces.

examples/ecs/anti-affinity.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: batch-worker
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: batch
        - Key: sablier.anti-affinity
          Value: streaming gaming # space separated

Learn more about anti-affinity.

Scale instead of stop
Next release

Keep the service running when idle instead of scaling it to zero:

sablier.idle.replicas set to 1 or more keeps that many tasks when idle.
sablier.active.replicas sets the desired count on the next session.

examples/ecs/scale-mode.yaml

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: warm-api
      Tags:
        - Key: sablier.enable
          Value: "true"
        - Key: sablier.group
          Value: warm-api
        - Key: sablier.idle.replicas
          Value: "1"
        - Key: sablier.active.replicas
          Value: "3"

On ECS, only the task count changes. The CPU and memory of a task belong to its task definition, so Sablier ignores the CPU and memory tags and logs a warning.

Learn more about scaling instead of stopping.

Readiness

Sablier reads the service and its PRIMARY deployment, and reports it:

stopped when its desired count is 0, even while tasks are still stopping.
error when the rollout state of the primary deployment is FAILED. Sablier shows the reason ECS gives.
starting while fewer tasks are running than desired, or while the primary deployment is IN_PROGRESS.
ready otherwise.

ECS counts a task as running once it reaches the RUNNING state. If your application needs more time after that, add sablier.ready-after.

A rollout fails when the deployment circuit breaker is on and the service can’t reach a steady state. Sablier reports the error until a new deployment replaces the failed one.

Limitations

One Sablier manages one cluster. Run one Sablier per cluster.
Sablier ignores services that use the DAEMON scheduling strategy: they have no desired count to scale.
CPU, memory and block I/O throttling are not possible on ECS: they belong to the task definition.
ECS publishes service changes to EventBridge only. Sablier polls the cluster every 10 seconds to detect services started, stopped, tagged or removed outside Sablier.

Examples

examples/ecs: every template on this page, with instructions to deploy them.

Systemd Nomad

AWS ECSNext release

Introduction

Install Sablier

Usage example

Provider reference

Configuration

Workloads

Instance names

Labels

Workload settings

EnableNext release

GroupNext release

Ready afterNext release

Ready on startNext release

Running hoursNext release

Anti-affinityNext release

Scale instead of stopNext release

Readiness

Limitations

Examples

AWS ECS
Next release

Enable
Next release

Group
Next release

Ready after
Next release

Ready on start
Next release

Running hours
Next release

Anti-affinity
Next release

Scale instead of stop
Next release