Destroy Stale Preview Environments Automatically

TL;DR

Destroy stale previews with a scheduled reconciler that lists open pull requests, inventories deployed environments, checks TTL and ownership metadata, and deletes only expired orphans. Start in dry-run mode, use an allow-listed resource prefix, log every decision, and retain close-event teardown as the normal cleanup path.

Key Takeaways

• Treat cleanup as reconciliation between desired pull request state and actual deployed resources.
• Use immutable pull request numbers and explicit labels instead of parsing branch names.
• Require both orphan status and an expired grace period before destructive cleanup.
• Run the cleanup script in dry-run mode by default and verify its candidate list before enabling deletion.
• Make deletion idempotent so retries and already-removed resources succeed safely.
• Record every decision and deletion in a machine-readable audit artifact.
• Keep close-event teardown as the fast path and scheduled cleanup as the recovery path.

To destroy stale preview environments automatically, run a scheduled reconciler that compares deployed preview IDs with currently open pull requests, applies a grace period, and removes only resources that are both orphaned and expired. This recovers from missed webhooks, deleted workflows, failed credentials, and manual deployments without risking active review environments.

This tutorial adds that safety net to the lifecycle described in the complete guide to ephemeral test environments. You will build a GitHub Actions workflow and a defensive Bash reconciler for Docker Compose previews on a shared Linux host.

The central rule is simple: never delete merely because an environment looks old. First prove that your automation owns it, prove that its pull request is no longer open, and prove that its grace period has expired.

What You Will Build

You will create an automatic cleanup system that:

• inventories Compose projects named preview-<PR_NUMBER> on a dedicated host;
• queries GitHub for the authoritative set of open pull request numbers;
• reads creation and expiry metadata from container labels;
• selects only owned, orphaned, expired environments;
• supports dry-run and destructive modes with the same code path;
• deletes Compose resources and environment directories idempotently;
• uploads a JSON Lines audit report after every scheduled run.

The example uses a single Docker host because its boundaries are easy to see. The same reconciliation design applies to Kubernetes namespaces, cloud stacks, and managed preview platforms. Replace only the inventory and deletion adapters.

Prerequisites

Use these prerequisites for the runnable implementation:

• A GitHub repository with Actions enabled.
• A dedicated Ubuntu 24.04 LTS preview host with Docker Engine 27 or newer and Docker Compose v2.
• Preview projects named preview-<number> and stored under /srv/previews/pr-<number>.
• Node.js 22 LTS only if you run the optional smoke checks from the related deployment tutorial.
• A restricted SSH deployment account that can operate Docker on the preview host.
• Repository secrets PREVIEW_HOST, PREVIEW_USER, and PREVIEW_SSH_KEY.

Check your tools locally and on the host:

gh --version
ssh -V
ssh previewdeploy@preview.example.com 'docker version && docker compose version'

The GitHub-hosted runner already provides GitHub CLI. The workflow will use its short-lived GITHUB_TOKEN with pull-requests: read; it does not need a personal access token. Confirm that existing preview deployments follow the create a preview environment per pull request tutorial, especially its numeric identity and idempotent close workflow.

Step 1: Add Machine-Readable Lifecycle Labels

Put ownership and lifecycle data on the application container. Add these labels to deploy/preview.compose.yaml in the workflow that creates each preview:

services:
  app:
    image: ${IMAGE_REF:?IMAGE_REF is required}
    labels:
      qajobfit.preview.managed: 'true'
      qajobfit.preview.repository: ${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required}
      qajobfit.preview.pr: ${PR_NUMBER:?PR_NUMBER is required}
      qajobfit.preview.created-at: ${CREATED_AT:?CREATED_AT is required}
      qajobfit.preview.expires-at: ${EXPIRES_AT:?EXPIRES_AT is required}
      qajobfit.preview.commit: ${GITHUB_SHA:?GITHUB_SHA is required}

Generate timestamps in the trusted deployment workflow, not from pull request code. GNU date is available on Ubuntu runners:

CREATED_AT=$(date -u +%s)
EXPIRES_AT=$(date -u -d '+7 days' +%s)
printf 'CREATED_AT=%s\nEXPIRES_AT=%s\n' \
  "$CREATED_AT" "$EXPIRES_AT" >> "$GITHUB_ENV"

Unix seconds avoid locale parsing. Redeployment should preserve the original creation time if you want inactivity-independent TTL, or refresh expiry only after a trusted activity signal. This tutorial deletes closed-PR orphans after expiry, so an open PR is protected regardless of age.

Verify Step 1: deploy a disposable preview, then run:

docker inspect preview-123-app-1 \
  --format '{{json .Config.Labels}}' | jq .

Expect managed to equal true, repository to match OWNER/REPO, pr to be numeric, and both timestamps to be integer strings. Do not continue if ownership or expiry metadata is absent.

Step 2: Build the Authoritative Open-PR Snapshot

The cleanup runner should query GitHub before it contacts the host. Add scripts/preview-cleanup/fetch-open-prs.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required}"
: "${OUTPUT_FILE:?OUTPUT_FILE is required}"

tmp_file=$(mktemp)
trap 'rm -f "$tmp_file"' EXIT

gh api --paginate \
  -H 'Accept: application/vnd.github+json' \
  -H 'X-GitHub-Api-Version: 2022-11-28' \
  "repos/${GITHUB_REPOSITORY}/pulls?state=open&per_page=100" \
  --jq '.[].number' | sort -n -u > "$tmp_file"

if ! grep -Eq '^[0-9]+
#39; "$tmp_file" && [[ -s "$tmp_file" ]]; then
  echo 'GitHub returned an invalid pull request list' >&2
  exit 1
fi

mv "$tmp_file" "$OUTPUT_FILE"

The REST request follows pagination and writes one number per line. An empty file is valid when the repository has no open pull requests. An API error stops the script because set -e is active. Never interpret an API failure as an empty desired state.

Make the file executable in Git. Pass the token only through the environment:

chmod +x scripts/preview-cleanup/fetch-open-prs.sh
GITHUB_REPOSITORY=OWNER/REPO OUTPUT_FILE=/tmp/open-prs.txt \
  GH_TOKEN=YOUR_TOKEN scripts/preview-cleanup/fetch-open-prs.sh

Verify Step 2: compare /tmp/open-prs.txt with the repository pull request page. Run sort -c -n /tmp/open-prs.txt; it should exit zero. Temporarily unset GH_TOKEN in a shell without stored GitHub CLI credentials and confirm the command fails instead of producing an apparently valid empty snapshot.

Step 3: Inventory Only Resources Your Reconciler Owns

Create /usr/local/bin/list-preview-projects on the preview host. It emits tab-separated project metadata from running or stopped Compose containers:

#!/usr/bin/env bash
set -Eeuo pipefail

docker ps -a \
  --filter 'label=qajobfit.preview.managed=true' \
  --format '{{.ID}}' | while IFS= read -r container_id; do
    [[ -n "$container_id" ]] || continue
    docker inspect "$container_id" --format \
'{{index .Config.Labels "qajobfit.preview.pr"}}\t{{index .Config.Labels "qajobfit.preview.repository"}}\t{{index .Config.Labels "qajobfit.preview.expires-at"}}\t{{index .Config.Labels "com.docker.compose.project"}}'
  done | sort -t 
#39;\t' -k1,1n -u

Install it as root-owned and non-writable by the deployment account:

sudo install -o root -g root -m 0755 list-preview-projects \
  /usr/local/bin/list-preview-projects

Filtering on an explicit management label avoids touching unrelated containers. The later cleanup script will additionally validate the repository, PR number, Compose project, and directory. Duplicate output is collapsed because one Compose project may contain multiple managed containers.

Verify Step 3: run /usr/local/bin/list-preview-projects. Expect rows such as 123<TAB>OWNER/REPO<TAB>1784000000<TAB>preview-123. Start an unrelated container and confirm it does not appear. Add a malformed label to a disposable test container and confirm the inventory shows the malformed value for the next step to reject.

Step 4: Write a Fail-Closed Reconciliation Script

Add scripts/preview-cleanup/reconcile.sh. It validates every destructive input and records one JSON object per decision:

#!/usr/bin/env bash
set -Eeuo pipefail

: "${EXPECTED_REPOSITORY:?EXPECTED_REPOSITORY is required}"
: "${OPEN_PRS_FILE:?OPEN_PRS_FILE is required}"
: "${INVENTORY_FILE:?INVENTORY_FILE is required}"
: "${AUDIT_FILE:?AUDIT_FILE is required}"
MODE=${MODE:-dry-run}
GRACE_SECONDS=${GRACE_SECONDS:-86400}
NOW_EPOCH=${NOW_EPOCH:-$(date -u +%s)}

[[ "$MODE" == dry-run || "$MODE" == delete ]] || exit 64
[[ "$GRACE_SECONDS" =~ ^[0-9]+$ ]] || exit 64
[[ "$NOW_EPOCH" =~ ^[0-9]+$ ]] || exit 64
[[ -f "$OPEN_PRS_FILE" && -f "$INVENTORY_FILE" ]] || exit 66
: > "$AUDIT_FILE"

log_decision() {
  jq -cn --arg pr "$1" --arg action "$2" --arg reason "$3" \
    --arg mode "$MODE" --argjson observedAt "$NOW_EPOCH" \
    '{pr:$pr,action:$action,reason:$reason,mode:$mode,observedAt:$observedAt}' \
    >> "$AUDIT_FILE"
}

while IFS=
#39;\t' read -r pr repository expires_at project; do
  [[ -n "$pr" ]] || continue

  if [[ ! "$pr" =~ ^[0-9]+$ || "$repository" != "$EXPECTED_REPOSITORY" || \
        "$project" != "preview-$pr" || ! "$expires_at" =~ ^[0-9]+$ ]]; then
    log_decision "$pr" skip invalid-metadata
    continue
  fi

  if grep -Fxq "$pr" "$OPEN_PRS_FILE"; then
    log_decision "$pr" keep pull-request-open
    continue
  fi

  delete_after=$((expires_at + GRACE_SECONDS))
  if (( NOW_EPOCH < delete_after )); then
    log_decision "$pr" keep grace-period-active
    continue
  fi

  if [[ "$MODE" == dry-run ]]; then
    log_decision "$pr" candidate orphan-expired
    continue
  fi

  preview_dir="/srv/previews/pr-$pr"
  if [[ -f "$preview_dir/compose.yaml" ]]; then
    docker compose -p "$project" -f "$preview_dir/compose.yaml" \
      down --volumes --remove-orphans
  else
    docker ps -aq --filter "label=com.docker.compose.project=$project" | \
      xargs -r docker rm -f
    docker volume ls -q --filter "label=com.docker.compose.project=$project" | \
      xargs -r docker volume rm
    docker network ls -q --filter "label=com.docker.compose.project=$project" | \
      xargs -r docker network rm
  fi
  rm -rf -- "$preview_dir"
  log_decision "$pr" deleted orphan-expired
done < "$INVENTORY_FILE"

Exact matching with grep -Fxq prevents PR 12 from matching 120. Numeric validation and a fixed directory prefix prevent shell injection and path traversal. A missing Compose file triggers label-scoped fallback removal, but no global prune command is used.

Verify Step 4: use fixture files before connecting to Docker:

printf '12\n' > /tmp/open.txt
printf '12\tOWNER/REPO\t100\tpreview-12\n13\tOWNER/REPO\t100\tpreview-13\n' > /tmp/inventory.tsv
EXPECTED_REPOSITORY=OWNER/REPO OPEN_PRS_FILE=/tmp/open.txt \
INVENTORY_FILE=/tmp/inventory.tsv AUDIT_FILE=/tmp/audit.jsonl \
MODE=dry-run NOW_EPOCH=200000 GRACE_SECONDS=10 ./scripts/preview-cleanup/reconcile.sh
jq . /tmp/audit.jsonl

Expect PR 12 to be kept because it is open and PR 13 to be a candidate. No Docker resource should change in dry-run mode.

Step 5: Run the Reconciler Remotely in Dry-Run Mode

Create .github/workflows/preview-cleanup.yaml with a manual trigger and a schedule. Start with deletion disabled:

name: Preview cleanup

on:
  workflow_dispatch:
    inputs:
      mode:
        description: Cleanup mode
        type: choice
        options: [dry-run, delete]
        default: dry-run
  schedule:
    - cron: '17 */6 * * *'

permissions:
  contents: read
  pull-requests: read

concurrency:
  group: preview-cleanup
  cancel-in-progress: false

jobs:
  reconcile:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    env:
      MODE: ${{ github.event_name == 'schedule' && 'dry-run' || inputs.mode }}
    steps:
      - uses: actions/checkout@v4
      - name: Fetch open pull requests
        env:
          GH_TOKEN: ${{ github.token }}
          OUTPUT_FILE: open-prs.txt
        run: scripts/preview-cleanup/fetch-open-prs.sh
      - name: Configure SSH
        env:
          SSH_KEY: ${{ secrets.PREVIEW_SSH_KEY }}
          PREVIEW_HOST: ${{ secrets.PREVIEW_HOST }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%s\n' "$SSH_KEY" > ~/.ssh/preview_key
          chmod 600 ~/.ssh/preview_key
          ssh-keyscan -H "$PREVIEW_HOST" >> ~/.ssh/known_hosts
      - name: Reconcile previews
        env:
          HOST: ${{ secrets.PREVIEW_HOST }}
          USER: ${{ secrets.PREVIEW_USER }}
        run: |
          scp -i ~/.ssh/preview_key open-prs.txt scripts/preview-cleanup/reconcile.sh \
            "$USER@$HOST:/tmp/"
          ssh -i ~/.ssh/preview_key "$USER@$HOST" "
            set -e
            /usr/local/bin/list-preview-projects > /tmp/inventory.tsv
            EXPECTED_REPOSITORY='$GITHUB_REPOSITORY' \
            OPEN_PRS_FILE=/tmp/open-prs.txt INVENTORY_FILE=/tmp/inventory.tsv \
            AUDIT_FILE=/tmp/preview-cleanup-audit.jsonl MODE='$MODE' \
            GRACE_SECONDS=86400 bash /tmp/reconcile.sh
          "
          scp -i ~/.ssh/preview_key \
            "$USER@$HOST:/tmp/preview-cleanup-audit.jsonl" audit.jsonl
      - uses: actions/upload-artifact@v4
        with:
          name: preview-cleanup-audit-${{ github.run_id }}
          path: audit.jsonl
          retention-days: 30

Pin third-party actions to reviewed commit SHAs if your organization requires immutable supply-chain references. cancel-in-progress: false prevents a new schedule from interrupting a deletion.

Verify Step 5: run the workflow manually with dry-run. Download the artifact and inspect jq . audit.jsonl. Every managed environment must have exactly one decision, active PRs must say keep, and no environment may disappear from the host.

Step 6: Test Deletion with a Disposable Orphan

Do not enable scheduled deletion first. Close a disposable PR or create a fixture project whose PR is definitely closed. Set its expiry in the past, wait through your chosen grace policy, then manually dispatch delete.

Before deletion, capture the expected project resources:

docker ps -a --filter label=com.docker.compose.project=preview-999
docker volume ls --filter label=com.docker.compose.project=preview-999
docker network ls --filter label=com.docker.compose.project=preview-999

After the workflow succeeds, repeat those commands and check the directory:

test ! -e /srv/previews/pr-999
docker ps -aq --filter label=com.docker.compose.project=preview-999 | grep -q . && exit 1 || true

The audit line must show deleted, orphan-expired, the PR number, and the observed timestamp. Dispatch delete a second time. It should succeed and make no additional changes because the environment no longer appears in inventory. This idempotency matters when GitHub retries a job after a network timeout.

Verify Step 6: confirm the disposable preview URL is unavailable, all project-scoped containers, volumes, and networks are gone, and unrelated projects remain healthy. Compare resource counts before and after rather than relying only on a successful workflow status.

Step 7: Destroy Stale Preview Environments Automatically on Schedule

After several clean dry runs and one verified manual deletion, change the scheduled expression for MODE to delete while preserving manual selection:

env:
  MODE: ${{ github.event_name == 'schedule' && 'delete' || inputs.mode }}

Keep the fast close-event teardown from your deployment workflow. The close handler should remove the environment immediately, while this scheduled job catches drift. If a close handler fails, TTL plus reconciliation provides recovery after the grace period.

Add a final workflow step that fails when invalid metadata appears in the audit. A failure creates a visible Actions signal without deleting the questionable resource:

- name: Reject invalid inventory metadata
  if: always()
  run: |
    if jq -e 'select(.reason == "invalid-metadata")' audit.jsonl >/dev/null; then
      echo 'Managed previews contain invalid cleanup metadata' >&2
      exit 1
    fi

Use your existing incident channel for repeated workflow failures. Do not put a long-lived webhook secret into untrusted pull request jobs. The cleanup workflow runs only from the default branch, with read-only repository permissions and isolated host credentials.

Verify Step 7: temporarily shorten TTL only for a disposable fixture. Confirm the first scheduled run keeps it during grace and a later run deletes it. Force the GitHub API request to fail in a test branch and confirm the workflow stops before SSH or deletion.

Step 8: Extend the Pattern Beyond Docker Compose

Keep the decision engine and replace the inventory and delete operations. For Kubernetes, list namespaces with labels such as preview-managed=true, repository=OWNER_REPO, pr=123, and expires-at=<unix-seconds>. Delete only a validated namespace name like preview-pr-123; never run broad label deletion across a production cluster.

For cloud stacks, use provider tags and delete the stack or project boundary rather than individual resources. The cleanup identity should have permission only for the preview account and naming prefix. Apply retention locks to shared registries, audit buckets, and DNS zones so a cleanup bug cannot remove platform infrastructure.

Database state needs its own verified ownership boundary. Use the Testcontainers ephemeral database seeding tutorial for disposable test data. If reviewers need production-shaped records, complete the preview environment production data masking tutorial before automating database copies or deletion.

Verify Step 8: in the target platform, inventory a known active preview, a closed expired fixture, an unrelated resource, and malformed metadata. A dry run must keep the active and unrelated resources, reject malformed metadata, and select only the expired fixture.

Troubleshooting

The workflow marks every environment as orphaned -> Stop deletion immediately. Inspect open-prs.txt, token permissions, repository identity, and API pagination. An API error must terminate the fetch step, never create an empty success result.

An open pull request appears as a deletion candidate -> Check exact numeric matching and GITHUB_REPOSITORY. Remove whitespace or carriage returns from transferred files. Keep grep -Fxq, not substring matching.

Inventory contains blank labels -> Redeploy the environment with lifecycle labels. The reconciler should report invalid-metadata and skip it. Do not guess ownership or derive expiry from container creation time.

Compose cleanup says the project does not exist -> Treat absence as success, then check label-scoped fallback resources and the preview directory. Verify that the project name is exactly preview-<PR_NUMBER>.

Volumes remain after deletion -> Confirm they are Compose-managed and carry the project label. External volumes are intentionally not removed by docker compose down --volumes; give preview data a project-scoped lifecycle instead of sharing it.

The SSH host key changes -> Fail the workflow and verify the host change through a trusted channel. Replace the stored known-host fingerprint deliberately. Do not disable host key checking.

Where To Go Next

You now have a conservative garbage collector for abandoned previews. Revisit the ephemeral test environments complete guide to map the same ownership, desired-state, TTL, and audit controls to Kubernetes or a managed platform.

Strengthen the surrounding lifecycle next:

• Build the fast path with one preview environment for every pull request.
• Create repeatable state with ephemeral database seeding using Testcontainers.
• Protect realistic fixtures with production data masking for preview environments.

Add dashboards for managed count, candidates, deletions, invalid metadata, and cleanup failures. Measure your own system over time, then set alerts from observed behavior rather than copying arbitrary thresholds.

Interview Questions and Answers

Q: Why is pull request close cleanup not sufficient?

Event delivery, workflow configuration, credentials, runners, and the target platform can fail independently. Close-event teardown remains the fast path, but scheduled reconciliation repairs missed or partial cleanup. Multiple idempotent paths provide resilience without making deletion less controlled.

Q: What conditions should be required before deleting a preview?

Require a trusted ownership marker, a valid resource identity, proof that the PR is not open, and an expired TTL plus grace period. The authoritative API query must succeed. If any input is missing or malformed, fail closed and preserve the resource.

Q: Why is age alone unsafe?

A legitimate review can remain open longer than the expected TTL, while a newly created orphan can already be unnecessary. Age describes time, not desired state or ownership. Combine time with the current PR state and validated metadata.

Q: How do you make cleanup idempotent?

Scope deletion to one validated project boundary and treat absence as success. Use declarative teardown such as docker compose down, then remove only label-matched leftovers. A retry should converge on the same empty result.

Q: What is fail-closed cleanup?

Fail-closed cleanup preserves resources when authority, inventory, metadata, or validation is uncertain. It may temporarily cost more, but it prevents an ambiguous condition from becoming destructive. Alert operators so they can repair the data or workflow.

Q: How would you adapt this design to Kubernetes?

Label a namespace with repository, PR, ownership, and expiry metadata. Compare those labels with the open PR snapshot, then delete only a validated expired orphan namespace. Use a preview-only service account and cluster or account boundary.

Best Practices to Destroy Stale Preview Environments Automatically

• Do reconcile desired state with actual resources. Do not run global Docker, Kubernetes, or cloud prune commands.
• Do use an explicit ownership label and strict identifier validation. Do not infer ownership from a loosely matching name.
• Do protect open PRs regardless of age. Do not let TTL alone authorize deletion.
• Do begin with dry runs and audit artifacts. Do not make the first test against real abandoned resources.
• Do keep API failure distinct from an empty PR list. Do not continue when authority cannot be queried.
• Do use least-privilege credentials in a preview-only boundary. Do not expose production credentials to cleanup automation.
• Do retry idempotently and alert on invalid metadata. Do not hide partial failures with unconditional || true.

Conclusion

To destroy stale preview environments automatically, combine an authoritative open-PR snapshot with owned resource inventory, strict validation, TTL grace, dry-run review, idempotent deletion, and audit logs. That design removes drift while preserving active and ambiguous environments.

Deploy the metadata first, observe several dry runs, delete one disposable orphan manually, and only then enable the schedule. Keep close-event teardown for speed and scheduled reconciliation for recovery, because dependable cleanup needs both.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Create a Preview Environment per Pull Request
• Docker Compose for test environments (2026)
• Ephemeral Test Environments Complete Guide (2026)
• Mask Production Data Preview Environments: A Safe Tutorial
• Use GitHub Actions OIDC for Test Environments
• A/B test validation: A Complete Guide for QA (2026)