Test Canary Release Metrics in Kubernetes (2026)

TL;DR

Send a controlled fraction of traffic to the canary, measure it beside the stable release, and promote only when error rate, p95 latency, traffic share, and minimum sample count all pass. The tutorial builds that loop on a local Kubernetes cluster with Prometheus, k6, PromQL, and an executable rollback gate.

Key Takeaways

• Compare canary and stable cohorts over the same traffic window instead of judging the canary in isolation.
• Gate on errors, tail latency, traffic share, request volume, and saturation rather than pod readiness alone.
• Tag every request and application metric with a bounded release label so PromQL can separate cohorts safely.
• Use k6 thresholds for client-visible failures and Prometheus queries for server-side release behavior.
• Require enough canary samples before promotion because a perfect ratio from ten requests proves very little.
• Exercise the rollback path with an intentionally degraded canary before trusting the production gate.

To test canary release metrics Kubernetes teams can trust, measure the candidate and stable versions during the same traffic window, enforce minimum sample volume, and reject the candidate when errors or tail latency exceed explicit limits. Pod readiness is only an admission signal. A safe promotion decision also needs request outcomes, latency distributions, traffic allocation, resource health, and a verified rollback action.

This tutorial gives you a complete local lab. You will deploy two versions of a small instrumented HTTP service, scrape both with Prometheus, send a 90/10 traffic mix with k6, query cohort-specific metrics, and run a shell gate that returns a CI-friendly exit code. Then you will inject a slow 20 percent error canary and prove that both the client test and server-side gate stop promotion. If Kubernetes objects are new to you, review Kubernetes basics for testers. For the broader release strategy, keep the canary testing guide nearby.

What You Will Build

• A kind cluster running Kubernetes 1.36.1 in a dedicated canary-lab namespace.
• Stable and canary Deployments exposing /api, /health, and Prometheus-format /metrics endpoints.
• A Prometheus server that preserves the release label for cohort queries.
• A k6 Job that sends approximately 90 percent of requests to stable and 10 percent to canary.
• A promotion gate that checks canary error rate, p95 latency, traffic share, and observed request count.

Separate Services make the lab split explicit. In production, let a Gateway API implementation, mesh, ingress controller, or delivery controller choose the backend while retaining the same metrics.

Prerequisites

Use these exact versions for a reproducible 2026 setup:

Install the host tools with your operating system's package manager and start Docker. Prometheus and k6 run as containers. Confirm the toolchain:

kind version
kubectl version --client
docker version --format '{{.Server.Version}}'
jq --version
curl --version | head -n 1

Verification: kind version reports v0.32.0, kubectl reports client v1.36.2, and docker version returns a server version instead of a daemon connection error. A newer patch release can work, but pinning the kind node image below keeps the Kubernetes API behavior reproducible.

Step 1: Create the Kubernetes Canary Lab

Pin the node image and its digest. The digest matters because a mutable tag can point to different bytes later, which makes a release experiment difficult to reproduce.

cat > kind-canary.yaml <<'YAML'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.36.1@sha256:3489c7674813ba5d8b1a9977baea8a6e553784dab7b84759d1014dbd78f7ebd5
YAML

kind create cluster --name canary-metrics --config kind-canary.yaml
kubectl create namespace canary-lab
kubectl config set-context --current --namespace=canary-lab

The namespace isolates the lab. Use explicit -n canary-lab flags in CI when jobs share a kubeconfig.

Verification: inspect both the client and server versions, then check node readiness.

kubectl version
kubectl get nodes
kubectl get namespace canary-lab

Expect one node with STATUS equal to Ready, a server minor version of 36, and an Active namespace. If the node stays NotReady, inspect docker ps and kubectl describe node before proceeding.

Step 2: Deploy Stable and Canary Metric Endpoints

Create a small Python service without third-party packages. It returns the active release in both JSON and the X-Release header. It also exposes a request counter and classic latency histogram in Prometheus text format. The bounded release and status labels are safe grouping dimensions; never use user IDs, request IDs, or URLs with arbitrary parameters as metric labels.

cat > app.yaml <<'YAML'
apiVersion: v1
kind: ConfigMap
metadata:
  name: catalog-code
data:
  app.py: |
    import json
    import os
    import threading
    import time
    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

    RELEASE = os.environ.get("RELEASE", "unknown")
    DELAY_MS = int(os.environ.get("DELAY_MS", "0"))
    ERROR_EVERY = int(os.environ.get("ERROR_EVERY", "0"))
    BUCKETS = [0.05, 0.1, 0.25, 0.5, 1.0]
    lock = threading.Lock()
    totals = {"200": 0, "500": 0}
    bucket_counts = {boundary: 0 for boundary in BUCKETS}
    duration_sum = 0.0

    class Handler(BaseHTTPRequestHandler):
      def log_message(self, format, *args):
        return

      def do_GET(self):
        global duration_sum
        if self.path == "/health":
          self.reply(200, {"status": "ok", "release": RELEASE})
          return
        if self.path == "/metrics":
          self.metrics()
          return
        if self.path != "/api":
          self.reply(404, {"error": "not found"})
          return

        started = time.monotonic()
        time.sleep(DELAY_MS / 1000)
        elapsed = time.monotonic() - started
        with lock:
          ordinal = totals["200"] + totals["500"] + 1
          status = 500 if ERROR_EVERY > 0 and ordinal % ERROR_EVERY == 0 else 200
          totals[str(status)] += 1
          duration_sum += elapsed
          for boundary in BUCKETS:
            if elapsed <= boundary:
              bucket_counts[boundary] += 1
        self.reply(status, {"release": RELEASE, "request": ordinal})

      def reply(self, status, payload):
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.send_header("X-Release", RELEASE)
        self.end_headers()
        self.wfile.write(body)

      def metrics(self):
        with lock:
          lines = [
            "# TYPE canary_http_requests_total counter",
            f'canary_http_requests_total{{release="{RELEASE}",status="200"}} {totals["200"]}',
            f'canary_http_requests_total{{release="{RELEASE}",status="500"}} {totals["500"]}',
            "# TYPE canary_http_request_duration_seconds histogram",
          ]
          for boundary in BUCKETS:
            lines.append(f'canary_http_request_duration_seconds_bucket{{release="{RELEASE}",le="{boundary}"}} {bucket_counts[boundary]}')
          count = totals["200"] + totals["500"]
          lines.extend([
            f'canary_http_request_duration_seconds_bucket{{release="{RELEASE}",le="+Inf"}} {count}',
            f'canary_http_request_duration_seconds_sum{{release="{RELEASE}"}} {duration_sum}',
            f'canary_http_request_duration_seconds_count{{release="{RELEASE}"}} {count}',
          ])
        body = ("\n".join(lines) + "\n").encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-stable
spec:
  replicas: 1
  selector:
    matchLabels: { app: catalog, release: stable }
  template:
    metadata:
      labels: { app: catalog, release: stable }
    spec:
      containers:
        - name: app
          image: python:3.13-alpine3.23
          command: ["python", "/app/app.py"]
          env:
            - { name: RELEASE, value: stable }
            - { name: DELAY_MS, value: "40" }
            - { name: ERROR_EVERY, value: "0" }
          ports:
            - { name: http, containerPort: 8080 }
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 2
          volumeMounts:
            - { name: code, mountPath: /app }
      volumes:
        - name: code
          configMap: { name: catalog-code }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-canary
spec:
  replicas: 1
  selector:
    matchLabels: { app: catalog, release: canary }
  template:
    metadata:
      labels: { app: catalog, release: canary }
    spec:
      containers:
        - name: app
          image: python:3.13-alpine3.23
          command: ["python", "/app/app.py"]
          env:
            - { name: RELEASE, value: canary }
            - { name: DELAY_MS, value: "65" }
            - { name: ERROR_EVERY, value: "0" }
          ports:
            - { name: http, containerPort: 8080 }
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 2
          volumeMounts:
            - { name: code, mountPath: /app }
      volumes:
        - name: code
          configMap: { name: catalog-code }
---
apiVersion: v1
kind: Service
metadata:
  name: catalog-stable
spec:
  selector: { app: catalog, release: stable }
  ports:
    - { name: http, port: 8080, targetPort: http }
---
apiVersion: v1
kind: Service
metadata:
  name: catalog-canary
spec:
  selector: { app: catalog, release: canary }
  ports:
    - { name: http, port: 8080, targetPort: http }
YAML

kubectl apply -f app.yaml
kubectl rollout status deployment/catalog-stable --timeout=120s
kubectl rollout status deployment/catalog-canary --timeout=120s

Stable waits 40 ms and healthy canary waits 65 ms, both below the later 250 ms p95 limit. /health excludes the injected business error because readiness answers whether the process can accept traffic.

Verification: call each Service from a temporary pod and confirm the reported cohort.

kubectl run curl-check --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  sh -c 'curl -fsS http://catalog-stable:8080/api && echo && curl -fsS http://catalog-canary:8080/api'

The first JSON object contains "release": "stable"; the second contains "release": "canary". A timeout usually means the Service selector has no Ready endpoints.

Step 3: Define Test Canary Release Metrics Kubernetes Gates

Write the decision contract before the rollout. Absolute limits protect the user promise; stable-relative comparisons catch regressions that remain barely inside a loose SLO.

Never average pod percentiles. Aggregate bucket rates by le, then apply histogram_quantile. Use rate() or increase() for counters. This lab keeps a five-minute query window.

Verification: review the contract with the service owner and state the required action for every failed signal. Here, hard failures block promotion, bad exposure invalidates the experiment, low volume extends observation, and saturation triggers rollback. Inconclusive does not mean successful.

Step 4: Install Prometheus and Confirm Both Targets

Prometheus scrapes each Service every five seconds. Separate jobs expose target health, and the application-supplied release label keeps queries portable.

cat > prometheus.yaml <<'YAML'
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    scrape_configs:
      - job_name: catalog-stable
        static_configs:
          - targets: ["catalog-stable:8080"]
      - job_name: catalog-canary
        static_configs:
          - targets: ["catalog-canary:8080"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels: { app: prometheus }
  template:
    metadata:
      labels: { app: prometheus }
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v3.12.0
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.retention.time=2h
          ports:
            - { name: web, containerPort: 9090 }
          readinessProbe:
            httpGet: { path: /-/ready, port: web }
            periodSeconds: 2
          volumeMounts:
            - { name: config, mountPath: /etc/prometheus }
      volumes:
        - name: config
          configMap: { name: prometheus-config }
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  selector: { app: prometheus }
  ports:
    - { name: web, port: 9090, targetPort: web }
YAML

kubectl apply -f prometheus.yaml
kubectl rollout status deployment/prometheus --timeout=120s

Verification: query target health from inside the cluster.

kubectl run prom-check --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  curl -fsS 'http://prometheus:9090/api/v1/query?query=up'

The JSON response should contain two results with value 1, one for job="catalog-stable" and one for job="catalog-canary". A missing job means the target is not configured; value 0 means Prometheus knows the target but cannot scrape it. Inspect kubectl logs deployment/prometheus and the Service endpoints separately.

Step 5: Send a Controlled 90/10 Traffic Mix With k6

Create a k6 test that chooses the canary for roughly one request in ten. Request tags produce release-specific client metrics and thresholds. The header assertion also detects misrouting, which a status-only load test would miss.

cat > test.js <<'JS'
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    canary_mix: {
      executor: 'constant-arrival-rate',
      rate: 20,
      timeUnit: '1s',
      duration: '90s',
      preAllocatedVUs: 10,
      maxVUs: 30,
    },
  },
  thresholds: {
    'http_req_failed{release:canary}': ['rate<0.01'],
    'http_req_duration{release:canary}': ['p(95)<250'],
    'checks{release:canary}': ['rate>0.99'],
  },
};

const stableUrl = __ENV.STABLE_URL;
const canaryUrl = __ENV.CANARY_URL;

export default function () {
  const release = Math.random() < 0.10 ? 'canary' : 'stable';
  const url = release === 'canary' ? canaryUrl : stableUrl;
  const response = http.get(url, {
    tags: { release, name: 'catalog_api' },
  });
  check(response, {
    'status is 200': (r) => r.status === 200,
    'release header matches route': (r) => r.headers['X-Release'] === release,
  }, { release });
  sleep(0.05);
}
JS

kubectl create configmap k6-canary-script --from-file=test.js
cat > k6-job.yaml <<'YAML'
apiVersion: batch/v1
kind: Job
metadata:
  name: canary-load
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: k6
          image: grafana/k6:2.1.0
          args: ["run", "/scripts/test.js"]
          env:
            - { name: STABLE_URL, value: "http://catalog-stable:8080/api" }
            - { name: CANARY_URL, value: "http://catalog-canary:8080/api" }
          volumeMounts:
            - { name: script, mountPath: /scripts }
      volumes:
        - name: script
          configMap: { name: k6-canary-script }
YAML

kubectl apply -f k6-job.yaml
kubectl wait --for=condition=complete job/canary-load --timeout=180s
kubectl logs job/canary-load

Constant arrival rate decouples offered rate from response time until k6 exhausts maxVUs. Model real production journeys instead of copying 20 requests/second. See the load testing guide and finding a performance bottleneck.

Verification: the k6 summary shows green thresholds for canary failure rate, canary p95, and canary checks. dropped_iterations should be zero. Because selection is random, the exact canary share will vary, but approximately 180 of the 1,800 scheduled iterations should reach it.

Step 6: Query Canary Release Metrics in Prometheus

Forward Prometheus to your host and keep the process running through Step 8.

kubectl port-forward service/prometheus 9090:9090 > /tmp/canary-prometheus-port-forward.log 2>&1 &
export PROMETHEUS_PORT_FORWARD_PID=$!
export PROMETHEUS_URL=http://127.0.0.1:9090

sleep 2
curl -fsS "$PROMETHEUS_URL/-/ready"

Start with error rate. or vector(0) converts an absent 500-series increase into numeric zero.

curl -fsS --get "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode 'query=(sum(rate(canary_http_requests_total{release="canary",status=~"5.."}[5m])) / sum(rate(canary_http_requests_total{release="canary"}[5m]))) or vector(0)' | jq '.data.result'

Calculate p95 from cumulative buckets, then measure exposure.

curl -fsS --get "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (le) (rate(canary_http_request_duration_seconds_bucket{release="canary"}[5m])))' | jq '.data.result'

curl -fsS --get "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode 'query=sum(rate(canary_http_requests_total{release="canary"}[5m])) / sum(rate(canary_http_requests_total[5m]))' | jq '.data.result'

Verification: expect an error rate of 0, p95 below 0.25 seconds, and traffic share near 0.10. Histogram quantiles are bucket-bound estimates. Put boundaries near the SLO or the gate lacks useful precision.

Step 7: Automate Test Canary Release Metrics Kubernetes in CI

Turn the PromQL contract into a portable gate. This script rejects missing or nonnumeric values, then applies four hard checks with awk. A failed query exits before comparison, so missing telemetry cannot approve a deployment.

cat > gate.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
PROMETHEUS_URL=${PROMETHEUS_URL:-http://127.0.0.1:9090}

query() {
  curl -fsS --get "$PROMETHEUS_URL/api/v1/query" \
    --data-urlencode "query=$1" | jq -er '.data.result[0].value[1]'
}

error_rate=$(query '(sum(rate(canary_http_requests_total{release="canary",status=~"5.."}[5m])) / sum(rate(canary_http_requests_total{release="canary"}[5m]))) or vector(0)')
p95=$(query 'histogram_quantile(0.95, sum by (le) (rate(canary_http_request_duration_seconds_bucket{release="canary"}[5m])))')
traffic_share=$(query 'sum(rate(canary_http_requests_total{release="canary"}[5m])) / sum(rate(canary_http_requests_total[5m]))')
total_requests=$(query 'sum(increase(canary_http_requests_total[5m]))')

number_pattern='^-?([0-9]+([.][0-9]*)?|[.][0-9]+)([eE][-+]?[0-9]+)?
#39;
for value in "$error_rate" "$p95" "$traffic_share" "$total_requests"; do
  [[ $value =~ $number_pattern ]] || { echo "FAIL: nonnumeric Prometheus result: $value"; exit 2; }
done

printf 'error_rate=%s p95_seconds=%s traffic_share=%s total_requests=%s\n' \
  "$error_rate" "$p95" "$traffic_share" "$total_requests"

awk -v e="$error_rate" -v p="$p95" -v s="$traffic_share" -v n="$total_requests" 'BEGIN {
  failed = 0
  if (e >= 0.01) { print "FAIL: canary error rate is at least 1%"; failed = 1 }
  if (p >= 0.25) { print "FAIL: canary p95 is at least 250 ms"; failed = 1 }
  if (s < 0.05 || s > 0.15) { print "FAIL: canary traffic is outside 5% to 15%"; failed = 1 }
  if (n <= 500) { print "FAIL: total request count is not above 500"; failed = 1 }
  exit failed
}'

echo "PASS: canary is eligible for the next rollout stage"
BASH

chmod +x gate.sh
./gate.sh

A production gate should add restarts, CPU throttling, memory, queue depth, and dependency signals. Separate metric evaluation from promotion authorization for an auditable decision. See the DevOps for QA roadmap.

Verification: the healthy lab prints four numeric values, no FAIL lines, the final PASS message, and exit code zero. Run echo $? immediately if you want to see the code. Archive both the raw Prometheus responses and the evaluated values as pipeline evidence.

Step 8: Prove the Gate Rejects a Bad Canary and Roll Back

A promotion test is incomplete until you watch it fail for the intended reason. Change only the canary to add 350 ms latency and return a 500 response every fifth business request. Stable remains unchanged, which gives you a useful control cohort.

kubectl set env deployment/catalog-canary DELAY_MS=350 ERROR_EVERY=5
kubectl rollout status deployment/catalog-canary --timeout=120s

kubectl delete job canary-load --ignore-not-found
kubectl apply -f k6-job.yaml
kubectl wait --for=condition=failed job/canary-load --timeout=180s
kubectl logs job/canary-load

The Job fails its canary error, duration, and check thresholds. After several scrapes, run the server gate while capturing its expected nonzero exit.

set +e
./gate.sh
gate_exit=$?
set -e
test "$gate_exit" -ne 0

kubectl scale deployment/catalog-canary --replicas=0
kubectl wait --for=delete pod -l app=catalog,release=canary --timeout=120s
kubectl get endpointslice -l kubernetes.io/service-name=catalog-canary

Scaling to zero models capacity removal. In production, first set route weight to zero, drain connections, then scale down or revert. Removing pods first can create 502 or 503 responses.

Verification: test "$gate_exit" -ne 0 succeeds, the gate output identifies error rate and p95 failures, and the canary EndpointSlice has no ready address. Stable should still answer normally:

kubectl run stable-after-rollback --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  curl -fsS http://catalog-stable:8080/api

A stable response proves recovery at the request layer, not merely that a deployment command returned zero.

Best Practices for Canary Deployment Metrics

• Use a release label with bounded values. stable, canary, and an immutable build identifier are useful. Raw commit messages or request-specific values create costly cardinality.
• Compare simultaneous windows. Stable and canary requests should traverse the same shared dependencies and regional conditions. Yesterday's baseline cannot explain today's database incident.
• Separate exposure checks from quality checks. Zero errors are meaningless when routing accidentally sent zero requests to the candidate.
• Set a sample floor. For rare failures, request count matters more than elapsed minutes. Extend the stage when evidence is insufficient.
• Keep health probes out of business-success metrics. Frequent /health requests can dilute an application error ratio if they share the same counter.
• Test long-lived protocols explicitly. WebSockets, streaming responses, gRPC streams, and sticky sessions may not rebalance when a weight changes.
• Watch version-specific saturation. A canary with one replica can throttle while ten stable replicas remain comfortable; aggregate-by-service dashboards can hide that asymmetry.
• Make rollback idempotent. Repeating the action should keep candidate traffic at zero without corrupting state or producing a second incident.

Read cloud-native performance testing before using a shared cluster for load. It covers resource limits, autoscaling, noisy neighbors, and generator placement that can otherwise distort the experiment.

Troubleshooting

Problem: Prometheus returns an empty result for p95 -> Generate requests, wait for at least two scrapes, and confirm the query window covers both samples. rate(metric[5m]) needs observations across time; a newly created series cannot produce a meaningful slope immediately.

Problem: up is zero for one catalog job -> Run kubectl get endpoints catalog-canary catalog-stable and verify label selectors match Ready pods. Then call /metrics from a curl pod. This separates service discovery, endpoint health, and text exposition failures.

Problem: canary traffic share is far outside 10 percent -> First count total requests. Random selection has wide variation at tiny volumes, so increase duration before changing the gate. If volume is adequate, inspect the real router's configured weight, route match precedence, session affinity, and retry behavior.

Problem: k6 reports release header matches route failures with HTTP 200 -> Traffic reached the wrong cohort or an intermediary cached a response. Check Service selectors, gateway backend references, cache keys, and whether the header is overwritten at ingress. A status assertion alone would falsely pass this release.

Problem: the gate passes although dashboards show failures -> Align label filters and time windows, then inspect whether health traffic is included in the denominator. Also check that the dashboard and gate use the same counter reset handling and histogram aggregation.

Problem: the k6 Job stays active or shows dropped iterations -> Inspect pod CPU limits and maxVUs. A constant-arrival-rate executor drops work when available VUs cannot sustain the target; reduce the offered rate for a laptop lab or allocate enough generator capacity before interpreting service latency.

Interview Questions and Answers

The structured Q&A below tests formulas and operational decisions. For broader cluster practice, use Docker and Kubernetes interview questions for QA automation.

Where To Go Next

Replace random client routing with the platform's real traffic layer, retaining cohort labels and PromQL. Add a 25 percent stage with a fresh window, plus business invariants where HTTP metrics do not represent success.

Next, strengthen one dimension at a time:

1. Use the canary testing guide to design stage progression, ownership, and abort policy.
2. Apply the API performance testing tutorial to build a workload from endpoint behavior instead of a single synthetic request.
3. Follow cloud-native performance testing to add Kubernetes resource and autoscaling evidence.
4. Practice finding a performance bottleneck when canary latency rises but error rate stays flat.
5. Use the DevOps for QA roadmap to place release gates inside a broader CI/CD learning plan.

When you are finished, stop the port-forward and remove the local cluster:

kill "$PROMETHEUS_PORT_FORWARD_PID" 2>/dev/null || true
kind delete cluster --name canary-metrics

Verification: kind get clusters no longer lists canary-metrics; other kind clusters remain untouched.

Conclusion

To test canary release metrics Kubernetes pipelines need both controlled exposure and a predeclared decision contract. Measure the candidate beside stable, verify that traffic actually reached it, aggregate counters and histograms correctly, and block promotion when client or server signals breach their limits. Readiness starts the experiment; it does not approve the release.

The deliberate failure supplies the strongest proof: k6 rejects the candidate, the Prometheus gate exits nonzero, rollback removes canary endpoints, and stable still serves. Repeat that chain for each routing platform and service-specific SLO.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Debug a failing test in VS Code in Cypress (2026)
• How to Debug a failing test in VS Code in Playwright (2026)
• How to Debug a failing test in VS Code in Selenium (2026)
• Flaky test quarantine in CI: Step by Step (2026)
• How to Choose a test automation tool in 2026 (2026)
• How to Run a single test in Cypress (2026)