Define SLO Tests With Prometheus (2026)

TL;DR

Define each SLO as a ratio of good events to total events, encode the ratio in Prometheus recording rules, and use promtool fixtures to test normal and budget-burning series. Run syntax and unit tests in CI with the same pinned Prometheus image used locally.

Key Takeaways

• Define the good-event and total-event PromQL before choosing alert thresholds.
• Test recording rules and alerts with synthetic counter series instead of waiting for a production incident.
• Keep SLO targets separate from short-window burn-rate alert multipliers.
• Use histogram buckets for latency compliance when the objective is a proportion under a boundary.
• Test healthy, boundary, breach, and no-traffic cases so a green fixture proves meaningful behavior.
• Pin the Prometheus image in local checks and CI so promtool semantics do not drift silently.
• Treat rule tests as one layer alongside metric contract, load, and live alert-routing checks.

To define slo tests with prometheus, turn each service promise into an explicit PromQL ratio, save that ratio as a recording rule, and exercise the rule with deterministic time series through promtool test rules. A useful test proves healthy traffic passes, bad traffic consumes the expected budget, and an alert fires only after its configured duration.

This tutorial builds that workflow for a fictional checkout-api. The service has a 30-day availability target of 99.9 percent and a latency target that at least 95 percent of requests complete within 500 ms. You will test both objectives without deploying the service or causing real failures. For the wider relationship between load generation, telemetry, and release decisions, read the cloud-native performance testing guide.

Prometheus rule tests are not browser tests and do not prove that instrumentation exists in production. They prove the mathematical contract: given known counters, recording rules return known ratios and alerts reach the expected state. That narrow scope makes them fast enough to run on every rule change.

What You Will Build

You will create a small, runnable SLO project containing:

• slo-rules.yml, with availability, latency, and burn-rate recording rules;
• slo-tests.yml, with synthetic counter and histogram inputs for promtool;
• prometheus.yml, which loads the tested rules in a local Prometheus server;
• compose.yml, which pins Prometheus 3.12.0 for repeatable execution; and
• .github/workflows/slo-rules.yml, which blocks a merge when syntax or behavior changes unexpectedly.

The examples model one service and one objective set, but the design scales by adding a service or slo label only when those labels have bounded values. Do not add request IDs, URLs, customer IDs, or other high-cardinality dimensions to SLO recording rules.

The completed test pyramid looks like this:

Prerequisites

Use these exact tutorial versions: Prometheus and promtool 3.12.0 from prom/prometheus:v3.12.0, Docker Engine 28.3 or later, Docker Compose 2.39 or later, Git 2.50 or later, and jq 1.7 or later. Prometheus 3.12.0 is the pinned compatibility boundary for the examples. If your organization runs another supported release, change the image only after the same fixture suite passes there.

Create an empty directory and enter it:

mkdir prometheus-slo-tests
cd prometheus-slo-tests

Confirm the tools and image:

docker version --format '{{.Server.Version}}'
docker compose version
git --version
jq --version
docker run --rm --entrypoint /bin/promtool \
  prom/prometheus:v3.12.0 --version

Verification: the final command reports promtool, version 3.12.0. Stop here if Docker cannot pull or execute the image. A different promtool version can parse or evaluate edge cases differently, which weakens the promise that local and CI results match.

Step 1: How to define slo tests with prometheus from service promises

Start with user-visible events, not infrastructure utilization. CPU saturation might explain an outage, but it is not the availability event experienced by a customer. For this checkout example, one HTTP request is the total event. Any response with a 5xx status is a bad availability event. The good-event ratio is therefore 1 - bad / total.

Write the objective sheet before writing PromQL:

The availability error budget is 1 - 0.999 = 0.001, or 0.1 percent of eligible requests. A burn rate of 1 consumes that budget at exactly the pace allowed by the 30-day objective. A burn rate of 14.4 means the service is consuming budget 14.4 times faster than allowed. The multiplier does not change the SLO target. It determines how urgently the alert should react.

For latency, a classic Prometheus histogram can count requests less than or equal to 500 ms through the le="0.5" bucket. Divide that bucket's rate by the _count rate. This directly tests the proportion promised by the objective. histogram_quantile(0.95, ...) answers a different question: it estimates the p95 value from buckets. The guide to reading p95 and p99 latency explains when that percentile view is useful.

Verification: review the sheet with the service owner. They should be able to classify a 429, timeout, cancelled request, health check, and retry as included or excluded. If any event is ambiguous, the SLO is not ready to encode.

Step 2: Create recording and alerting rules

Save the following as slo-rules.yml. The first rules normalize raw metrics into stable five-minute rates. Later rules reuse those names, so keep the group order unchanged.

groups:
  - name: checkout-slo
    interval: 30s
    rules:
      - record: job:slo_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{job="checkout-api"}[5m]))

      - record: job:slo_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[5m]))

      - record: job:slo_error_ratio:rate5m
        expr: job:slo_errors:rate5m / job:slo_requests:rate5m

      - record: job:slo_availability:ratio_rate5m
        expr: 1 - job:slo_error_ratio:rate5m

      - record: job:slo_latency_le_500ms:ratio_rate5m
        expr: |
          sum by (job) (
            rate(http_request_duration_seconds_bucket{job="checkout-api",le="0.5"}[5m])
          )
          /
          sum by (job) (
            rate(http_request_duration_seconds_count{job="checkout-api"}[5m])
          )

      - record: job:slo_error_ratio:rate30m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[30m]))

      - record: job:slo_error_ratio:rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[1h]))

      - record: job:slo_error_ratio:rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api",code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[6h]))

      - alert: CheckoutAvailabilityFastBurn
        expr: |
          (job:slo_error_ratio:rate1h > 14.4 * 0.001)
          and
          (job:slo_error_ratio:rate5m > 14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          service: checkout-api
        annotations:
          summary: Checkout API is burning the availability budget quickly

      - alert: CheckoutAvailabilitySlowBurn
        expr: |
          (job:slo_error_ratio:rate6h > 6 * 0.001)
          and
          (job:slo_error_ratio:rate30m > 6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          service: checkout-api
        annotations:
          summary: Checkout API is steadily burning the availability budget

      - alert: CheckoutLatencySLOBreach
        expr: job:slo_latency_le_500ms:ratio_rate5m < 0.95
        for: 5m
        labels:
          severity: ticket
          service: checkout-api
        annotations:
          summary: Fewer than 95 percent of checkout requests finish within 500 ms

The two-window availability alerts require a long and short signal to agree. The long window provides confidence that the budget burn is material. The short window lets the alert recover sooner after the incident stops. The numbers are an explicit policy choice for this tutorial, not universal values. Model your own paging and ticket thresholds against the objective window and response process.

Check the file before adding tests:

docker run --rm --entrypoint /bin/promtool \
  -v "$PWD:/work" -w /work \
  prom/prometheus:v3.12.0 check rules slo-rules.yml

Verification: promtool prints SUCCESS: 11 rules found. It should report eight recording rules and three alerting rules. A zero exit code proves the file parses, but it does not prove the equations express the intended service promise.

Step 3: Test the healthy availability path

Create slo-tests.yml with a first fixture. Promtool's expanding notation 0+600x10 means start at zero, add 600 at each evaluation interval, and produce ten additional samples. With a one-minute interval, that represents ten requests per second. The error counter stays flat.

rule_files:
  - slo-rules.yml

evaluation_interval: 1m
group_eval_order:
  - checkout-slo

tests:
  - name: healthy availability produces a ratio of one
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout-api",code="200"}'
        values: '0+600x10'
      - series: 'http_requests_total{job="checkout-api",code="500"}'
        values: '0+0x10'
    promql_expr_test:
      - expr: job:slo_requests:rate5m{job="checkout-api"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_requests:rate5m{job="checkout-api"}'
            value: 10
      - expr: job:slo_error_ratio:rate5m{job="checkout-api"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_error_ratio:rate5m{job="checkout-api"}'
            value: 0
      - expr: job:slo_availability:ratio_rate5m{job="checkout-api"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_availability:ratio_rate5m{job="checkout-api"}'
            value: 1

Run the behavioral test:

docker run --rm --entrypoint /bin/promtool \
  -v "$PWD:/work" -w /work \
  prom/prometheus:v3.12.0 test rules slo-tests.yml

This fixture asserts intermediate signals as well as the final availability. If the final expression fails, the request rate and error ratio narrow the defect quickly. Testing only the last rule would tell you less about whether aggregation, label matching, or arithmetic changed.

Verification: the command ends with SUCCESS. Change the expected availability from 1 to 0.99, rerun, and confirm promtool reports the exact expression, labels, expected value, and actual value. Restore 1 and rerun successfully. This deliberate red test proves that the fixture is actually being discovered.

Step 4: Test the latency SLO at and below its boundary

Append a second item beneath tests: in slo-tests.yml. Keep the indentation aligned with the first - name item. The bucket grows by 570 requests per minute while the count grows by 600, producing exactly 570 / 600 = 0.95.

  - name: latency compliance equals the 95 percent objective
    interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_bucket{job="checkout-api",le="0.5"}'
        values: '0+570x10'
      - series: 'http_request_duration_seconds_count{job="checkout-api"}'
        values: '0+600x10'
    promql_expr_test:
      - expr: job:slo_latency_le_500ms:ratio_rate5m{job="checkout-api"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_latency_le_500ms:ratio_rate5m{job="checkout-api"}'
            value: 0.95
    alert_rule_test:
      - eval_time: 10m
        alertname: CheckoutLatencySLOBreach
        exp_alerts: []

  - name: sustained 90 percent latency compliance fires the breach alert
    interval: 1m
    input_series:
      - series: 'http_request_duration_seconds_bucket{job="checkout-api",le="0.5"}'
        values: '0+540x12'
      - series: 'http_request_duration_seconds_count{job="checkout-api"}'
        values: '0+600x12'
    promql_expr_test:
      - expr: job:slo_latency_le_500ms:ratio_rate5m{job="checkout-api"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_latency_le_500ms:ratio_rate5m{job="checkout-api"}'
            value: 0.9
    alert_rule_test:
      - eval_time: 10m
        alertname: CheckoutLatencySLOBreach
        exp_alerts:
          - exp_labels:
              job: checkout-api
              severity: ticket
              service: checkout-api
            exp_annotations:
              summary: Fewer than 95 percent of checkout requests finish within 500 ms

Run the same test command again:

docker run --rm --entrypoint /bin/promtool \
  -v "$PWD:/work" -w /work \
  prom/prometheus:v3.12.0 test rules slo-tests.yml

The boundary fixture matters because the alert uses < 0.95, not <= 0.95. Exactly 95 percent complies with the written promise. The breach fixture evaluates after enough time for the five-minute for duration to elapse. If you tested only the expression at one instant, you could miss an alert that never reaches firing state.

A histogram bucket is cumulative. The le="0.5" series must never exceed the _count series and both must behave as counters. In a fuller histogram fixture, each higher bucket must contain at least as many observations as the lower bucket.

Verification: promtool reports SUCCESS for both latency cases. Then set the breach fixture's eval_time to 4m. The expected firing alert should fail because the condition has not remained active for five minutes. Restore 10m after confirming the timing test.

Step 5: Prove fast and slow error-budget burns

Append two more fixtures under tests:. Twelve errors out of 600 requests per minute creates a 0.02 error ratio. That exceeds both the fast threshold 14.4 * 0.001 = 0.0144 and the slow threshold 6 * 0.001 = 0.006. The evaluation times distinguish the alert durations.

  - name: two percent errors fire the fast burn alert
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout-api",code="200"}'
        values: '0+588x20'
      - series: 'http_requests_total{job="checkout-api",code="500"}'
        values: '0+12x20'
    alert_rule_test:
      - eval_time: 10m
        alertname: CheckoutAvailabilityFastBurn
        exp_alerts:
          - exp_labels:
              job: checkout-api
              severity: page
              service: checkout-api
            exp_annotations:
              summary: Checkout API is burning the availability budget quickly

  - name: two percent errors sustained for thirty minutes fire slow burn
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout-api",code="200"}'
        values: '0+588x35'
      - series: 'http_requests_total{job="checkout-api",code="500"}'
        values: '0+12x35'
    alert_rule_test:
      - eval_time: 30m
        alertname: CheckoutAvailabilitySlowBurn
        exp_alerts:
          - exp_labels:
              job: checkout-api
              severity: ticket
              service: checkout-api
            exp_annotations:
              summary: Checkout API is steadily burning the availability budget

Run all fixtures:

docker run --rm --entrypoint /bin/promtool \
  -v "$PWD:/work" -w /work \
  prom/prometheus:v3.12.0 test rules slo-tests.yml

Notice that each status-code series is a separate monotonically increasing counter. PromQL aggregates them by job, so the total is 600 per minute and the bad count is 12. Never model a counter by entering percentage values directly. That can make rate() results look plausible while testing a time series that no real exporter could emit.

Add a just-below fixture in your production suite. For the fast alert, 8 errors per 600 requests is an error ratio near 0.01333, below 0.0144, so exp_alerts: [] should hold. Boundary pairs catch accidental changes from > to >=, incorrect budget constants, and unit conversion errors.

Verification: all five fixtures pass in one promtool invocation. Search the output for failures is not enough. Confirm the process status with echo $?; it must print 0. Temporarily change 14.4 to 144 and confirm the fast-burn fixture turns red, then restore the intended multiplier.

Step 6: Load the tested rules into Prometheus

Rule tests use the Prometheus evaluator, but a local server catches bad mount paths and omitted rule_files. Save this as prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/slo-rules.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]

Save this as compose.yml:

services:
  prometheus:
    image: prom/prometheus:v3.12.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./slo-rules.yml:/etc/prometheus/slo-rules.yml:ro
    ports:
      - 9090:9090

Start Prometheus and query its APIs:

docker compose up -d
curl -fsS http://localhost:9090/-/ready
curl -fsS http://localhost:9090/api/v1/rules \
  | jq -r '.data.groups[] | select(.name == "checkout-slo") | .rules[].name'

The checkout expressions have no values because this local server does not scrape checkout-api. That is expected. The purpose of this step is to prove that the server loads the same group that promtool tested. A rule can pass a unit fixture but remain absent at runtime because the configuration points at another directory or glob.

Open http://localhost:9090/rules if you want to inspect health and last evaluation. For sustained QA observability, connect these recordings to the Grafana dashboard guide for test metrics, but keep dashboards downstream of tested recording rules. A panel query should not become the only copy of your SLO math.

Verification: readiness returns Prometheus Server is Ready. and the API lists all eight recording names plus three alerts under checkout-slo. Run docker compose logs prometheus and confirm there is no error for loading or evaluating rules. Stop the server with docker compose down after inspection.

Step 7: Run define slo tests with prometheus in CI

Commit syntax checking and behavioral testing as separate commands. A syntax failure then produces a short parsing error, while a semantic failure shows the mismatched series. Save this workflow as .github/workflows/slo-rules.yml in the sample project:

name: SLO rule tests

on:
  pull_request:
    paths:
      - slo-rules.yml
      - slo-tests.yml
      - .github/workflows/slo-rules.yml
  push:
    branches: [main]

jobs:
  promtool:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4

      - name: Check rule syntax
        run: |
          docker run --rm --entrypoint /bin/promtool \
            -v "$PWD:/work" -w /work \
            prom/prometheus:v3.12.0 check rules slo-rules.yml

      - name: Test SLO behavior
        run: |
          docker run --rm --entrypoint /bin/promtool \
            -v "$PWD:/work" -w /work \
            prom/prometheus:v3.12.0 test rules slo-tests.yml

Keep the image tag identical across prerequisites, local commands, Compose, and CI. For stronger supply-chain repeatability, resolve the approved multi-architecture image digest in your registry and pin image@sha256:...; do not copy a digest from an unrelated platform. Automate dependency review so security updates produce an intentional version-change pull request followed by the full fixture suite.

This workflow tests stored Prometheus rules, not live Alertmanager delivery. Add a deployment-stage smoke check that verifies the rule group health, alert routing, inhibition, and receiver integration in a non-production environment. If performance traffic produces the raw metrics, pair these tests with k6 thresholds and checks. The two gates answer different questions: k6 assesses one generated workload, while Prometheus evaluates service behavior across an observation window.

Verification: push a branch with the known-good files and confirm both steps pass. On a second branch, change the healthy expected availability to 0.99; the behavioral step must fail and the workflow must block merging. Revert the mutation rather than weakening the expected result. The CI integration guide for test frameworks covers artifact retention and required-check policy.

Troubleshooting

Problem: promtool reports an unexpected empty vector -> Check that fixture labels match every selector. job="checkout-api" and job="checkout_api" are different series. Also allow at least two input samples for rate() and evaluate after the range has useful data. Query intermediate recording rules in promql_expr_test before changing the final expected value.

Problem: the expected ratio is close but not exactly equal -> Use integer counter increments that produce an exact binary-friendly result where possible, or set fuzzy_compare: true at the test-file root when only the final floating-point mantissa bit differs. Do not use fuzzy comparison to hide a visible discrepancy such as 0.949 versus 0.95.

Problem: an alert expression is true but exp_alerts is empty -> Account for the for duration. The alert first enters pending state and fires only after the condition remains active for the configured time. Move eval_time later, and include enough fixture samples to cover the range vector plus for.

Problem: expected alert labels do not match -> Prometheus preserves labels from the expression result and then applies labels from the alert rule. Include job, service, and severity in exp_labels. If an aggregation drops job, fix the PromQL or remove that label from the expectation based on the intended routing contract.

Problem: Compose starts but the checkout group is absent -> Verify the host filenames, read-only volume targets, and rule_files path. Run docker compose config, inspect docker compose logs prometheus, and call /api/v1/rules. A healthy server can still run with zero application rules if the configuration never references them.

Problem: tests pass while the production SLO is wrong -> Compare exporter metric names, status labels, histogram boundaries, scrape interval, and excluded traffic against the fixture contract. Unit tests validate the rules you wrote, not the instrumentation you meant to have. Generate controlled success and failure requests in staging and compare raw counters with the expected increments.

Interview Questions and Answers

Q: Why test Prometheus SLO rules instead of only reviewing PromQL?

A review can spot intent and maintainability problems, but synthetic time series prove exact evaluation behavior. Promtool tests also cover rule ordering, label output, alert annotations, and for timing. They turn an operational policy into a regression suite with a deterministic exit code.

Q: What is the difference between an SLI, SLO, and error budget?

An SLI is the measured ratio, such as non-5xx requests divided by all eligible requests. The SLO is the target for that indicator over a window, such as 99.9 percent across 30 days. The error budget is the allowed bad fraction, which is 0.1 percent in that example.

Q: Why use two windows in a burn-rate alert?

The long window filters brief noise and establishes material budget consumption. The short window verifies that the problem is still happening, which improves recovery behavior. Requiring both gives a better urgency signal than one hypersensitive or sluggish window.

Q: Why use a histogram bucket ratio for a latency objective?

A promise that 95 percent of requests finish within 500 ms is directly represented by the rate of the le="0.5" bucket divided by the count rate. A quantile estimates a percentile value and can be limited by bucket resolution. The bucket ratio matches the wording of the objective.

Q: What cases belong in an SLO rule suite?

Include healthy traffic, exact boundary behavior, a clear breach, pending versus firing alert timing, missing traffic, counter resets, and label variants that should aggregate. Add a regression fixture for every rule defect found in review or production. Each case should protect a distinct decision.

Q: Can promtool tests replace production alert tests?

No. Promtool proves expression and alert-rule behavior on supplied series. It does not prove scrape discovery, metric emission, remote-write delay, Alertmanager routing, notification templates, or receiver delivery. Validate those paths separately after deployment.

Common Mistakes

• Averaging per-instance error ratios instead of summing bad and total event rates before division. A tiny canary and a large production pool should not have equal weight.
• Testing only a green fixture. A suite that never sees a breach may pass even when its alert is unreachable.
• Treating 99.9 percent as the alert threshold for every five-minute window. SLO compliance and incident response use different windows and purposes.
• Using histogram_quantile for a threshold-proportion promise when a matching cumulative bucket already exists.
• Omitting alert labels and annotations from expectations, which allows routing contracts to drift unnoticed.
• Copying universal burn-rate multipliers without considering the objective window, traffic volume, and response policy.
• Dividing by synthetic gauge values even though production emits counters. Test inputs must respect monotonic counter behavior.
• Floating the latest container tag in CI. An unrelated tool upgrade should not silently change rule evaluation on a routine pull request.
• Adding high-cardinality labels to recorded SLO series. That multiplies storage cost and makes aggregate health harder to interpret.
• Declaring success after promtool passes without confirming live metrics share the tested names, labels, and histogram buckets.

Where To Go Next

Connect the unit-tested policy to evidence from a representative workload. Use the load testing guide to select traffic shape and environment controls, then compare the resulting request and histogram counters with these fixtures. When the SLO fails under load, follow the performance bottleneck investigation workflow instead of adjusting the objective until the build turns green.

Extend the suite one risk at a time. Useful next fixtures cover zero traffic, a counter reset, multiple instances, excluded health endpoints, 429 policy, and a latency bucket missing from one deployment. After deployment, query rule health from the Prometheus API, verify Alertmanager routing in a sandbox, and attach the rule-test job as a required check for changes to monitoring code.

Conclusion

Reliable SLO testing starts with an unambiguous event contract. Encode bad and total events in recording rules, test exact boundaries with promtool, verify alert duration and labels, load the rules into a real Prometheus server, and run the same pinned toolchain in CI. That sequence catches mathematical regressions before they become misleading dashboards or silent alerts.

The finished suite is intentionally small. Its value comes from specificity: a known input rate, a known error ratio, a known latency boundary, and a known firing time. Keep those expectations under review with the service objective, and add live instrumentation and delivery checks around them for a complete operational test strategy.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Detecting flaky tests with machine learning (2026)
• Generating API tests from OpenAPI with AI (2026)
• Generating Selenium tests with AI (2026)
• How to Detect Flaky Tests With OpenTelemetry Traces (2026)
• Accessibility testing with Cypress (2026)
• Accessibility testing with Playwright (2026)