Test Alert Routing With PagerDuty (2026)

TL;DR

Send controlled Events API v2 events through a sandbox Global Integration, then poll the Incidents API by unique dedup key and assert the exact service ID. Include positive routes, an unmatched catch-all case, repeated triggers, acknowledge, resolve, and failure-safe cleanup.

Key Takeaways

• Assert the destination service through the Incidents API because Events API acceptance does not prove routing.
• Use isolated services, test responders, scoped secrets, and explicit cleanup before creating live incidents.
• Carry stable routing data in the dedup key so trigger, acknowledge, and resolve events follow the same route.
• Give every run unique keys and repeat one key deliberately to verify open-alert deduplication.
• Test the catch-all path with a valid unmatched event and assert that it creates no incident.
• Compare immutable service IDs rather than display names or notification channels.
• Run external paging smoke tests manually or on a conservative schedule outside the fast merge gate.

To test alert routing with PagerDuty, send controlled Events API v2 payloads through one Global Integration, then query the Incidents API and assert the service ID, incident status, and deduplication key. A 202 Accepted response proves only that PagerDuty ingested the event. It does not prove that an orchestration rule selected the intended service.

This tutorial builds a safe routing contract for two sandbox services, Checkout and Platform. PagerDuty evaluates a stable prefix in dedup_key, routes each event to the matching service, and leaves an unknown prefix on the default catch-all path. You will verify the result with Bash, curl, and jq rather than trusting a notification or a green HTTP response.

Run the exercise in a dedicated test account or against test-only services whose responders expect the notifications. Never point these commands at a production integration key. If alert delivery is new territory, the DevOps roadmap for QA engineers supplies the monitoring and incident-response context around this focused contract test.

What You Will Build

You will create a small command-line suite that:

• sends valid trigger, acknowledge, and resolve events to https://events.pagerduty.com/v2/enqueue;
• routes checkout:<run-id> and platform:<run-id> keys to different PagerDuty services;
• proves the catch-all route creates no incident for unknown:<run-id>;
• polls the REST API until asynchronous processing becomes observable;
• confirms repeated triggers stay attached to one open incident; and
• resolves every test incident, including cleanup after a failed assertion.

The evidence layers answer different questions:

Treat the suite as a routing contract, not as one oversized end-to-end check. The contract boundary starts with a payload, passes through the Global Integration and route graph, and ends at a service-owned incident. Responder assignment, device delivery, and human acknowledgement remain valuable game-day checks, but they have different failure causes and owners. Separating those layers tells you whether to inspect the emitter, Event Orchestration, the service, or a user's notification rules.

This distinction matters for any event-driven integration. The end-to-end webhook testing guide uses the same principle: transport acceptance, downstream state, and user-visible delivery are separate assertions.

Prerequisites

Use PagerDuty Events API v2 and REST API v2. For the local commands, use GNU Bash 5.3.15, curl 8.21.0, and jq 1.8.2. The scripts rely on curl --fail-with-body, Bash arrays, and current jq object construction. Newer compatible patch releases are acceptable, but record the versions used by your CI runner.

You also need:

• a PagerDuty role allowed to create or edit Event Orchestrations;
• two test-only services with escalation policies assigned to test responders;
• a Global Integration and service routing, which PagerDuty makes available across current packages;
• an admin-provisioned read-only General Access REST API key; and
• the email address of the PagerDuty user associated with the REST request.

PagerDuty uses two credentials here for different trust boundaries. The 32-character Events integration key can ingest events but cannot read incidents. The General Access REST key authorizes the read-only verification calls and should not be placed in an event payload. Keep both out of command-line arguments where process listings could expose them. The From header identifies the PagerDuty user responsible for REST requests made with the account-level key.

Create the working directory and check the exact tools:

mkdir pagerduty-routing-contract
cd pagerduty-routing-contract

bash --version | head -n 1
curl --version | head -n 1
jq --version

Verification: confirm the output identifies Bash 5.3.15, curl 8.21.0, and jq-1.8.2. If your workstation ships an older Bash, install the required release and invoke these files with that binary. Do not continue with an unannounced production escalation policy.

Step 1: Create isolated PagerDuty test services

In PagerDuty, create QA Routing - Checkout and QA Routing - Platform. Give each service a distinct escalation policy so a wrong destination is visible. The policy may target the same consenting test user, but separate policies make ownership assertions clearer. Keep the services enabled because a disabled or unstaffed service can change incident behavior.

Record each service's alert-grouping, urgency, acknowledgement-timeout, and auto-resolve settings with the test evidence. The status checks assume a new matched event remains triggered until this suite acknowledges it, so an automatic workflow that acknowledges immediately will produce a legitimate mismatch. Set auto-resolve longer than the test's two-minute polling window. Unique dedup keys prevent ordinary cross-run grouping, but a content-based grouping rule can still combine alerts and deserves an explicit test if production uses it.

Copy each service ID from its URL or REST representation. Create a read-only General Access REST API key for the verification calls. Enter the values interactively so they do not appear in shell history:

umask 077
read -rsp "PagerDuty REST API token: " PD_API_TOKEN
printf '\n'
read -rp "PagerDuty user email: " PD_FROM_EMAIL
read -rp "Checkout service ID: " PD_CHECKOUT_SERVICE_ID
read -rp "Platform service ID: " PD_PLATFORM_SERVICE_ID

export PD_API_TOKEN PD_FROM_EMAIL
export PD_CHECKOUT_SERVICE_ID PD_PLATFORM_SERVICE_ID
declare -px PD_API_TOKEN PD_FROM_EMAIL \
  PD_CHECKOUT_SERVICE_ID PD_PLATFORM_SERVICE_ID \
  > pagerduty-test.env
chmod 600 pagerduty-test.env

Treat both the REST token and later integration key as secrets. Add pagerduty-test.env, routing-run.env, receipts, and result files to the sample project's ignore rules before using a shared repository.

Verify that both IDs exist and are readable:

source ./pagerduty-test.env

for service_id in \
  "$PD_CHECKOUT_SERVICE_ID" \
  "$PD_PLATFORM_SERVICE_ID"
do
  curl --silent --show-error --fail-with-body \
    --header "Authorization: Token token=$PD_API_TOKEN" \
    --header 'Accept: application/vnd.pagerduty+json;version=2' \
    --header "From: $PD_FROM_EMAIL" \
    "https://api.pagerduty.com/services/$service_id"
done | jq -s -e \
  --arg checkout "$PD_CHECKOUT_SERVICE_ID" \
  --arg platform "$PD_PLATFORM_SERVICE_ID" \
  'map(.service.id) | sort == ([$checkout, $platform] | sort)'

Verification: jq prints true and exits with status 0. A 401 points to the token, while a 404 usually means an ID belongs to another PagerDuty account.

Step 2: Configure Event Orchestration routing rules

Open AIOps -> Event Orchestration, create an orchestration named QA Alert Routing Contract, and use its default Global Integration. In Service Routes, add these rules in order:

Use the PCL editor so the condition text is reviewable and can be copied into change evidence. Anchor the regular expressions at the beginning and include the colon delimiter. A looser pattern such as checkout could accidentally accept checkout-old: or not-checkout:. Before saving, review four boundaries: a valid Checkout key, a valid Platform key, an unknown prefix, and a near miss such as checkoutx:. This tutorial automates the first three; add the near miss when a routing defect has previously involved regex breadth.

The rules are intentionally mutually exclusive. PagerDuty evaluates ordered routing logic, so overlapping conditions make the winning rule depend on position instead of ownership. A precise prefix also gives application teams a versionable contract: changing checkout: is a breaking integration change, while changing the human-readable summary is not.

Leave the catch-all behavior at its default. An unmatched trigger is retained as a suppressed alert but is not sent to a service and never creates an incident. Do not use a catch-all production service for this test because a miss would look like a pass somewhere else.

Routing on dedup_key is deliberate. PagerDuty evaluates trigger, acknowledge, and resolve events against service routes. All three actions carry the same key, so checkout:... continues to match the Checkout rule even though acknowledge and resolve requests do not need a payload. Conditions based only on payload.custom_details.team require equivalent routing data on every follow-up action.

Copy the Global Integration key without printing it:

source ./pagerduty-test.env
read -rsp "PagerDuty Global Integration key: " PD_ROUTING_KEY
printf '\n'
export PD_ROUTING_KEY
declare -px PD_ROUTING_KEY >> pagerduty-test.env

test "${#PD_ROUTING_KEY}" -eq 32
printf 'Integration key format accepted\n'

Verification: the command prints Integration key format accepted. The length check protects against copying an orchestration email address or truncated value; the live route is tested in Step 4.

Step 3: Build a reusable Events API v2 client

Save the following as pd-event.sh. Trigger events include the three required Common Event Format fields: summary, source, and severity. Acknowledge and resolve events send only the global routing key, action, and original dedup key.

#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "$0")/pagerduty-test.env"
: "${PD_ROUTING_KEY:?missing PD_ROUTING_KEY}"

action=${1:?usage: pd-event.sh ACTION DEDUP_KEY [SEVERITY] [SUMMARY]}
dedup_key=${2:?missing dedup key}

case "$action" in
  trigger)
    severity=${3:-warning}
    summary=${4:-QA routing contract test}
    case "$severity" in
      critical|error|warning|info) ;;
      *) printf 'invalid severity: %s\n' "$severity" >&2; exit 64 ;;
    esac
    team=${dedup_key%%:*}
    request=$(jq -n \
      --arg routing_key "$PD_ROUTING_KEY" \
      --arg dedup_key "$dedup_key" \
      --arg summary "$summary" \
      --arg severity "$severity" \
      --arg team "$team" \
      '{
        routing_key: $routing_key,
        event_action: "trigger",
        dedup_key: $dedup_key,
        payload: {
          summary: $summary,
          source: "qa-routing-contract",
          severity: $severity,
          component: "alert-router",
          group: "qa-sandbox",
          class: "routing-contract",
          custom_details: {
            team: $team,
            test_key: $dedup_key
          }
        }
      }')
    ;;
  acknowledge|resolve)
    request=$(jq -n \
      --arg routing_key "$PD_ROUTING_KEY" \
      --arg action "$action" \
      --arg dedup_key "$dedup_key" \
      '{
        routing_key: $routing_key,
        event_action: $action,
        dedup_key: $dedup_key
      }')
    ;;
  *)
    printf 'invalid action: %s\n' "$action" >&2
    exit 64
    ;;
esac

response_file=$(mktemp)
trap 'rm -f "$response_file"' EXIT

http_code=$(curl --silent --show-error --fail-with-body \
  --retry 4 --retry-delay 2 --retry-max-time 30 \
  --header 'Content-Type: application/json' \
  --request POST \
  --data "$request" \
  --output "$response_file" \
  --write-out '%{http_code}' \
  'https://events.pagerduty.com/v2/enqueue')

test "$http_code" = '202'
jq -ce --arg key "$dedup_key" '
  if .status == "success" and .dedup_key == $key
  then {status, message, dedup_key}
  else error("unexpected Events API response")
  end
' "$response_file"

Using jq variables matters because summaries and keys can contain whitespace, quotes, or backslashes. Hand-built JSON may pass a simple sample and fail on the first realistic diagnostic message. PagerDuty permits a dedup key up to 255 characters; the short team prefix and generated run ID stay well below that limit. The temporary response file lets the client assert HTTP 202 and the JSON receipt separately, then the exit trap removes it on success or failure.

The retry flags cover transient HTTP failures such as 429 and selected 5xx responses. They do not turn a malformed 400 payload into a success. The response body check also prevents a syntactically valid but unexpected API response from becoming a green test.

Validate the script without sending an event:

chmod +x pd-event.sh
bash -n pd-event.sh

set +e
./pd-event.sh invalid checkout:syntax-check
status=$?
set -e

test "$status" -eq 64
printf 'Client validation passed\n'

Verification: Bash prints Client validation passed, and no PagerDuty event is created. This proves the file parses and the action allowlist rejects unknown operations before the network call.

Step 4: Test alert routing with PagerDuty across both services

Create one unique key per expected path. The prefix is routing data; the timestamp and Bash random value prevent an older resolved incident from contaminating the current result.

source ./pagerduty-test.env

RUN_ID=$(date -u +%Y%m%dT%H%M%SZ)-$RANDOM
CHECKOUT_KEY="checkout:$RUN_ID"
PLATFORM_KEY="platform:$RUN_ID"
UNKNOWN_KEY="unknown:$RUN_ID"
export RUN_ID CHECKOUT_KEY PLATFORM_KEY UNKNOWN_KEY
declare -px RUN_ID CHECKOUT_KEY PLATFORM_KEY UNKNOWN_KEY \
  > routing-run.env

./pd-event.sh trigger "$CHECKOUT_KEY" warning \
  "QA TEST: checkout route $RUN_ID" | tee checkout-receipt.json
./pd-event.sh trigger "$PLATFORM_KEY" warning \
  "QA TEST: platform route $RUN_ID" | tee platform-receipt.json
./pd-event.sh trigger "$UNKNOWN_KEY" info \
  "QA TEST: catch-all route $RUN_ID" | tee unknown-receipt.json

Warning is suitable for routable sandbox tests when the service maps it to low urgency. The unknown event uses info and should be suppressed by the catch-all. Service urgency and responder notification preferences still control what humans receive, so coordinate the run rather than assuming severity guarantees silence.

Write the expected outcomes before reading PagerDuty: Checkout creates one Checkout incident, Platform creates one Platform incident, and unknown creates none. A correctly routed incident with no device notification passes this routing scope but opens a separate delivery investigation. An incident from the unknown event fails even if it lands on a sandbox service, because the catch-all contract promises suppression rather than fallback ownership.

Check the three ingestion receipts:

jq -s -e \
  --arg checkout "$CHECKOUT_KEY" \
  --arg platform "$PLATFORM_KEY" \
  --arg unknown "$UNKNOWN_KEY" \
  'length == 3
   and all(.status == "success")
   and (map(.dedup_key) | sort
        == ([$checkout, $platform, $unknown] | sort))' \
  checkout-receipt.json platform-receipt.json unknown-receipt.json

Verification: jq returns true. Record this as the ingestion assertion only. PagerDuty processes Events API v2 asynchronously, so service routing requires the polling assertion in the next step.

Step 5: Verify each incident reached the expected service

Create pd-incidents.sh to query by the exact incident key. The REST API's incident_key filter also finds incidents whose child alert has the matching alert key.

#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "$0")/pagerduty-test.env"
key=${1:?usage: pd-incidents.sh DEDUP_KEY}

curl --silent --show-error --fail-with-body \
  --retry 4 --retry-delay 2 \
  --get 'https://api.pagerduty.com/incidents' \
  --header "Authorization: Token token=$PD_API_TOKEN" \
  --header 'Accept: application/vnd.pagerduty+json;version=2' \
  --header "From: $PD_FROM_EMAIL" \
  --data-urlencode "incident_key=$key" \
  | jq -c '{incidents: .incidents}'

Now add a bounded poller. It fails immediately if the same unique key returns multiple incidents or reaches the wrong service. It waits when the incident exists but has not reached the requested lifecycle state.

#!/usr/bin/env bash
set -euo pipefail

key=${1:?usage: wait-for-incident.sh KEY SERVICE_ID [STATUS]}
expected_service=${2:?missing expected service ID}
expected_status=${3:-triggered}

for ((attempt = 1; attempt <= 24; attempt++)); do
  result=$(./pd-incidents.sh "$key")
  count=$(jq '.incidents | length' <<<"$result")

  if ((count > 1)); then
    printf 'multiple incidents found for %s\n' "$key" >&2
    exit 1
  fi

  if ((count == 1)); then
    incident=$(jq -c '.incidents[0]' <<<"$result")
    actual_service=$(jq -r '.service.id' <<<"$incident")
    actual_status=$(jq -r '.status' <<<"$incident")

    if [[ "$actual_service" != "$expected_service" ]]; then
      printf 'route mismatch: expected %s, got %s\n' \
        "$expected_service" "$actual_service" >&2
      exit 1
    fi

    if [[ "$actual_status" == "$expected_status" ]]; then
      printf '%s\n' "$incident"
      exit 0
    fi
  fi

  sleep 5
done

printf 'timed out waiting for %s on %s in %s state\n' \
  "$key" "$expected_service" "$expected_status" >&2
exit 1

The poller allows up to 120 seconds because event acceptance and incident indexing are asynchronous. A fixed sleep on positive cases is either slow on healthy runs or flaky under load; polling stops as soon as the required state appears. It also reports a wrong service immediately instead of waiting until timeout.

The Incidents endpoint applies the incident_key filter server-side. When alerts are grouped, a parent incident can omit its own incident key even though a child alert has the requested key, so pd-incidents.sh deliberately trusts the filtered result instead of filtering the returned objects again. The negative check runs only after both same-run positive sentinels are visible, then adds a quiet period. That ordering gives evidence that PagerDuty was processing this batch before zero incidents is accepted.

Run the destination assertions and then check the negative route after a quiet period:

chmod +x pd-incidents.sh wait-for-incident.sh
source ./pagerduty-test.env
source ./routing-run.env

./wait-for-incident.sh \
  "$CHECKOUT_KEY" "$PD_CHECKOUT_SERVICE_ID" triggered \
  > checkout-incident.json
./wait-for-incident.sh \
  "$PLATFORM_KEY" "$PD_PLATFORM_SERVICE_ID" triggered \
  > platform-incident.json

sleep 20
./pd-incidents.sh "$UNKNOWN_KEY" \
  | jq -e '.incidents | length == 0'

jq -s -e \
  --arg checkout "$PD_CHECKOUT_SERVICE_ID" \
  --arg platform "$PD_PLATFORM_SERVICE_ID" \
  'length == 2
   and .[0].service.id == $checkout
   and .[1].service.id == $platform' \
  checkout-incident.json platform-incident.json

Verification: both jq commands print true. The two positive records prove distinct service destinations; the zero count proves that acceptance of the unknown event did not create an incident. This is stronger than checking which phone rang because responder settings can change independently of route selection.

Step 6: Prove deduplication and lifecycle routing

Send the Checkout trigger again with the identical key. PagerDuty should add a trigger entry to the existing open alert, not create another incident. Then acknowledge and resolve both positive cases through the same Global Integration.

source ./pagerduty-test.env
source ./routing-run.env

./pd-event.sh trigger "$CHECKOUT_KEY" warning \
  "QA TEST: repeated checkout signal $RUN_ID" \
  > checkout-repeat-receipt.json

./pd-incidents.sh "$CHECKOUT_KEY" \
  | jq -e '.incidents | length == 1'

./pd-event.sh acknowledge "$CHECKOUT_KEY" > checkout-ack.json
./pd-event.sh acknowledge "$PLATFORM_KEY" > platform-ack.json
./wait-for-incident.sh \
  "$CHECKOUT_KEY" "$PD_CHECKOUT_SERVICE_ID" acknowledged \
  > checkout-acknowledged.json
./wait-for-incident.sh \
  "$PLATFORM_KEY" "$PD_PLATFORM_SERVICE_ID" acknowledged \
  > platform-acknowledged.json

./pd-event.sh resolve "$CHECKOUT_KEY" > checkout-resolve.json
./pd-event.sh resolve "$PLATFORM_KEY" > platform-resolve.json
./pd-event.sh resolve "$UNKNOWN_KEY" > unknown-resolve.json
./wait-for-incident.sh \
  "$CHECKOUT_KEY" "$PD_CHECKOUT_SERVICE_ID" resolved \
  > checkout-resolved.json
./wait-for-incident.sh \
  "$PLATFORM_KEY" "$PD_PLATFORM_SERVICE_ID" resolved \
  > platform-resolved.json

jq -s -e '.[0].id == .[1].id' \
  checkout-incident.json checkout-resolved.json
jq -s -e '.[0].id == .[1].id' \
  platform-incident.json platform-resolved.json

Verification: the count assertion and both ID comparisons return true. Together, those checks protect three different invariants: one open incident after a duplicate trigger, one stable incident ID across lifecycle actions, and the expected status progression. Acknowledge stops further escalation for the open incident, while resolve closes it. Reusing the same dedup key after resolution would create a new incident on a later trigger, which is why each test run generates fresh keys.

Retries, duplicate deliveries, and lifecycle order are related risks. Expand those cases with the webhook retry and backoff testing guide and the event ordering and duplicate validation guide.

Step 7: Automate test alert routing with PagerDuty

Turn the manual checks into one repeatable smoke suite. Save routing-contract.sh beside the earlier scripts:

#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "$0")/pagerduty-test.env"
run_id=$(date -u +%Y%m%dT%H%M%SZ)-$RANDOM
unknown_key="unknown:$run_id"
opened=()

cleanup() {
  set +e
  for key in "${opened[@]}"; do
    ./pd-event.sh resolve "$key" >/dev/null
  done
  ./pd-event.sh resolve "$unknown_key" >/dev/null
}
trap cleanup EXIT

: > routing-results.jsonl

for spec in \
  "checkout|$PD_CHECKOUT_SERVICE_ID" \
  "platform|$PD_PLATFORM_SERVICE_ID"
do
  IFS='|' read -r team expected_service <<<"$spec"
  key="$team:$run_id"

  ./pd-event.sh trigger "$key" warning \
    "QA TEST: automated $team route $run_id" >/dev/null
  opened+=("$key")

  incident=$(./wait-for-incident.sh \
    "$key" "$expected_service" triggered)
  jq -c --arg team "$team" \
    '{team: $team, outcome: "routed",
      incidentId: .id, serviceId: .service.id}' \
    <<<"$incident" >> routing-results.jsonl
done

checkout_key="checkout:$run_id"
./pd-event.sh trigger "$checkout_key" warning \
  "QA TEST: automated duplicate $run_id" >/dev/null
./pd-incidents.sh "$checkout_key" \
  | jq -e '.incidents | length == 1' >/dev/null

./pd-event.sh trigger "$unknown_key" info \
  "QA TEST: automated catch-all $run_id" >/dev/null
sleep 20
unknown_count=$(./pd-incidents.sh "$unknown_key" \
  | jq '.incidents | length')
test "$unknown_count" -eq 0
jq -cn --arg key "$unknown_key" \
  '{team: "unknown", outcome: "no-incident",
    dedupKey: $key}' >> routing-results.jsonl

Run it only in the isolated environment:

chmod +x routing-contract.sh
./routing-contract.sh

jq -s -e '
  length == 3
  and ([.[] | select(.outcome == "routed")] | length == 2)
  and ([.[] | select(.outcome == "no-incident")] | length == 1)
' routing-results.jsonl

Verification: jq prints true, and the exit trap resolves both routed incidents even when a later assertion fails. Check the two sandbox services after the run and confirm there are no triggered or acknowledged test incidents.

The JSON Lines artifact contains case names, outcomes, incident IDs, and destination IDs without storing either credential. Preserve it with the job URL and orchestration revision so a later audit can reproduce what was tested. The random suffix prevents ordinary collisions but is not a concurrency lock. Configure the scheduler to allow one routing run at a time, because simultaneous exercises can confuse responders and make notification evidence ambiguous.

Use this suite as a manual dispatch or a low-frequency scheduled job, not as a live-paging check on every pull request. Store both PagerDuty secrets in the CI secret manager, restrict the job to the sandbox environment, prevent concurrent runs, and retain routing-results.jsonl as evidence. The test automation CI/CD guide explains how to place a stateful smoke test outside the fast merge gate.

Troubleshooting

Problem: Events API returns 400 -> Inspect the response body captured by --fail-with-body. Trigger requests require payload.summary, payload.source, and one of critical, error, warning, or info for payload.severity. Confirm jq produced an object rather than a quoted JSON string.

Problem: Events API returns 202 but no positive incident appears -> Verify the Global Integration key belongs to the orchestration you edited, inspect recent orchestration events, and compare the exact dedup prefix with the route condition. A 202 receipt precedes routing and can coexist with a catch-all suppressed alert.

Problem: the incident reaches the wrong service -> Review rule order and make the two regexes mutually exclusive. Query the incident's service.id instead of comparing display names, which administrators can rename without changing identity.

Problem: acknowledge or resolve is accepted but the incident stays open -> Reuse both the original routing_key and exact case-sensitive dedup_key. For global routing, ensure the follow-up event still satisfies a route; the prefix-based conditions in this tutorial were chosen for that reason.

Problem: the REST check returns 401 or 403 -> Regenerate the REST key, confirm it belongs to the same account, preserve the Token token=... authorization format, and send the associated email in From. Do not substitute the 32-character Events integration key for a REST token.

Problem: one run reports multiple incidents -> Stop reusing run IDs, check whether the earlier incident was already resolved, and confirm every trigger in the current run uses the same Global Integration. A trigger with an old key after resolution starts a new incident rather than reopening the resolved record.

Interview Questions and Answers

Q: Why is a 202 response insufficient for PagerDuty routing validation?

Events API v2 is asynchronous. The receipt confirms ingestion, but orchestration evaluation and incident creation happen afterward. A strong test polls PagerDuty's system of record and asserts the incident's service ID.

Q: Why route this test on a dedup-key prefix?

The prefix survives across trigger, acknowledge, and resolve requests. That lets every lifecycle action satisfy the same Global Integration route without adding an unnecessary payload to follow-up events. The unique suffix prevents cross-run collisions.

Q: How would you test the catch-all rule?

Send a valid event whose key matches no explicit route, wait beyond the normal processing delay, and query incidents by that exact key. The assertion is zero incidents, while the Events API receipt remains successful because suppression occurs after ingestion.

Q: What is the difference between an alert and an incident in this test?

The dedup key identifies an alert and merges subsequent events into that open alert. PagerDuty associates the alert with an incident for response coordination. The REST query may find the parent incident through its child alert key, so the test asserts both correlation and destination.

Q: How do you avoid paging production responders during routing tests?

Use dedicated services, escalation policies, responders, and a Global Integration that cannot reach production services. Label summaries with QA TEST, select a controlled urgency policy, announce the exercise, and always resolve in an exit trap. Isolation is the primary safeguard.

Q: Where should this test run in CI?

Place it in a protected sandbox job invoked manually or on a conservative schedule. Serialize executions, inject scoped secrets at runtime, retain the result artifact, and keep it outside the fast unit-test gate. Live incident creation is valuable but stateful and externally visible.

Common Mistakes

• Treating HTTP acceptance as proof of the selected escalation path.
• Matching only a trigger-only custom field, then losing acknowledge and resolve events at the router.
• Reusing a resolved dedup key and interpreting the newly created incident as a duplicate.
• Comparing mutable service names instead of stable PagerDuty service IDs.
• Leaving the catch-all pointed at a real team, which converts a negative case into unwanted paging.
• Printing integration keys in logs or committing pagerduty-test.env.
• Running route tests concurrently with shared dedup keys.
• Checking only the notification channel and ignoring suppressed or wrongly grouped alerts.
• Omitting cleanup when a middle assertion fails.

Where To Go Next

Add contract cases for every production route condition: environment, region, tenant tier, severity, and source system. Give each case a unique key, an expected service ID, and a defined catch-all outcome. Keep the matrix small enough that one failure identifies a single routing rule.

Next, validate the source that generates these payloads. Use the GitHub Actions OIDC test-environment tutorial to reduce long-lived cloud credentials around your sandbox job, while keeping PagerDuty tokens scoped and stored as secrets. Practice the response flow separately in a coordinated game day so notification delivery, on-call assignment, acknowledgement timing, and escalation handoffs receive human verification.

Conclusion

To test alert routing with PagerDuty reliably, separate ingestion from routing evidence. Send deterministic Events API v2 events, poll incidents by unique dedup key, compare immutable service IDs, test the unmatched path, and carry the same route data through acknowledge and resolve.

The finished suite validates the operational contract without depending on guesswork from a phone notification. Keep it isolated, run it intentionally, resolve everything, and update the case matrix whenever Event Orchestration rules change.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• AI test data generation with Faker and LLMs (2026)
• Automating Jira to test case with n8n (2026)
• Building a test case generator with LangChain (2026)
• Connecting Claude to your test suite with MCP (2026)
• Generating test cases from a PRD with AI (2026)
• How to Test Browser Permissions With Playwright TypeScript (2026)