Publish Test Evidence as CI Artifacts

TL;DR

Generate evidence into one predictable directory, then upload it in a post-test step that runs on success or failure. Keep the test verdict authoritative, validate expected files before upload, redact secrets, and set retention according to diagnostic value and data sensitivity.

Key Takeaways

• Publish reports and diagnostics even when tests fail.
• Preserve the test command exit code while running artifact upload in a guaranteed post-test step.
• Use predictable per-run directories and collision-free artifact names.
• Upload compact evidence by default, with richer traces and videos for failures.
• Treat test artifacts as sensitive data with redaction, access control, and deliberate retention.
• Verify artifact presence, contents, and failure behavior with controlled test cases.

To publish test evidence CI artifacts reliably, make evidence creation and upload separate pipeline responsibilities. The test runner writes machine-readable results and diagnostics, while the CI job uploads them even after a test failure without replacing the real test verdict.

This tutorial builds that pattern with Playwright, GitHub Actions, and GitLab CI. It complements the Test Automation CI/CD Complete Guide for 2026, which explains the wider pipeline design around triggers, environments, quality gates, and feedback.

You will finish with downloadable HTML, JUnit XML, screenshots, traces, and logs whose names, retention, access, and failure behavior are explicit. The examples use supported runner and CI features, so you can paste them into a current TypeScript project.

TL;DR: Publish Test Evidence CI Artifacts

Create one test-results/ tree, run tests, validate that the expected outputs exist, and upload the tree with an if: always() or when: always rule. Never use || true on the test command. It makes a broken suite look green.

What You Will Build

By the end, you will have:

• A Playwright configuration that produces HTML, JUnit XML, traces, screenshots, and failure videos.
• A small smoke test that proves both passing and failing evidence paths.
• A GitHub Actions workflow that uploads evidence on every terminal test outcome.
• A GitLab CI job with equivalent artifact and JUnit report behavior.
• Validation, redaction, retention, and naming controls suitable for shared CI.

The final artifact is useful without a vendor dashboard. A developer can download it, open the HTML report, map a failure to its JUnit case, and inspect the trace locally.

Prerequisites

Use Node.js 22 LTS or a currently supported Node.js release, npm, Git, and a repository hosted in GitHub or GitLab. The examples use Playwright Test and Ubuntu runners.

Install the project dependencies:

npm init -y
npm install --save-dev @playwright/test typescript
npx playwright install --with-deps chromium

Confirm the tools:

node --version
npx playwright --version

You also need permission to edit CI workflow files and read job artifacts. If tests call protected environments, prefer short-lived identity such as the pattern in GitHub Actions OIDC test environments tutorial. Do not place cloud credentials inside uploaded logs or reports.

Step 1: Define the Evidence Contract

Before configuring a reporter, define which files the job promises to publish. A simple contract prevents a pipeline from reporting a successful upload when the test runner wrote nothing.

Create this directory model conceptually:

test-results/
  junit.xml
  html/
    index.html
  raw/
    <test-specific files>

Use junit.xml for machines, html/ for people, and raw/ for traces, screenshots, and videos. Keep everything beneath one root so the upload step cannot accidentally archive the repository, dependency cache, .env file, or home directory.

Choose a stable identity for the artifact. Include the workflow, operating system or browser when a matrix exists, and the CI run or job ID. Do not include branch names without sanitizing them because slashes and spaces produce awkward paths. CI systems already associate an artifact with its run, so avoid duplicating excessive metadata in every filename.

Verification: Review the contract with one developer and one person who investigates failures. They should be able to answer where the JUnit file lives, which directory contains a trace, and whether evidence appears when tests fail. Add those paths to the workflow instead of using a broad glob such as **/*.

Step 2: Configure Playwright Evidence

Create playwright.config.ts:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  outputDir: 'test-results/raw',
  retries: process.env.CI ? 1 : 0,
  reporter: [
    ['line'],
    ['junit', { outputFile: 'test-results/junit.xml' }],
    ['html', { outputFolder: 'test-results/html', open: 'never' }]
  ],
  use: {
    baseURL: process.env.BASE_URL || 'https://example.com',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure'
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } }
  ]
});

The line reporter keeps the job log readable. JUnit provides structured status for CI. HTML gives engineers a navigable report. Playwright stores attachments beneath outputDir and avoids collisions by creating test-specific output directories.

trace: 'on-first-retry' controls evidence volume. In CI, a failed first attempt triggers the retry and captures a trace. If your policy prohibits retries, use retain-on-failure, but expect larger output because tracing starts for every test.

Verification: Run npx playwright test and then execute:

test -f test-results/junit.xml
test -f test-results/html/index.html
find test-results -maxdepth 3 -type f -print

Both test commands should exit zero. The final command should list the XML and HTML entry point. A passing suite may have no screenshot, trace, or video, which is correct under the chosen policy.

Step 3: Add Controlled Pass and Failure Tests

Create tests/evidence.spec.ts:

import { test, expect } from '@playwright/test';

test('publishes evidence for a passing check', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/Example Domain/);
});

test('controlled failure for pipeline verification', async ({ page }) => {
  test.skip(process.env.EVIDENCE_FAILURE !== '1', 'Enable only for evidence checks');
  await page.goto('/');
  await expect(page.getByRole('heading')).toHaveText('Wrong heading');
});

The failure is opt-in, so normal builds remain green. When enabled, it creates a screenshot and retained video. Because CI retries once, it also creates a trace for the retry. The assertion uses an actual locator and produces a useful expected-versus-received message.

Run both modes:

rm -rf test-results
npx playwright test
rm -rf test-results
EVIDENCE_FAILURE=1 CI=true npx playwright test

The first command sequence should exit zero. The second test command should exit nonzero after the failed retry.

Verification: After the controlled failure, run:

test -f test-results/junit.xml
test -f test-results/html/index.html
find test-results/raw -type f \( -name '*.zip' -o -name '*.png' -o -name '*.webm' \) -print

Expect XML and HTML files plus failure media. Open a trace locally with npx playwright show-trace path/to/trace.zip. Disable EVIDENCE_FAILURE after proving the pipeline.

Step 4: Validate and Sanitize Before Upload

A successful upload of empty or unsafe output is not useful. Add scripts/verify-test-evidence.mjs:

import { access, readFile } from 'node:fs/promises';
import { constants } from 'node:fs';

const required = [
  'test-results/junit.xml',
  'test-results/html/index.html'
];

for (const file of required) {
  await access(file, constants.R_OK);
}

const xml = await readFile('test-results/junit.xml', 'utf8');
if (!xml.includes('<testsuite') && !xml.includes('<testsuites')) {
  throw new Error('JUnit output has no test suite element');
}

const forbidden = [
  /authorization:\s*bearer\s+[^<\s]+/i,
  /-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----/,
  /AWS_SECRET_ACCESS_KEY\s*=/
];

for (const pattern of forbidden) {
  if (pattern.test(xml)) throw new Error(`Sensitive pattern found: ${pattern}`);
}

console.log('Required test evidence is readable and structurally valid');

This check deliberately scans the JUnit file, where error messages and test names can leak data. Extend scanning to HTML and text logs with a maintained secret scanner when your organization has one. Do not attempt to sanitize binary traces after creation. Prevent secrets from entering browser storage, request headers, console messages, and test titles, or restrict trace capture for sensitive flows.

Run the validator after Playwright finishes. Since a failing test returns nonzero, the CI workflow needs separate test, validate, and upload steps with unconditional execution. Validation should report an evidence problem, but it must not turn a failed test into a pass.

Verification: Run node scripts/verify-test-evidence.mjs. It should print the success message. Temporarily rename junit.xml and rerun it. The command should fail, proving that missing evidence is visible. Restore the file before continuing.

Step 5: Publish Test Evidence CI Artifacts in GitHub Actions

Create .github/workflows/e2e.yml:

name: e2e

on:
  pull_request:
  workflow_dispatch:
    inputs:
      verify_failure_path:
        description: Run the controlled failure
        required: false
        default: false
        type: boolean

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      CI: 'true'
      EVIDENCE_FAILURE: ${{ inputs.verify_failure_path && '1' || '0' }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run end-to-end tests
        run: npx playwright test
      - name: Validate evidence
        if: ${{ always() }}
        run: node scripts/verify-test-evidence.mjs
      - name: Upload test evidence
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-chromium-${{ github.run_id }}-${{ github.run_attempt }}
          path: test-results/
          if-no-files-found: error
          retention-days: 14
          compression-level: 6

GitHub evaluates each step in order. always() makes validation and upload run after success, failure, or cancellation when the runner can still execute steps. The original test step remains failed, so the job remains failed. if-no-files-found: error prevents a reassuring but empty artifact operation.

Pin actions to full commit SHAs if your supply-chain policy requires immutable references. Keep permissions minimal and do not expose secrets to workflows from untrusted pull requests. Artifact access follows repository and Actions permissions, so private test data still requires a deliberate audience and retention policy.

Verification: Push the workflow and run it normally. Download the artifact from the workflow run and open html/index.html. Then dispatch it with verify_failure_path enabled. Confirm the job is red, the artifact still exists, JUnit records the failure, and the trace opens.

Step 6: Publish Equivalent Evidence in GitLab CI

Create or extend .gitlab-ci.yml:

stages:
  - test

e2e:
  stage: test
  image: mcr.microsoft.com/playwright:v1.55.0-noble
  variables:
    CI: 'true'
    EVIDENCE_FAILURE: '0'
  before_script:
    - npm ci
  script:
    - npx playwright test
  after_script:
    - node scripts/verify-test-evidence.mjs
  artifacts:
    when: always
    name: e2e-$CI_JOB_ID-$CI_JOB_RETRY
    expire_in: 14 days
    paths:
      - test-results/
    reports:
      junit: test-results/junit.xml

Match the Playwright container tag to the package version pinned in package-lock.json. A version mismatch can require browser binaries that the image does not contain. Periodically update both together and run the controlled failure test.

GitLab uploads paths for download and ingests the JUnit file as a test report. when: always publishes after pass or failure. The after_script runs in a separate shell context, so do not depend on shell variables exported only inside script. Files in the working directory remain available.

Verification: Run the pipeline with the default variable and confirm the Tests view contains the passing case. Start a manual pipeline with EVIDENCE_FAILURE=1. Confirm the job fails, the artifact browser contains HTML and raw diagnostics, and the Tests view shows the failed assertion. For complex suite separation, see GitLab CI child pipelines for test suites.

Step 7: Add Matrix-Safe Naming and Merge Rules

A browser or shard matrix needs one artifact per producer. Never let several jobs publish the same generic name and assume the last upload represents the complete suite.

For GitHub Actions, add a matrix and include its dimensions:

strategy:
  fail-fast: false
  matrix:
    browser: [chromium, firefox]
steps:
  - name: Run tests
    run: npx playwright test --project=${{ matrix.browser }}
  - name: Upload evidence
    if: ${{ always() }}
    uses: actions/upload-artifact@v4
    with:
      name: e2e-${{ matrix.browser }}-${{ github.run_id }}-${{ github.run_attempt }}
      path: test-results/
      if-no-files-found: error
      retention-days: 14

If you shard, include ${{ strategy.job-index }} or the explicit shard number in both the artifact name and report path. A later aggregation job can download all artifacts and merge supported report formats. Do not concatenate JUnit XML blindly because multiple XML declarations and root elements produce invalid XML. Use a maintained JUnit merge tool or preserve individual files for CI-native ingestion.

Set fail-fast: false so one failed matrix job does not cancel other evidence producers. Cancellation can be valid for expensive suites, but the final report must say which scopes never ran. Missing results are not passing results.

Verification: Intentionally fail only Firefox. Confirm Chromium completes, both artifacts have unique names, and the workflow remains failed. Count expected producers against downloaded artifacts. If two browsers and three shards run, expect six artifact sets unless a job never started.

Step 8: Set Retention, Access, and Cost Controls

Retention should follow diagnostic value and sensitivity, not habit. JUnit and small HTML reports can survive longer than videos and traces, but one combined artifact has one retention setting. Split lightweight reports from heavy diagnostics if storage pressure or policy requires different lifetimes.

Use the shortest period that covers normal triage. Restrict repository membership, protected environments, and project artifact access. Test reports may contain customer names, internal URLs, email addresses, cookies, source paths, request bodies, and screenshots of private screens. Prefer synthetic data and redact at the source.

Avoid uploading dependency directories, browser caches, or the whole workspace. Compression consumes CPU and may barely reduce already compressed .zip, .png, and .webm files. Tune only after measuring your own runs. Do not invent a universal retention number or size limit.

Verification: Inspect the artifact as a user with the lowest intended access. Confirm unauthorized users cannot read it. Check the displayed expiration date, archive contents, compressed size, and download time. Search extracted text files for a planted fake secret, then confirm the pipeline blocks it. Never use a real credential for this test.

Troubleshooting

Problem: The test fails and no artifact appears -> Make the upload unconditional with if: ${{ always() }} in GitHub Actions or artifacts: when: always in GitLab. Confirm the test process writes inside the repository workspace before it exits.

Problem: The job becomes green after failed tests -> Remove || true, continue-on-error, or shell error suppression from the test step. Let the test command return its real exit code and put upload in a separate unconditional step.

Problem: The artifact exists but is empty -> Use an exact root path, add if-no-files-found: error where supported, and run the validator. Check whether reporter paths are relative to a changed working directory.

Problem: Traces or screenshots are missing -> Check the Playwright use policy and whether the scenario actually failed or retried. A trace configured as on-first-retry does not exist for a passing test or a failed run with retries disabled.

Problem: Matrix artifacts overwrite or cannot merge -> Put browser, shard, run ID, and attempt in names. Preserve each JUnit file or merge it with a format-aware tool. Never use one writable directory across concurrent jobs.

Problem: Evidence leaks tokens or personal data -> Stop publishing affected runs, revoke exposed credentials, shorten or delete artifacts according to platform capabilities, and fix collection at the source. Add sentinel-secret checks and review screenshots, traces, videos, logs, and HTML, not only JUnit XML.

Flaky tests create misleading evidence even when publishing works. Apply the ownership and exit criteria in quarantine flaky tests in a CI pipeline rather than hiding the failure with retries.

Where To Go Next

Return to the complete test automation CI/CD guide to connect artifacts to triggers, gates, environments, and deployment decisions.

Then deepen the parts that match your platform:

• Use GitHub Actions OIDC for test environments to replace long-lived cloud secrets with short-lived credentials.
• Split large GitLab portfolios with GitLab CI child pipelines for test suites while preserving evidence from every child.
• Establish an accountable flaky test quarantine workflow so retries and quarantine do not corrupt release confidence.
• Improve result portability with Playwright HTML report practices and JUnit testing fundamentals.

Start by enabling the controlled failure on a manual run. A pipeline is not evidence-ready until you have watched it fail, preserve the correct verdict, and publish enough information to explain that failure.

Interview Questions and Answers

Q: Why should artifact upload be separate from test execution?

A separate unconditional step can run after the test command returns nonzero. This preserves the true test status while still publishing diagnostics. It also makes upload errors distinguishable from test failures.

Q: What evidence should every CI test job publish?

Publish a machine-readable result such as JUnit XML and enough human-readable context to investigate. Add screenshots, traces, videos, and sanitized logs according to failure type and cost. The exact contract should name paths and expected consumers.

Q: How do you prevent artifacts from hiding a failed test?

Do not suppress the test exit code. Run upload under the CI platform's unconditional post-step rule and let the earlier failed step keep the job failed. Verify this with a controlled assertion failure.

Q: How do you handle parallel jobs?

Give each browser, shard, job, run, and retry a collision-free artifact identity. Publish independently, then reconcile expected producers and merge only with format-aware tooling. A missing shard must remain visible.

Q: What security risks exist in test artifacts?

Reports can contain tokens, cookies, headers, personal data, screenshots, internal URLs, and source paths. Use synthetic data, prevent sensitive capture, scan text outputs, restrict access, and expire artifacts deliberately.

Q: When should traces and videos be retained?

Retain them when their diagnostic value justifies storage and exposure. Failure-only or failed-retry policies are strong defaults for many suites. Sensitive workflows may require narrower capture or no media.

Q: How do you prove artifact publishing is reliable?

Exercise pass, failure, retry, cancellation, missing-file, matrix, and secret-sentinel cases. Confirm the job verdict, expected artifact count, report structure, media readability, access, and expiration each time.

Best Practices

• Keep all outputs below one dedicated evidence root.
• Publish a machine-readable result and a human-readable report.
• Preserve the original test exit code.
• Make upload run after both success and failure.
• Fail visibly when required evidence is missing.
• Use unique names for runs, attempts, browsers, and shards.
• Capture heavy diagnostics only when useful.
• Redact before serialization and use synthetic test data.
• Set explicit access and retention rules.
• Test the failure path regularly, not only during initial setup.

Conclusion

To publish test evidence CI artifacts well, define a narrow evidence contract, configure the runner to populate it, validate required files, and upload from an unconditional CI step. The test command must remain the authority for the job verdict.

Run the controlled failure now. Confirm the pipeline stays red while its report, screenshot, video, and trace remain downloadable to the intended audience. That single exercise proves more than a green pipeline with an untested upload step.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• GitLab CI for test automation: Step by Step (2026)
• How to Add CI to a test framework (2026)
• Selenium Test Web Components Slots Tutorial: Test Web Component Slots with Selenium
• Test scenarios vs test cases (2026)
• TypeScript Decorators Test Metadata Tutorial: Create Test Metadata
• A/B test validation: A Complete Guide for QA (2026)