Allure report in CI: Step by Step (2026)

TL;DR

To create an Allure report in CI, configure the test-framework adapter to write `allure-results`, run the tests, execute `allure generate allure-results --clean -o allure-report`, and upload or publish `allure-report`. Make report steps run after test failure, but add a final gate that preserves the test command's original nonzero status.

Key Takeaways

• A framework adapter writes raw allure-results, then the Allure CLI converts those files into a separate HTML report directory.
• Install and lock the adapter and report generator independently because they solve different pipeline stages.
• Generate and upload the report even after tests fail, then restore the original failing job status.
• Clean result directories before each logical run and merge shard outputs only after every shard finishes.
• Persist the prior report's history folder when trend charts and retry history are required across builds.
• Attach concise, redacted evidence and keep secrets, tokens, customer data, and oversized payloads out of reports.
• Treat the HTML report as a diagnostic artifact, while test status and explicit quality policy remain the CI gate.

An Allure report in CI should be available precisely when tests fail. The reliable pipeline has two independent stages: a framework adapter writes raw result files during execution, then the Allure command-line tool converts those files into a static HTML report. CI preserves that report as an artifact or publishes it, while the job still fails when the test command fails.

The common broken setup generates a beautiful report only on green builds, or marks the pipeline green because the report command was the last successful process. This guide fixes both problems. It uses pytest and allure-pytest for a runnable example, plus the Allure command-line package installed through npm. The same architecture applies to JUnit, TestNG, Cypress, Playwright, and other supported adapters, with framework-specific result configuration.

Allure Report 2 remains the mature line with broad integrations, while Allure Report 3 is the newer line and has features such as its own GitHub Action summary integration. Do not assume every plugin, configuration file, or action is interchangeable across those lines. Pin a supported toolchain and upgrade it deliberately.

TL;DR

rm -rf allure-results allure-report
python -m pytest --alluredir=allure-results
npx allure generate allure-results --clean -o allure-report

The three shell commands show the data flow but need status handling in CI. A direct sequential shell stops after pytest fails, so the complete workflow later in this guide captures the failure, generates the report, uploads it, and then fails the job.

1. What an Allure Report in CI Contains

Allure is not the test runner. A framework integration observes test execution and writes structured files into a results directory. Those files can include test status, steps, labels, parameters, links, attachments, container relationships, and fixture information. The Allure CLI reads that directory and produces a browsable static report.

Keep the three artifacts conceptually separate:

The result directory is not a ready-to-browse report. Uploading only allure-results can be useful for later generation, but it does not give a reviewer the normal Allure interface. Conversely, an HTML report cannot repair missing metadata that the adapter never emitted.

Generate one report from one logical test run. If stale result files remain from earlier executions, the report can display tests that did not run or combine incompatible environments. Clean the current results directory before execution, then restore only intentional history data.

Allure status should support, not replace, CI semantics. A test process must still return a nonzero code for failure according to the team's policy. The report provides diagnosis and trends. It should never be used as a reason to ignore the runner exit code.

2. Choose and Pin the Adapter and Generator

The adapter and generator have separate release cycles. allure-pytest integrates with pytest and writes results. allure-commandline provides the allure executable and requires a Java runtime for the Allure 2 CLI it distributes. Lock both through the project's normal dependency systems.

A small Python project can use these files:

project/
├── package.json
├── package-lock.json
├── requirements.txt
├── tests/
│   └── test_orders.py
├── allure-results/       # generated, ignored
└── allure-report/        # generated, ignored

requirements.txt can pin compatible versions selected by the project:

pytest==8.4.1
allure-pytest==2.15.0
requests==2.32.4

The versions are illustrative for a locked example. Confirm compatibility and resolve current patched versions when installing. Do not copy pins over a repository's reviewed lock or constraint policy.

Install the report generator once and commit the resolved npm lockfile:

npm install --save-dev allure-commandline

Then expose scripts without hiding paths:

{
  "private": true,
  "scripts": {
    "test:allure": "python -m pytest --alluredir=allure-results",
    "allure:generate": "allure generate allure-results --clean -o allure-report",
    "allure:open": "allure open allure-report"
  },
  "devDependencies": {
    "allure-commandline": "^2.34.0"
  }
}

The exact resolved version belongs in package-lock.json. Use allure open for an already generated local report. allure serve can generate and open a temporary report locally, but it is not a CI publication strategy.

3. Create a Runnable pytest Allure Example

Install the Python dependencies in an isolated environment, then write tests with useful hierarchy and steps.

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

A self-contained example avoids external services:

# tests/test_orders.py
import allure
import pytest


@allure.epic("Commerce")
@allure.feature("Orders")
class TestOrderTotal:
    @allure.story("Price calculation")
    @allure.title("Calculate total for {quantity} items")
    @pytest.mark.parametrize(
        ("unit_price", "quantity", "expected"),
        [(12.5, 2, 25.0), (10.0, 0, 0.0), (7.25, 3, 21.75)],
    )
    def test_total(self, unit_price: float, quantity: int, expected: float) -> None:
        with allure.step("Calculate the order total"):
            actual = unit_price * quantity

        with allure.step("Verify the total"):
            assert actual == expected

Run it and generate HTML:

rm -rf allure-results allure-report
python -m pytest --alluredir=allure-results
npx allure generate allure-results --clean -o allure-report
npx allure open allure-report

The title placeholder uses pytest parameter names, and the hierarchy labels group results in Allure views. Steps should represent diagnostic business actions, not wrap every assertion mechanically. Excessive nested steps make a simple failure harder to scan.

For API or browser tests, add attachments only where they help investigation. Allure will already capture status and traceback data from the adapter. Do not duplicate entire runner logs in every test.

4. Build an Allure Report in CI Locally First

Before editing a pipeline, prove the exact commands on a clean checkout or container. A local pass confirms four separate things: the runner can discover tests, the adapter writes nonempty results, Java can run the report generator, and the generated directory contains index.html.

Use explicit checks in a diagnostic script or manual run:

rm -rf allure-results allure-report
python -m pytest --alluredir=allure-results
test -n "$(find allure-results -type f -print -quit)"
npx allure generate allure-results --clean -o allure-report
test -f allure-report/index.html

The sequence above stops when pytest fails in a shell configured with fail-fast behavior, so it is a success-path verification, not the final CI implementation. CI needs report generation after failure.

Add generated directories to .gitignore unless repository policy intentionally publishes static output from a dedicated branch:

allure-results/
allure-report/

Do not ignore source fixtures or configuration that the adapter needs. Keep paths consistent across local scripts, CI steps, artifact definitions, and history restoration. A small naming mismatch such as allure-result versus allure-results can produce an empty report without an obvious test error.

If the CLI reports that Java is unavailable, install a supported JDK and confirm java -version in the same environment. The npm package supplies the command wrapper, not an embedded replacement for every runtime dependency.

5. Add Useful Metadata and Safe Attachments

A report becomes valuable when it answers which feature failed, under which parameters, in which environment, and with what concise evidence. Use adapter APIs near the start of the test so metadata survives early failure.

import json
import allure


def attach_json(name: str, payload: dict) -> None:
    redacted = {
        key: ("[REDACTED]" if key.lower() in {"token", "password"} else value)
        for key, value in payload.items()
    }
    allure.attach(
        json.dumps(redacted, indent=2),
        name=name,
        attachment_type=allure.attachment_type.JSON,
    )


@allure.feature("Checkout API")
@allure.story("Create order")
def test_order_payload_contract() -> None:
    request_payload = {
        "sku": "BK-101",
        "quantity": 2,
        "token": "test-only-secret",
    }
    attach_json("Request payload", request_payload)
    assert request_payload["quantity"] > 0

The helper is intentionally simple. Production redaction should handle nested objects, headers, cookies, query strings, and known sensitive business fields. Prefer an allowlist of safe evidence when payload structures are large or unpredictable.

Write environment context to allure-results/environment.properties before generation when it helps reproduce failures:

Test.Environment=staging
Browser=chromium
Region=us-east
Commit=abc1234

Generate this file from non-secret CI variables. Do not include access tokens, database credentials, internal cookies, or personally identifiable data. A report artifact is often downloadable by more people than raw secret logs.

Use issue and test-case links only with trusted URLs and repository-defined link patterns. A report is untrusted test output and should not become a path for exposing private endpoints.

6. Generate an Allure Report in GitHub Actions

The workflow below runs pytest, generates the report even if pytest fails, uploads both raw results and HTML, then restores a failing status. It uses only official GitHub setup and artifact actions plus project dependencies.

name: Pytest with Allure

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.13'
          cache: pip

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '21'

      - name: Install locked dependencies
        run: |
          python -m pip install -r requirements.txt
          npm ci

      - name: Clean current results
        run: rm -rf allure-results allure-report

      - name: Run pytest
        id: tests
        continue-on-error: true
        run: python -m pytest --alluredir=allure-results

      - name: Generate Allure HTML
        if: always()
        run: npx allure generate allure-results --clean -o allure-report

      - name: Upload Allure HTML
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-report-${{ github.run_id }}
          path: allure-report
          if-no-files-found: error
          retention-days: 14

      - name: Upload raw Allure results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-results-${{ github.run_id }}
          path: allure-results
          if-no-files-found: error
          retention-days: 7

      - name: Preserve test failure
        if: steps.tests.outcome == 'failure'
        run: exit 1

continue-on-error applies only to the test step. It lets later steps execute, while steps.tests.outcome remains failure when pytest returned nonzero. The last step converts that outcome back into a failed job after artifacts exist.

Adjust supported Python, Node, and Java versions to repository policy. The workflow assumes locked requirements.txt and package-lock.json files.

7. Preserve Failure Status Without Losing the Report

Exit-code handling is the most important CI detail. A test command can return codes for failed tests, collection errors, interruption, or no tests collected. Unless policy says otherwise, preserve every nonzero runner result.

There are two sound patterns:

A GitLab CI job can use the one-shell pattern and let artifacts upload with when: always:

allure_tests:
  image: your-org/python-node-java-test-image:2026.07
  script:
    - rm -rf allure-results allure-report
    - |
      set +e
      python -m pytest --alluredir=allure-results
      test_status=$?
      set -e
      npx allure generate allure-results --clean -o allure-report
      exit $test_status
  artifacts:
    when: always
    expire_in: 14 days
    paths:
      - allure-report/
      - allure-results/

The custom image name is an explicit organizational placeholder. Build it from pinned runtimes and dependencies or replace it with approved setup steps.

Do not append || true to pytest without later restoring the status. Do not rely on the report generator's success code as the job result. Report generation proves that HTML was created, not that tests passed.

8. Store Artifacts or Publish a Static Report

Artifact upload is the safest default. It is access-controlled by the CI platform, has explicit retention, and does not expose every branch report at a public URL. Reviewers download the artifact and open index.html, or use a platform preview mechanism if available.

Static hosting is useful for stable links, trend navigation, and stakeholder access. It also introduces authorization, retention, branch collision, and content-security decisions. A report can contain screenshots, request data, internal hostnames, exception messages, and test account details. Never publish it publicly by accident.

Compare the choices:

For pull requests, use a path or artifact name containing run ID and commit so concurrent builds cannot overwrite each other. For main-branch publication, update an alias such as latest only after generation succeeds, while retaining immutable build-specific reports for audit.

The Allure vs Extent Reports guide can help teams choose a reporting model. Reporting is separate from execution distribution, which is covered in Selenium Grid vs Playwright sharding.

9. Preserve Allure History Across CI Runs

Allure trend widgets need history data from the previous generated report. For Allure 2, copy the prior report's history directory into the new allure-results/history directory before running allure generate.

rm -rf allure-report

if [ -d previous-allure-report/history ]; then
  mkdir -p allure-results/history
  cp -R previous-allure-report/history/. allure-results/history/
fi

npx allure generate allure-results --clean -o allure-report

The pipeline must download previous-allure-report from trusted durable storage first. After generating the new report, publish or upload its history directory for the next main-branch run. Use a serialized update or immutable build paths plus an atomic latest pointer so two concurrent jobs do not race.

Do not restore history into a dirty directory before tests. Clean current raw results, run the adapter, then add only the history folder. Never copy every prior result file into the new results directory, since that makes old tests look like part of the current execution.

History identifiers depend on adapter and test identity. Renaming tests, changing parameter representation, or moving framework labels can break continuity. Treat large hierarchy changes as reporting migrations and explain the expected trend reset.

Keep history only where it has meaning. Pull request branches may compare with the target branch baseline but should not all update one shared history concurrently. A protected main or scheduled pipeline is usually the owner of canonical trends.

10. Merge Parallel Shard Results Safely

Parallel workers must never write into one shared network directory concurrently unless the adapter explicitly guarantees it. Give each shard a unique local results path, upload it, and merge after all shards finish. Allure result filenames are normally unique, so merging means copying each shard's files into one clean aggregate directory.

A conceptual CI flow is:

shard 1 -> allure-results-1 -> upload
shard 2 -> allure-results-2 -> upload
shard 3 -> allure-results-3 -> upload
                         |
                         v
merge job -> allure-results -> allure generate -> allure-report

The merge job can run:

rm -rf allure-results allure-report
mkdir -p allure-results
cp -R downloaded/allure-results-1/. allure-results/
cp -R downloaded/allure-results-2/. allure-results/
cp -R downloaded/allure-results-3/. allure-results/
npx allure generate allure-results --clean -o allure-report

Download paths and shard count come from the CI platform. Validate that every expected shard artifact exists before generation. If one shard crashes without producing results, a partial green-looking report is dangerous. The overall test gate must aggregate every shard outcome independently from the report generator.

Environment metadata can conflict during a merge. If shards run the same environment, generate one aggregate environment.properties file in the merge job. If they represent different browsers or regions, encode those dimensions as test parameters or labels rather than letting the last copied environment file win.

Retrying a failed shard can also create duplicate attempts. Confirm how the adapter and Allure version group retries, and keep CI attempt identifiers in metadata for diagnosis.

11. Control Security, Size, and Retention

Allure attachments are convenient enough to become a data leak. Browser screenshots can contain email addresses and tokens. API logs can contain authorization headers. Trace files, videos, and full response bodies can make one report several gigabytes. Define an evidence policy before broad rollout.

Use these controls:

• Redact authorization, cookies, passwords, tokens, session IDs, and sensitive business fields before attachment.
• Use fictional or generated test data, not copied production records.
• Attach response excerpts or schema failures rather than every successful payload.
• Capture screenshots on failure or at important checkpoints, not after every trivial step.
• Set artifact retention by branch and purpose. Pull request evidence can expire sooner than release evidence.
• Restrict report download and static hosting with the same care as test environment access.
• Scan generated artifacts if organizational policy requires secret detection or malware checks.
• Delete reports for removed branches and respect privacy deletion requirements.

Never rely on allure.attach() alone to sanitize data. It stores the bytes it receives. Redaction must happen before the call, and it needs tests.

CI logs and reports have different access and retention paths. A value masked by the CI log renderer may still appear unmasked inside an Allure attachment. Verify the generated artifact with a seeded fake secret before declaring the pipeline safe.

12. Troubleshoot Empty or Missing Allure Reports

Use the data flow to locate failure. First, did the test runner execute? Second, did the adapter write raw files? Third, could the CLI read them? Fourth, did CI upload the directory that was actually generated?

Add temporary directory listings without printing file contents that may contain secrets:

find allure-results -maxdepth 1 -type f -printf '%f\n' | sort | head -50
test -f allure-report/index.html

On macOS, the shown GNU find -printf option may not exist, so use the command in Linux CI or replace it with a portable listing. Diagnostics should report filenames and counts, not dump result JSON containing attachments or parameters.

If the HTML works locally but not when opened from downloaded artifacts, confirm the archive preserved the complete directory tree. Allure reports contain many linked assets and should be opened through the generated index.html, sometimes with a simple local HTTP server depending on browser restrictions.

13. Allure Report in CI Best Practices

• Pin the test adapter, Allure generator, Java runtime, and CI actions through normal dependency policy.
• Keep allure-results and allure-report as separate clean directories.
• Generate the report after every test outcome and preserve the runner's original status.
• Upload raw results for forensic regeneration and HTML for human review, with appropriate retention.
• Add feature, story, parameter, and environment context that helps triage.
• Keep step names business-readable and avoid wrapping every line.
• Restore only the prior history folder, never all old raw results.
• Merge parallel shard results in one validated downstream job.
• Redact before attachment and test the redaction with fake secrets.
• Publish reports only behind access controls appropriate to their contents.
• Monitor report generation time and artifact size as the suite grows.
• Document the owner for adapter upgrades, history storage, and failed report jobs.

For API pipelines, compare this approach with the artifact and exit-code patterns in Postman Newman in CI. Java teams should align the generator with the repository's build ownership described in Maven vs Gradle for testers.

Allure 3 provides a GitHub Action that reads generated Allure 3 HTML report summaries and can post pull request information and quality gate checks. It does not remove the need to configure a framework adapter and generate a compatible report first. Do not point it at Allure 2 output and assume feature parity.

Interview Questions and Answers

Q: Explain the Allure pipeline architecture.

A framework adapter writes structured result files during test execution. The Allure CLI transforms those raw files into a static HTML report. CI then uploads or publishes the report, while the test runner's exit code remains the quality gate.

Q: Why do reports often disappear on failed builds?

CI normally stops after the nonzero test command or later steps use a success-only condition. I make report generation and upload unconditional, then explicitly restore the test failure after artifacts exist. This preserves both diagnosis and correct status.

Q: What is the difference between allure-results and allure-report?

allure-results contains adapter-emitted raw data and attachments. allure-report is generated HTML and assets for people. The CLI consumes the former and produces the latter.

Q: How do you keep Allure history in ephemeral CI agents?

I download the previous trusted report history from durable storage, copy only its history folder into current results, and generate the new report. Then I publish the new history atomically for the next canonical run. Pull requests do not race to update it.

Q: How do you handle parallel tests?

Every shard writes a unique local results directory and uploads it. A merge job validates that all expected shards arrived, copies their files into one clean aggregate directory, restores history, and generates one report. Shard pass or fail status is aggregated separately.

Q: What should be attached to an Allure report?

Only evidence that materially reduces diagnosis time, such as a failed response excerpt, screenshot, or trace link. I redact secrets and sensitive fields before attachment and bound size. Successful routine payloads generally do not need duplication.

Q: How do you prevent stale tests in a report?

I delete current result and report directories before the run. I never append a new run to leftover raw results. Cross-run continuity uses only the documented history folder.

Q: How would you troubleshoot an empty report?

I verify test discovery and exit output, confirm the adapter is installed, count files in the configured results path, run the CLI with that exact path, and check for generated index.html. Then I inspect artifact path and conditions without dumping sensitive result contents.

Common Mistakes

• Treating Allure as the test runner instead of a result adapter and report generator.
• Uploading allure-results and expecting it to be browsable HTML.
• Generating reports only when tests pass.
• Swallowing the pytest exit code with || true and never restoring failure.
• Reusing an uncleared result directory and showing tests from prior runs.
• Installing the adapter but not the Allure CLI, or installing the CLI without Java.
• Mixing Allure 2 plugins and Allure 3 actions without checking compatibility.
• Copying all old result files to create history.
• Letting parallel shards write to one shared directory or generating before all shards arrive.
• Attaching tokens, headers, customer data, or complete successful payloads.
• Publishing reports publicly without reviewing their contents and access policy.
• Allowing report generation success to override a failed test gate.

Conclusion

A dependable Allure report in CI preserves two truths at the same time: reviewers receive a complete diagnostic report, and a failing test still fails the pipeline. Achieve that by separating raw result emission from HTML generation, running report steps after every outcome, uploading controlled artifacts, and restoring the original test status.

Implement the single-job workflow first and verify it with one intentional test failure. Confirm that the report downloads, shows the failure, contains no seeded fake secret, and leaves the job red. Then add protected publication, history, and shard merging only when the basic evidence path is trustworthy.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Flaky test quarantine in CI: Step by Step (2026)
• Parallel test sharding in CI: Step by Step (2026)
• Azure DevOps test pipelines: Step by Step (2026)
• CircleCI for test automation: Step by Step (2026)
• Docker for Playwright: Step by Step (2026)
• Docker for Selenium Grid: Step by Step (2026)