Allure vs ReportPortal Test Reporting (2026)

TL;DR

Allure is the better default for teams that need rich, portable reports generated from local result files. ReportPortal is the stronger platform when a staffed team needs persistent cross-run analytics, shared defect classification, dashboards, and centralized governance.

Key Takeaways

• Choose Allure for portable per-run reports, file-first capture, and repository-owned CI publishing.
• Choose ReportPortal when multiple teams need centralized history, dashboards, shared defect classification, and cross-run analysis.
• Compare both products with the same seeded failures, retries, traces, attachments, and outage scenarios.
• Keep Playwright's exit code and machine-readable output authoritative even when a reporting integration fails.
• Treat history as an identity and retention problem, not a feature that becomes useful without governance.
• Redact secrets before capture and test every local, generated, uploaded, and streamed evidence channel.

Choosing between allure vs reportportal test reporting depends on whether you need a portable report artifact or a continuously available test analytics service. Allure is the cleaner default for teams that want framework adapters, rich per-run evidence, and static reports published by CI. ReportPortal is stronger when many teams need centralized launch history, shared defect classification, dashboards, and automated failure analysis.

The important distinction is operational. Allure adapters write result files, then Allure Report generates a site. ReportPortal agents stream execution data to a deployed server that stores, indexes, and analyzes launches. One asks you to manage artifacts and history files. The other asks you to operate or buy a service, manage access, and keep agents connected.

This guide builds the same Playwright test for both products, injects the same failure, and gives you a decision method based on triage rather than screenshots. For focused setup guides, see test reporting with Allure and test reporting with ReportPortal.

TL;DR

Choose Allure if your main question is, "What failed in this build?" Choose ReportPortal if the harder question is, "Which failures repeat across teams, branches, and releases, and how should we classify them?" Neither tool should decide the CI result. Keep Playwright's exit code and a machine-readable reporter authoritative.

What You Will Build

You will create a small but diagnostic comparison harness:

• One Playwright 1.61.1 suite with named steps and a JSON attachment.
• An Allure Report 3.14.3 pipeline using allure-playwright 3.10.2.
• A ReportPortal pipeline using @reportportal/agent-js-playwright 5.4.2.
• A controlled assertion failure enabled by SEED_FAILURE=1.
• Two CI paths that publish evidence without masking a failed test command.
• A scorecard for setup, diagnostics, history, governance, security, and ownership.

The examples use separate Playwright configuration files. You can switch reporters without editing test behavior, which keeps the proof of concept fair.

Prerequisites

Use Node.js 24.x and npm 11.x. The Allure Playwright adapter supports modern Node releases, and the versions here are pinned so another install does not silently change your trial. Install Docker Engine with Docker Compose 2.2 or newer only if you are deploying a local ReportPortal evaluation instance.

You also need a ReportPortal 26.0.2 instance or a compatible managed endpoint, a project name, and a user API key. The official Docker deployment recommends at least 2 CPUs, 6 GB RAM, and 20 GB free disk for a simple installation. Treat that as an evaluation floor, not a production sizing guarantee.

Confirm the local tools before writing tests:

node --version
npm --version
docker compose version

Expect Node to print v24.x, npm to print 11.x, and Compose to report at least v2.2. Skip the Docker command when another team already operates ReportPortal.

Step 1: Model the allure vs reportportal test reporting architecture

Allure has two distinct components. The framework adapter records test events in allure-results; the allure CLI reads those files and generates allure-report. CI can retain both. Raw results make regeneration possible, while the generated site gives humans a navigable artifact. Allure Report 3 can keep history through its configured history path, but that persistence remains your pipeline's responsibility.

ReportPortal's Playwright agent is a networked reporter. It opens a launch, sends suites, tests, logs, steps, and attachments to the ReportPortal API, then finishes the launch. The platform stores successive runs and offers filters, dashboards, defect types, and analysis across them. A lost API connection is therefore part of the reporting failure model.

Keep reporting downstream from assertions in both designs:

Playwright test -> assertion and exit code
                -> line or JSON reporter -> CI truth
                -> Allure adapter -> result files -> generated site
                -> ReportPortal agent -> API -> persistent analytics

Do not let an HTML page, dashboard widget, or upload status turn a red suite green. The reporter can enrich test truth, but it cannot replace it. This separation is also the foundation of adding reporting to a test framework.

Validate the design before implementation by naming the failure boundary. Stop the ReportPortal server during one trial and corrupt one copied Allure attachment during another. The test exit code must remain correct, and the pipeline must surface reporting degradation separately.

Step 2: Create the pinned Playwright project

Create a disposable proof-of-concept directory and install exact package versions. Allure Report 3 is the allure package, not the older allure-commandline package used for Allure Report 2.

mkdir test-reporting-poc
cd test-reporting-poc
npm init -y
npm install --save-dev @playwright/test@1.61.1 typescript@5.9.2 \
  allure@3.14.3 allure-playwright@3.10.2 \
  @reportportal/agent-js-playwright@5.4.2
npx playwright install chromium

Add scripts so every later command uses the same filenames:

{
  "scripts": {
    "test:allure": "playwright test --config=playwright.allure.config.ts",
    "report:allure": "allure generate allure-results --output allure-report",
    "test:reportportal": "playwright test --config=playwright.reportportal.config.ts"
  }
}

Merge only the scripts object into the package.json produced by npm init; retain its generated metadata and devDependencies. Commit package-lock.json so CI resolves the same graph.

Check the installation rather than trusting npm's final message:

npx playwright --version
npx allure --version
npm ls allure-playwright @reportportal/agent-js-playwright

The output should identify Playwright 1.61.1, Allure 3.14.3, Allure Playwright 3.10.2, and the ReportPortal agent 5.4.2. A peer-dependency error is a reason to stop and resolve compatibility, not to add --force.

Step 3: Write one test with useful failure evidence

Create tests/reporting.spec.ts. The test uses only Playwright APIs understood by both reporters. It attaches structured context and changes one expectation when the failure flag is present.

import { test, expect } from "@playwright/test";

test("example domain has the expected identity", async ({ page }, testInfo) => {
  await test.step("open the public fixture", async () => {
    await page.goto("https://example.com/");
  });

  await test.step("record navigation context", async () => {
    await testInfo.attach("navigation-context", {
      body: Buffer.from(
        JSON.stringify(
          {
            url: page.url(),
            project: testInfo.project.name,
            retry: testInfo.retry
          },
          null,
          2
        )
      ),
      contentType: "application/json"
    });
  });

  await test.step("check the page identity", async () => {
    const expectedTitle =
      process.env.SEED_FAILURE === "1" ? "Seeded wrong title" : /Example Domain/;
    await expect(page).toHaveTitle(expectedTitle);
    await expect(page.getByRole("heading", { level: 1 }))
      .toHaveText("Example Domain");
  });
});

This fixture is intentionally small. The title mismatch yields an assertion diff, while Playwright's retained trace shows navigation and DOM state. The JSON attachment tells you which project and retry produced the result. A real API suite would attach a sanitized request summary, response status, correlation ID, and bounded response excerpt instead.

Run a reporter-neutral syntax and behavior check with Playwright's line reporter:

npx playwright test tests/reporting.spec.ts --reporter=line

Expect one passed test. If example.com is blocked in your environment, replace it with a stable service your team owns before comparing reporters. Both products must observe the same system and test data.

Step 4: Configure and verify Allure Report 3

Create playwright.allure.config.ts. The configuration retains a trace only when a failure occurs, applies stable labels, and writes raw data to a dedicated directory.

import { defineConfig } from "@playwright/test";
import * as os from "node:os";

export default defineConfig({
  testDir: "./tests",
  retries: process.env.CI ? 1 : 0,
  reporter: [
    ["line"],
    [
      "allure-playwright",
      {
        resultsDir: "allure-results",
        detail: true,
        suiteTitle: true,
        environmentInfo: {
          os_platform: os.platform(),
          node_version: process.version,
          target: process.env.TEST_ENV ?? "local"
        },
        globalLabels: {
          layer: "web",
          team: "quality-platform"
        }
      }
    ]
  ],
  use: {
    trace: "retain-on-failure",
    screenshot: "only-on-failure"
  }
});

Generate a clean passing run and its site:

npm run test:allure
npm run report:allure
npx allure open allure-report/awesome

Allure should show one test, three named steps, and navigation-context. Close the local server with Ctrl+C. The detail option also records Playwright API activity, so decide whether that extra depth helps or obscures your domain steps.

Verify raw and rendered outputs without depending on the browser:

find allure-results -name "*-result.json" -type f
test -f allure-report/awesome/index.html

The first command must list at least one result JSON file, and the second must exit with status 0. For production publication and retained history, continue with Allure reports in CI.

Step 5: Connect the same suite to ReportPortal

Deploy ReportPortal from its official Compose bundle only for an evaluation. Review the downloaded file, set a strong initial administrator password, pin the reviewed source revision in team documentation, and plan backup coverage before storing valuable history.

curl -LO https://raw.githubusercontent.com/reportportal/reportportal/master/docker-compose.yml
docker compose -p reportportal up -d --force-recreate
docker compose -p reportportal ps

Wait until the login page reports its API, Jobs, Authorization, and UI services as ready. Sign in, create or select a project, then generate a personal API key from the profile page. Never commit that key or place it directly in Playwright configuration.

Create playwright.reportportal.config.ts:

import type { PlaywrightTestConfig } from "@playwright/test";

function required(name: string): string {
  const value = process.env[name];
  if (!value) {
    throw new Error(`Missing required environment variable: ${name}`);
  }
  return value;
}

const rpConfig = {
  apiKey: required("RP_API_KEY"),
  endpoint: required("RP_ENDPOINT"),
  project: required("RP_PROJECT"),
  launch: process.env.RP_LAUNCH ?? "playwright-reporting-poc",
  description: "Controlled comparison run for test reporting",
  attributes: [
    { key: "suite", value: "e2e" },
    { key: "target", value: process.env.TEST_ENV ?? "local" }
  ],
  includeTestSteps: true,
  includePlaywrightProjectNameToCodeReference: true,
  uploadTrace: true,
  uploadVideo: false,
  launchUuidPrint: true,
  launchUuidPrintOutput: "STDOUT"
};

const config: PlaywrightTestConfig = {
  testDir: "./tests",
  retries: process.env.CI ? 1 : 0,
  reporter: [
    ["line"],
    ["json", { outputFile: "test-results/results.json" }],
    ["@reportportal/agent-js-playwright", rpConfig]
  ],
  use: {
    trace: "retain-on-failure",
    screenshot: "only-on-failure"
  }
};

export default config;

Export credentials in your shell, replacing each placeholder:

export RP_API_KEY="replace-with-generated-key"
export RP_ENDPOINT="http://localhost:8080/api/v2"
export RP_PROJECT="replace-with-project-name"
export RP_LAUNCH="playwright-reporting-poc"
npm run test:reportportal

The terminal should print a launch UUID because launchUuidPrint is enabled. Confirm test-results/results.json is nonempty, then open the matching launch in ReportPortal and locate the attachment under the test item:

test -s test-results/results.json
docker compose -p reportportal ps

These checks prove that Playwright completed and the local services remain running. The UI check proves ingestion, which a local JSON file alone cannot establish.

Step 6: Inject the same failure and compare triage

Run Allure with the controlled failure. Capture the expected nonzero status before generating the report.

set +e
SEED_FAILURE=1 npm run test:allure
allure_status=$?
set -e
test "$allure_status" -ne 0
npm run report:allure

Run ReportPortal against the identical code and flag:

set +e
SEED_FAILURE=1 npm run test:reportportal
reportportal_status=$?
set -e
test "$reportportal_status" -ne 0

In Allure, find the title assertion, inspect the trace, open the JSON context, and note whether the suite hierarchy leads you to the failure quickly. In ReportPortal, repeat those actions, assign a defect type, inspect prior executions for the same test case, and see whether nested steps make the failing action obvious.

Time two engineers independently. Use the median time to classify the seeded fault, not the faster outlier. Record missing evidence and wrong assumptions. Do not claim that either product reduces triage by a fixed percentage based on one artificial test.

Also test incomplete reporting. Point RP_ENDPOINT to an unreachable host and observe the agent's retry and failure behavior. Preserve the native JSON reporter so test truth remains available. For Allure, interrupt report generation after the test finishes and prove that allure-results can be archived and regenerated elsewhere.

Step 7: Publish both options safely in GitHub Actions

Use two jobs during the bounded evaluation. Keep them independent so a ReportPortal outage does not prevent the Allure artifact from being produced. Store RP_API_KEY, RP_ENDPOINT, and RP_PROJECT as repository or environment secrets.

name: reporting-comparison

on:
  workflow_dispatch:

jobs:
  allure:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run Playwright with Allure
        run: npm run test:allure
      - name: Generate Allure site
        if: ${{ always() && hashFiles('allure-results/*') != '' }}
        run: npm run report:allure
      - name: Upload raw results and report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: allure-report
          path: |
            allure-results
            allure-report
          if-no-files-found: error
          retention-days: 14

  reportportal:
    runs-on: ubuntu-latest
    env:
      RP_API_KEY: ${{ secrets.RP_API_KEY }}
      RP_ENDPOINT: ${{ secrets.RP_ENDPOINT }}
      RP_PROJECT: ${{ secrets.RP_PROJECT }}
      RP_LAUNCH: playwright-${{ github.run_number }}
      TEST_ENV: ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run Playwright and stream results
        run: npm run test:reportportal
      - name: Preserve native JSON
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: playwright-json
          path: test-results/results.json
          if-no-files-found: error
          retention-days: 14

A failing test step leaves its job red, while if: always() still publishes diagnostics. The ReportPortal job receives no secret on untrusted fork workflows unless you explicitly change GitHub permissions, which is the safer default.

Verify the workflow in a branch:

gh workflow run reporting-comparison
gh run list --workflow=reporting-comparison --limit=1
gh run watch

After completion, confirm that the failed or passed job conclusion matches Playwright, the Allure artifact exists, and the ReportPortal launch has the run-number suffix. The broader GitHub Actions for Playwright guide covers browser caching and matrix expansion.

Step 8: Evaluate history, security, and ownership

A one-run report hides the largest difference. Allure history needs durable storage and a stable test identity. ReportPortal keeps launches centrally, but meaningful history still depends on consistent code references, project boundaries, launch names, attributes, and retention rules. Random data in a test title fragments both systems.

Use a weighted scorecard after at least five comparable runs:

Scan generated or exported evidence for a fake canary secret before publication:

grep -R "QA_CANARY_SECRET_7F31" allure-results allure-report && exit 1 || true
grep -R "QA_CANARY_SECRET_7F31" test-results && exit 1 || true

For ReportPortal, search the launch through its UI or approved API as well, because streamed logs and attachments are not represented by local JSON. Redact authorization headers, cookies, tokens, personal data, and customer screenshots at capture time. Access control cannot undo unnecessary collection.

Which Should You Choose for allure vs reportportal test reporting

Choose Allure when each repository owns its pipeline, teams mostly investigate the current build, and static artifacts fit existing CI retention. It is also the safer choice for disconnected or restricted networks because adapters can write locally and publish later. Budget work for site hosting, result cleanup, stable labels, history persistence, and generator upgrades.

Choose ReportPortal when a quality platform team can operate the service and multiple suites need one queryable history. Its value increases when defect classification, shared dashboards, test-item history, launch comparison, and organization-wide analysis are active workflows rather than wishlist items. Budget the server estate, backups, upgrades, authentication, storage growth, agent compatibility, and incident response.

Use this decision rule:

• Small team, few repositories, no central analytics owner: start with Allure.
• Many teams, repeated triage, and a staffed platform function: pilot ReportPortal.
• Regulated environment: compare authorization, audit, retention, residency, and deletion requirements before features.
• Unreliable network between runners and reporting services: favor file-first capture or prove the agent's outage behavior.
• Mixed Java, Python, and JavaScript estate: test every official adapter against a common metadata contract.

Do not run both forever. A four-week trial is reasonable, but two human-facing sources create disagreement over counts, retries, and defect state. Select one primary investigation surface while retaining native runner output for CI and interchange.

Troubleshooting

Allure shows results from an earlier run -> The adapter appends to an existing results directory. Create a run-specific directory or clean the exact workspace path before execution. Never merge directories merely by reusing a long-lived folder.

The Allure report opens without the expected test -> Inspect allure-results for a result JSON before debugging the UI. Confirm that the Playwright configuration actually loaded allure-playwright, then regenerate from the correct directory.

ReportPortal returns 401 or 403 -> Regenerate a user API key, verify the project membership, and confirm that RP_ENDPOINT includes the API base expected by the agent. Avoid falling back to old token examples because apiKey is the current option.

ReportPortal creates duplicate or fragmented history -> Stabilize test titles and enable project-name contribution only when browser or device projects should have distinct identities. Align launch attributes so branch and target filters are predictable.

Attachments are missing -> Use Playwright's awaited testInfo.attach() call, retain traces on failure, and keep uploadTrace: true for the ReportPortal agent. Check size limits, storage health, and whether a test process was killed before final reporter callbacks.

CI is green despite a failed test -> Remove shell constructs that swallow the runner's exit code. Put generation and artifact upload in later always() steps rather than appending || true to the test command.

Interview Questions and Answers

Q: What is the architectural difference between Allure and ReportPortal?

Allure usually records framework events as local files and generates a report afterward. ReportPortal agents send those events to a persistent service during execution. The distinction changes outage handling, history ownership, infrastructure, access, and backup responsibilities.

Q: Which tool handles long-term test analytics better?

ReportPortal is designed around stored launches, test-item history, dashboards, and failure classification across runs. Allure can show history, but the pipeline must preserve and restore the appropriate history data. I would choose based on whether the team needs a report artifact or a shared analytics workflow.

Q: How would you compare them fairly?

I would run the same tests, retries, parallel workers, attachments, and seeded failures through separate reporter configurations. Then I would time classification, test an offline reporting dependency, measure storage, and score operational work. Demo screenshots would not be accepted as evidence.

Q: Why keep Playwright JSON or JUnit XML?

A machine-readable runner result remains useful when report generation or network delivery fails. CI status should follow the runner's exit code, while richer systems support human investigation. That boundary prevents reporting availability from changing test truth.

Q: What metadata needs governance?

Teams need stable test titles or IDs, controlled owner and component values, environment identifiers, and consistent branch or release attributes. Without those conventions, history fragments and filters become unreliable. Metadata should come from framework or pipeline context where possible.

Q: How do you stop reports from leaking secrets?

Capture an allowlisted subset of request and response data, mask sensitive fields before attachment, and restrict artifact or platform access. Add a canary secret to a reporting acceptance test and search every output channel for it. Retention and deletion controls complete the policy.

The concise model answers in the interviewQnA field provide additional practice without duplicating these explanations.

Common Mistakes

• Comparing an Allure HTML artifact with ReportPortal as if both were only page templates.
• Choosing centralized analytics without assigning a team to operate, upgrade, back up, and support it.
• Using random values in test names, which breaks history matching.
• Publishing traces and payloads without size limits or redaction.
• Cleaning a broad workspace path instead of an explicit per-run results directory.
• Forgetting that ReportPortal ingestion requires network availability from every runner.
• Treating a retry pass as a clean first-attempt pass.
• Allowing reporter calls to spread through page objects and business assertions.
• Scraping a human report to decide whether CI passed.
• Keeping both tools active after the evaluation without defining an authoritative surface.

Make the proof of concept adversarial. Include simultaneous failures, a killed worker, a server outage, a malformed attachment, and a renamed test. Normal green runs rarely reveal reporting architecture weaknesses.

Where To Go Next

Deepen the chosen route with test reporting with Allure or test reporting with ReportPortal. If Allure wins, implement durable publication using Allure Report in CI. If your Playwright evidence is weak, configure Playwright trace on retry before adding more dashboard features.

Use Docker Compose for test environments when the ReportPortal evaluation must run beside controlled dependencies. Keep the reporting proof of concept in source control as an acceptance suite for later agent, generator, or server upgrades.

Conclusion

Allure is the practical default for portable, repository-owned reports with strong single-run diagnostics. ReportPortal earns its larger operational footprint when centralized history, shared classification, dashboards, and cross-run analysis save enough triage effort to justify a service.

Run the same seeded failures through both, preserve native runner truth, and score the full lifecycle. The best 2026 choice is the one your team can secure, operate, and use consistently after the demo is over.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Test reporting with Allure (2026)
• Test reporting with ReportPortal (2026)
• Test scenarios vs test cases (2026)
• Allure vs Extent Reports: Which to Choose in 2026
• Database Rider vs DbUnit Test Data (2026)
• GitHub Actions vs Jenkins for Test Automation (2026)