How to Build a QA dashboard (2026)

TL;DR

To build a QA dashboard, invent a metric dictionary, ingest CI results with stable metadata, store them, and render a few decision-focused views for main-branch health, flakes, duration, and ownership.

Key Takeaways

• Define metric formulas and owners before drawing charts.
• Standardize CI outputs as JUnit/JSON with suite, SHA, and run id.
• Mark retries honestly so flake rate is visible.
• Filter by component, risk tag, and owner rather than one global pass rate.
• Ship a small operational board first, then an exec summary.
• Link every red tile to CI logs and artifacts.
• Alert on sustained critical-path failure, not every PR red.

If you want to know how to build a QA dashboard, start from decisions, not charts. A useful dashboard tells a release owner whether risk is rising, tells an engineer which suite is burning time, and tells a QA lead where flaky tests and coverage gaps concentrate. Pretty graphs without stable definitions become theater.

This 2026 guide shows how to design metrics, collect machine-readable results from CI, store them, and render a practical dashboard with open tools. Examples use GitHub Actions artifacts, JUnit/JSON reports, and a lightweight Node service you can host or adapt to Grafana. The principles apply whether you use Allure, ReportPortal, Buildkite Test Analytics, or a custom stack.

TL;DR

Build the data contract first. Only then pick visualization. If the contract is wrong, every chart will mislead.

1. Who needs how to build a QA dashboard guidance

How to build a QA dashboard well begins with audience:

• Pull request authors need "what failed on my change" and a link to logs, not a portfolio of KPIs.
• Squad leads need trends: flake rate by package, slowest suites, recurring failures.
• Release managers need gate status for critical journeys and known residual risk.
• Leadership needs a short narrative: are we safer than last month, and where is investment needed?

One screen rarely serves all four. Plan at least two surfaces: an operational board for engineers and a weekly summary for release risk. Forcing every metric onto one wall of tiles guarantees clutter.

Write a one-page product brief for the dashboard itself: primary user, primary decision, refresh rate, and non-goals. Non-goals matter. A QA dashboard is not a substitute for product analytics, APM, or a full BI warehouse, even if some data overlaps.

2. Define a metric dictionary before you write UI code

Every tile needs a precise definition, owner, and data source. Ambiguous "pass rate" is how arguments start in release meetings.

Store definitions in the repo (docs/qa-metrics.md) next to the collectors. When a metric changes formula, version it so historical charts do not silently rewrite the past.

Related reading on making reports trustworthy lives in add reporting to a test framework and Allure report in CI.

3. Standardize test result inputs from CI

Dashboards die when every suite emits a different shape. Standardize on one or two interchange formats:

• JUnit XML for broad tool compatibility.
• A small JSON lines (JSONL) event stream for custom fields (flake, owner, component, risk tag).

Example Playwright config emitting JUnit:

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["list"],
    ["junit", { outputFile: "test-results/junit.xml" }],
    ["json", { outputFile: "test-results/report.json" }],
  ],
  retries: process.env.CI ? 1 : 0,
});

Enrich CI with metadata environment variables:

- name: Run tests
  env:
    CI: true
    SUITE_NAME: web-e2e
    COMPONENT: checkout
    GIT_SHA: ${{ github.sha }}
    BRANCH: ${{ github.ref_name }}
    RUN_ID: ${{ github.run_id }}
  run: npx playwright test

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: junit-${{ github.run_id }}
    path: test-results/

Without suite name, component, and SHA, you cannot filter or bisect later. Capture them at generation time, not by reverse-engineering job names months later.

4. Choose storage: warehouse, TSDB, or simple database

For most mid-size teams, a relational database is enough:

Illustrative schema for PostgreSQL:

CREATE TABLE test_runs (
  id BIGSERIAL PRIMARY KEY,
  run_id TEXT NOT NULL,
  suite_name TEXT NOT NULL,
  component TEXT,
  git_sha TEXT NOT NULL,
  branch TEXT NOT NULL,
  started_at TIMESTAMPTZ NOT NULL,
  finished_at TIMESTAMPTZ,
  status TEXT NOT NULL,
  UNIQUE (run_id, suite_name)
);

CREATE TABLE test_results (
  id BIGSERIAL PRIMARY KEY,
  run_id TEXT NOT NULL,
  suite_name TEXT NOT NULL,
  test_name TEXT NOT NULL,
  status TEXT NOT NULL, -- passed, failed, skipped, flaky
  duration_ms INTEGER,
  retry_count INTEGER DEFAULT 0,
  owner TEXT,
  risk_tag TEXT,
  failure_message TEXT
);

CREATE INDEX idx_results_suite_time ON test_results (suite_name, id DESC);
CREATE INDEX idx_results_status ON test_results (status);

Keep raw JUnit artifacts in object storage for evidence. Store aggregates and per-test rows in the database for queries. Do not dump megabyte HTML reports into Postgres.

5. Ingest pipeline: from artifact to row

A typical flow:

1. CI finishes tests and uploads artifacts.
2. A workflow_run or workflow_call job downloads JUnit.
3. A small parser inserts rows and computes rollups.
4. Dashboard API reads rollups.

Illustrative Node parser (simplified):

// scripts/ingest-junit.ts
import fs from "node:fs";
import { XMLParser } from "fast-xml-parser";
import pg from "pg";

const parser = new XMLParser({ ignoreAttributes: false, attributeNamePrefix: "" });

type Case = {
  name: string;
  classname?: string;
  time?: string;
  failure?: unknown;
  skipped?: unknown;
};

async function main() {
  const xml = fs.readFileSync(process.argv[2], "utf8");
  const doc = parser.parse(xml);
  const suite = doc.testsuite ?? doc.testsuites?.testsuite;
  const suites = Array.isArray(suite) ? suite : [suite];
  const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
  await client.connect();

  const runId = process.env.RUN_ID!;
  const suiteName = process.env.SUITE_NAME!;
  const gitSha = process.env.GIT_SHA!;
  const branch = process.env.BRANCH!;

  await client.query(
    `INSERT INTO test_runs (run_id, suite_name, git_sha, branch, started_at, status)
     VALUES ($1,$2,$3,$4,NOW(),'ingested')
     ON CONFLICT (run_id, suite_name) DO NOTHING`,
    [runId, suiteName, gitSha, branch]
  );

  for (const s of suites) {
    const cases: Case[] = [].concat(s.testcase ?? []);
    for (const c of cases) {
      const status = c.failure ? "failed" : c.skipped ? "skipped" : "passed";
      const durationMs = Math.round(parseFloat(c.time ?? "0") * 1000);
      await client.query(
        `INSERT INTO test_results
         (run_id, suite_name, test_name, status, duration_ms)
         VALUES ($1,$2,$3,$4,$5)`,
        [runId, suiteName, `${c.classname ?? ""}.${c.name}`, status, durationMs]
      );
    }
  }

  await client.end();
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Wire ingest in GitHub Actions after tests:

- name: Ingest JUnit
  if: always()
  env:
    DATABASE_URL: ${{ secrets.QA_METRICS_DATABASE_URL }}
    RUN_ID: ${{ github.run_id }}
    SUITE_NAME: web-e2e
    GIT_SHA: ${{ github.sha }}
    BRANCH: ${{ github.ref_name }}
  run: npx tsx scripts/ingest-junit.ts test-results/junit.xml

Mark flaky tests at ingest time if retries > 0 and final status is passed while intermediate attempts failed. Honest flake marking is a dashboard superpower.

6. How to build a QA dashboard UI that people open daily

Resist twenty widgets. Start with five:

1. Main branch critical path status (green/yellow/red).
2. Pass rate trend (14 days) by suite.
3. Top flaky tests this week.
4. Slowest tests p95 with owners.
5. Open failures older than 24 hours on main.

Illustrative API handler:

// server/routes/summary.ts
import type { Request, Response } from "express";
import { pool } from "../db";

export async function getSummary(_req: Request, res: Response) {
  const { rows: passRows } = await pool.query(`
    SELECT suite_name,
           ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'passed') / NULLIF(COUNT(*),0), 1) AS pass_rate
    FROM test_results
    WHERE id IN (
      SELECT id FROM test_results ORDER BY id DESC LIMIT 5000
    )
    GROUP BY suite_name
    ORDER BY suite_name
  `);

  const { rows: flaky } = await pool.query(`
    SELECT test_name, COUNT(*) AS flaky_count
    FROM test_results
    WHERE status = 'flaky'
      AND id > (SELECT COALESCE(MAX(id),0) - 20000 FROM test_results)
    GROUP BY test_name
    ORDER BY flaky_count DESC
    LIMIT 10
  `);

  res.json({ passRates: passRows, topFlaky: flaky });
}

Front-end can be React, Grafana panels, or a static page hitting the API. What matters is latency under a few seconds and deep links into CI run URLs for each failure.

Use color with discipline: red means action required, yellow means watch, green means within SLO. Decorative rainbow charts train people to ignore the board.

7. Slice by component, risk tag, and ownership

Global pass rate hides a burning checkout suite behind a green catalog suite. Always support filters:

• component / service
• suite name
• risk_tag (smoke, critical-path, regression, nightly)
• owner (team or CODEOWNERS path)
• branch (main vs PR)

Populate risk_tag from annotations in tests when your framework supports it, or from directory conventions (tests/smoke/**). Directory conventions beat tribal knowledge.

test.describe("checkout smoke @critical-path", () => {
  test("pays with saved card", async ({ page }) => {
    // ...
  });
});

At ingest, parse @critical-path from the suite title into risk_tag. Release views should default to critical-path only.

For ownership mapping patterns and triage culture, see defect triage process.

8. Make flaky tests a first-class dashboard citizen

Flakes destroy trust in both CI and the dashboard. Surface:

• Flake rate by suite over time.
• Ranked list of flaky tests with last failure message sample.
• Quarantine status if you use a quarantine list.

-- flaky rate for a suite over recent results
SELECT
  suite_name,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE status = 'flaky')
    / NULLIF(COUNT(*) FILTER (WHERE status IN ('passed','failed','flaky')), 0)
  , 2) AS flake_pct
FROM test_results
WHERE suite_name = $1
GROUP BY suite_name;

Pair the chart with a policy: quarantine after N flakes in M days, owner assigned, expiry date. Dashboard without policy becomes a wall of shame nobody owns. For operational patterns, review flaky test quarantine in CI.

9. Connect defects and escapes for outcome metrics

Automation dashboards that ignore production misses optimize the wrong thing. Ingest defect metadata weekly if not automatically:

• count of S1/S2 production defects
• whether a regression test was added after the fix
• component tags matching test components

Even a simple spreadsheet import beats pure vanity pass rates. Over a quarter, plot escape defects versus automated critical-path coverage. Use the plot to justify investment, not to blame individuals.

// illustrative weekly rollup insert
await pool.query(
  `INSERT INTO quality_outcomes (week_start, component, escape_s1, escape_s2, notes)
   VALUES ($1, $2, $3, $4, $5)`,
  ["2026-07-07", "checkout", 0, 1, "Payment timeout under load"]
);

10. Access control, cost, and retention

QA dashboards often show internal URLs, failure messages with data samples, and branch names. Apply:

• SSO for the UI
• read-only DB roles for the API
• redaction of secrets in failure messages at ingest
• retention: hot detailed rows 30-90 days, daily aggregates longer

Cost control:

• aggregate nightly; do not store full stdout in Postgres
• sample PR runs if volume is extreme; always store main fully
• compress artifacts in object storage with lifecycle rules

Document RPO/RTO only if the dashboard is on the release critical path. Many teams accept dashboard downtime while still blocking merges on CI status checks.

11. Grafana alternative path

If you already run Grafana, export Prometheus metrics from the ingest job instead of building a custom UI:

// pseudo-metrics endpoint for Prometheus scrape
// GET /metrics
// qa_suite_pass_rate{suite="web-e2e"} 97.5
// qa_suite_duration_ms{suite="web-e2e",quantile="0.95"} 812000
// qa_flaky_tests{suite="web-e2e"} 4

Grafana panels then become standard time series. Keep a separate table or log-based view for "top flaky tests" because ranked lists are awkward as pure gauges.

Choose custom UI when you need deep product-specific workflows (quarantine buttons, owner assignment). Choose Grafana when your org standardizes observability there and QA metrics should live beside service SLOs.

12. Rollout plan: from pilot to default home page

1. Pilot one suite on main with JUnit ingest and three charts.
2. Validate definitions with the squad in a 30-minute review: do numbers match CI UI?
3. Add flake marking and owners.
4. Expand suites with the same metadata contract.
5. Add critical-path release view.
6. Automate alerts: main critical-path red for > 30 minutes posts to the team channel with run link.
7. Retire duplicate spreadsheets only after two weeks of matching numbers.

Alert carefully. A noisy Slack stream is worse than no dashboard. Alert on sustained critical-path red and on flake rate SLO breaches, not on every PR failure.

13. Example SLO pack for a mature board

These numbers are illustrative starting points, not universal industry benchmarks. Calibrate to product risk.

14. Anti-patterns unique to QA dashboards

• Celebrating 100% pass rate achieved by deleting tests or over-quarantining.
• Mixing local developer runs into production quality metrics.
• Counting skipped tests as passes.
• Resetting history after a tool migration without labeling the break.
• Building a dashboard only the QA manager understands because labels are internal slang.

Name metrics in product language ("checkout smoke green") as well as suite language ("web-e2e-3").

Interview Questions and Answers

Q: How would you start if leadership asks for a QA dashboard next week?

I define three metrics with written formulas, instrument one suite to emit JUnit plus metadata, store results, and ship a single-page view for main branch health. I refuse twenty vanity charts before the data contract is trusted.

Q: How do you calculate flake rate honestly?

I retain attempt-level outcomes. A test that fails then passes on retry is flaky, not a clean pass. Flake rate is flaky outcomes divided by eligible outcomes in a fixed window, excluding skips.

Q: What is the difference between a test report and a QA dashboard?

A report explains one run. A dashboard aggregates many runs over time with ownership and risk filters to support decisions. Both need stable test identity.

Q: How do you stop the dashboard from becoming a blame tool?

Focus on systems metrics, pair red tiles with run links and owners, review trends in blameless forums, and invest in quarantine and fix capacity when flake SLOs breach.

Q: Should PR results and main results share the same charts?

Usually separate them. PR noise is high and branches differ. Main tip health and critical-path history should be the release default.

Q: How do you represent quarantined tests?

Show them as explicit quarantined counts with owners and expiry, not as passes. Track whether quarantine size is growing.

Q: What metadata is non-negotiable on every ingested run?

Suite name, git SHA, branch, run id, timestamps, and per-test status with duration. Owner and risk tag should follow immediately after.

Common Mistakes

• Jumping to chart libraries before defining metric formulas.
• Ingesting only successful runs, which paints a false green world.
• Ignoring retries so flakes look like healthy green.
• No owners on failing tests, so red tiles linger indefinitely.
• One global pass rate for a multi-service monorepo.
• Storing huge HTML blobs in the primary database.
• Alerting on every failure until the channel is muted.
• Never linking tiles back to CI logs and artifacts.

Conclusion

How to build a QA dashboard in 2026 is a product design problem wrapped around CI telemetry. Write metric definitions, emit consistent JUnit/JSON with metadata, ingest into durable storage, and ship a small set of decision-oriented views for engineers and release owners. Add flake honesty, ownership, and critical-path filters before executive vanity tiles.

Next step: pick one suite on main, emit JUnit with suite name and SHA, insert into a table, and put pass rate, top failures, and p95 duration on a single internal page. Expand only after that page is opened in standup without arguments about what the numbers mean.

15. Concrete wireframe of the two primary views

View A: Engineering operations (default home for QA and dev leads)

1. Header: environment label (main), last ingest timestamp, link to CI.
2. Row of SLO chips: critical path, flake %, p95 duration.
3. Trend chart: pass rate by suite, last 14 days.
4. Table: top 10 flaky tests with owner, count, last message snippet, deep link.
5. Table: slowest 10 tests with p95 and trend arrow.
6. Table: main failures open > 24 hours.

View B: Release readiness (used on cut day)

1. Candidate SHA and branch.
2. Critical-path checklist with last run time.
3. Known quarantines affecting critical path.
4. Open S1/S2 defects for the release train.
5. Notes field for residual risk accepted by EM/PM.

Implement View A first. View B can be a filtered mode of the same API with risk_tag=critical-path and a defects side panel. If product managers will not open a separate URL, embed View B as a Confluence macro or Slack Block Kit summary generated from the same API.

// scripts/post-release-summary.ts
const res = await fetch(`${process.env.QA_API}/summary?branch=main&risk=critical-path`);
const data = await res.json();
const text = [
  `*Release readiness* SHA ${process.env.GIT_SHA}`,
  `Critical path pass rate: ${data.criticalPassRate}%`,
  `Open critical failures: ${data.openCritical}`,
  `Quarantined critical tests: ${data.quarantinedCritical}`,
].join("\n");
// post text to Slack webhook

16. Testing the dashboard itself

Treat the dashboard as production software:

• Unit test metric formulas with fixture JUnit files.
• Contract test the ingest API against sample XML.
• Synthetic check: a scheduled job asserts last ingest time is fresh.
• Accessibility and auth tests if the UI is custom.

import { describe, it, expect } from "vitest";
import { computePassRate } from "../src/metrics";

describe("computePassRate", () => {
  it("excludes skipped by default", () => {
    const rate = computePassRate([
      { status: "passed" },
      { status: "passed" },
      { status: "failed" },
      { status: "skipped" },
    ]);
    expect(rate).toBeCloseTo(66.7, 1);
  });
});

A dashboard that lies because of an off-by-one in skip handling is worse than no dashboard. Lock formulas with tests and show the formula in a tooltip on each tile.

Migration from spreadsheets and tribal status meetings

Many teams already have a Friday quality spreadsheet. Migration path:

1. Parallel-run: dashboard metrics posted beside the spreadsheet for two weeks.
2. Diff review: explain every mismatch (usually retries, skips, or timezone).
3. Cutover: spreadsheet becomes export from API or is deleted.
4. Meeting change: standup projects the board; people do not retype numbers.

Expect social friction. Numbers that were hand-massaged will look worse when honest. Leadership support for honest flake visibility is required, or teams will recreate private spreadsheets.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Add CI to a test framework (2026)
• How to Add logging to a test framework (2026)
• How to Add parallel execution to a framework (2026)
• How to Add reporting to a test framework (2026)
• How to Build a BDD framework with Cucumber and Playwright (2026)
• How to Build a Cypress framework from scratch (2026)