Pairwise and orthogonal array testing (2026)

TL;DR

Use pairwise testing as the flexible default for constrained or mixed-level software configurations. Use an orthogonal array when a suitable construction fits and balanced pair representation matters. In both cases, verify valid pair coverage and supplement the array with risk-focused tests.

Key Takeaways

• Pairwise testing covers every valid two-factor level pair at least once.
• Strength-2 orthogonal arrays add equal pair frequency when the level structure fits.
• Meaningful factors, representative levels, and reviewed constraints determine the design's value.
• Generated rows are configurations that still need actions, oracles, setup, cleanup, and evidence.
• Verify coverage independently against pairs that occur in at least one valid full row.
• Add boundaries, states, negative cases, and targeted higher-strength interactions based on risk.
• Store the model, constraints, tool version, seed, and generated rows for reproducibility.

Pairwise and orthogonal array testing are combinatorial test design techniques for selecting a small, defensible set of configurations from a much larger combination space. Pairwise testing covers every valid pair of factor values at least once. Orthogonal arrays add mathematical balance and uniformity when the factors and level structure fit a suitable array.

Neither technique proves that a system is correct, and neither should replace boundary, state, negative, or risk-based testing. Their value is specific: they make interaction coverage visible and repeatable when exhaustive combination testing is too expensive.

TL;DR

Start with a factor model, remove impossible combinations as explicit constraints, generate the rows, and verify coverage. Then add risk-driven cases for interactions stronger than two, boundaries within each value, workflows, and failure recovery.

1. What Pairwise and Orthogonal Array Testing Mean

A factor is a dimension that can influence behavior, such as browser, operating system, account type, locale, or payment method. A level is one selected value for a factor. A test row assigns one level to every factor. The full Cartesian product contains every possible row.

Pairwise testing creates a covering array of strength 2. For every two factors, every valid pair of their levels occurs in at least one test row. If Browser has Chrome and Firefox, while Locale has English and Arabic, the suite must include Chrome-English, Chrome-Arabic, Firefox-English, and Firefox-Arabic somewhere. The same condition applies to every other pair of factors.

An orthogonal array also has a formal strength. In a strength-2 orthogonal array, every ordered pair of levels for any two columns appears the same number of times. This equality provides pairwise coverage plus balance. Traditional notation such as L9(3^4) describes nine runs, four three-level factors, and a standard strength-2 construction.

The terms are often used loosely, which creates confusion. Every suitable strength-2 orthogonal array provides pair coverage, but a pairwise covering array does not have to be orthogonal or balanced. A constrained mixed-level product is usually easier to model with a pairwise generator than with a textbook orthogonal array.

The defect hypothesis matters. These methods assume that many interaction failures are triggered by a small number of factors. Strength 2 targets two-way interactions. If a tax error requires region, customer class, and exemption status together, ordinary pairwise coverage does not guarantee that triple.

2. Why Combination Testing Needs a Model

Suppose a checkout supports 4 browsers, 3 account types, 5 payment methods, 4 locales, and 3 shipping speeds. Exhaustive coverage needs 4 x 3 x 5 x 4 x 3, or 720 rows, before considering input boundaries, order states, and failures. Repeating 720 similar happy paths may be less useful than a smaller interaction suite plus focused tests for money, authorization, and recovery.

The first mistake is jumping from 720 to a generator without examining the model. A factor should represent a behaviorally meaningful choice. "Chrome 124," "Chrome 125," and "Chrome 126" may not be useful levels if the product supports evergreen Chrome and the behavior under test does not depend on those releases. Conversely, grouping all mobile devices into "mobile" can hide rendering engines, input modes, or permission differences.

Levels are representatives, not complete partitions by default. A level called "large file" is useless until the team defines its size, format, and relation to limits. If file size behavior changes at 10 MB, include values around that boundary in dedicated tests. Combinatorial design selects interactions among representatives. Boundary value analysis selects revealing representatives within a dimension.

Constraints also encode product truth. A wallet available only in one region should not produce invalid test rows elsewhere. Document that rule instead of silently deleting inconvenient rows. Constraints can make some nominal pairs impossible. Coverage must therefore be measured against valid pairs, not against the unconstrained Cartesian product.

This modeling discipline connects naturally to equivalence partitioning and boundary value analysis, which helps choose meaningful levels before combination generation begins.

3. Pairwise vs Orthogonal Arrays: A Practical Comparison

Use pairwise generation when the application has unequal numbers of levels, business constraints, changing configuration, or a need for the smallest practical suite. Use an orthogonal array when a known array fits the factor-level structure and balance is important, especially for controlled experiments where effects are compared across evenly represented conditions.

Balance is not automatically better for software fault detection. If risk is concentrated in one payment method, equal representation may waste scarce execution time. You can keep pair coverage and add extra high-risk rows. The final suite will not be orthogonal, but it may be more valuable for a release decision.

Small suite size is also not the only optimization target. A mathematically minimal array can be difficult to set up because each row changes every environment dimension. A slightly larger suite may reduce fixture churn, reuse accounts, or make failures easier to isolate. Treat execution cost and diagnostic quality as part of test design.

For a broader view of selecting techniques by information value, see black box test design techniques.

4. Build a Factor and Constraint Model

Begin with one test objective. "Test checkout" is too broad. "Evaluate whether a signed-in shopper can authorize a standard purchase across supported client, region, and payment configurations" gives the model a stable purpose.

List candidate factors from requirements, architecture, production data, incident history, feature flags, and team knowledge. For each factor, record its levels, why behavior might vary, setup owner, and observability. Remove dimensions that cannot influence the chosen objective. Add dimensions that cross service or policy boundaries.

Consider this model:

Now identify constraints. Guest plus Bank transfer may be unsupported. Wallet may be unavailable for Business accounts. Express delivery may be unavailable for one region. Write each rule in business language and validate it with product or domain owners. Constraints should represent impossible or unsupported states, not cases the team dislikes testing.

Separate environment constraints from model constraints. If the test lab temporarily lacks WebKit, the combination is still valid and remains a coverage gap. Do not declare it impossible. If a provider sandbox cannot simulate a decline, record a testability limitation and pursue a stub, contract test, or lower-level check.

Assign stable identifiers to factors and values so the generated artifact can be reviewed in source control. Preserve display labels separately. A rename such as "Corporate" to "Business" should not silently appear as an entirely new level if its semantics have not changed.

5. Generate and Verify a Pairwise Covering Set

Production teams often use established combinatorial generators, but understanding the mechanics prevents blind trust. The runnable Python program below uses only the standard library. It enumerates all candidate rows, builds the set of pairs covered by each row, then greedily selects the candidate that covers the most currently uncovered pairs.

from itertools import combinations, product

factors = {
    "browser": ["chromium", "firefox", "webkit"],
    "account": ["guest", "member", "business"],
    "region": ["us", "de", "jp"],
    "payment": ["card", "bank", "wallet"],
}

names = list(factors)

def valid(row):
    item = dict(zip(names, row))
    if item["account"] == "guest" and item["payment"] == "bank":
        return False
    if item["account"] == "business" and item["payment"] == "wallet":
        return False
    return True

def pairs_for(row):
    indexed = list(enumerate(row))
    return {
        ((left_index, left_value), (right_index, right_value))
        for (left_index, left_value), (right_index, right_value)
        in combinations(indexed, 2)
    }

candidates = [
    row for row in product(*(factors[name] for name in names))
    if valid(row)
]
required_pairs = set().union(*(pairs_for(row) for row in candidates))
uncovered = set(required_pairs)
suite = []

while uncovered:
    best = max(candidates, key=lambda row: len(pairs_for(row) & uncovered))
    newly_covered = pairs_for(best) & uncovered
    if not newly_covered:
        raise RuntimeError("No valid candidate can cover the remaining pairs")
    suite.append(best)
    uncovered -= newly_covered
    candidates.remove(best)

for number, row in enumerate(suite, start=1):
    print(number, dict(zip(names, row)))

covered = set().union(*(pairs_for(row) for row in suite))
assert required_pairs <= covered
print(f"rows={len(suite)}, valid_pairs={len(required_pairs)}")

This greedy implementation is educational, not a claim of global minimum size. Ties depend on candidate order, and larger models need more efficient search. The important verification is the final assertion against valid pairs. For team use, save the model, constraints, generator identity, generator version, random seed if applicable, and produced rows.

Review the output for setup feasibility and high-risk gaps. Pairwise coverage may include Card-US somewhere, but not necessarily Card-US-Business. Add targeted rows for critical three-way interactions and name why each override exists.

6. Construct and Read an Orthogonal Array

An L9(3^4) array can assign four factors with three levels each to nine rows. One common representation is:

Map columns to factors and digits to actual levels. If A is browser, 0 might be Chromium, 1 Firefox, and 2 WebKit. For any two columns, all nine ordered pairs from 00 through 22 appear exactly once. Here lambda equals 1.

Do not map levels casually. If D is transaction amount and the digits mean $1, $100, and $10,000, those are experimental representatives, not boundary coverage. Add dedicated checks around product thresholds. If one factor has only two meaningful levels, duplicating a value to force three levels changes frequency and interpretation. A mixed-level array or pairwise generator is cleaner.

Constraints are particularly important. If a row maps to an impossible combination, deleting it destroys the array's balance and may remove multiple pairs. Substituting a value can duplicate other pairs. Once an array is altered, recompute pair frequencies and call it an adapted covering design rather than assuming orthogonality remains.

Orthogonal arrays originated in experimental design, where balanced combinations help separate factor effects under specific assumptions. Software test execution often produces pass or fail results with complex state, not independent measurements. Use the array for coverage unless you have a valid experimental protocol and statistical expertise for stronger claims.

7. Turn Generated Rows Into Executable Tests

A generated row is a configuration, not a complete test. Add preconditions, action, oracle, cleanup, and observability. "Firefox, Business, Germany, Card" says nothing about the order contents, expected tax, authorization result, persisted state, or side effects.

Create one stable base scenario. For checkout, the action might create a known basket, enter an address, select delivery, authorize payment, and verify the final order. Define independent oracles: UI confirmation, API state, provider record, inventory reservation, and accounting event where access permits. Avoid making every oracle depend on the same calculation used by the system.

Parameterize automated checks only when each row follows the same meaningful workflow. If Bank transfer creates a pending order while Card completes immediately, one assertion template with many conditionals may hide distinct state models. Split behaviorally different workflows and apply combinatorial design inside each.

Manual execution also benefits from data-driven rows. Present readable labels and expected policy alongside factor codes. Capture actual configuration, build, time, test identity, and correlation identifiers. If a row fails, rerun a nearby passing row that changes one relevant factor when possible. The comparison can isolate the interaction.

Generated tests need maintenance. When a level is retired, a feature flag changes, or a constraint moves, regenerate and review. Do not manually patch a spreadsheet indefinitely because coverage claims will drift from the model. Keep targeted additions in a separate, traceable layer so regeneration does not erase them.

If these rows feed a wider regression pack, regression testing strategy explains how to combine interaction coverage with change impact, critical journeys, and historical failures.

8. Pairwise and Orthogonal Array Testing for Real Systems

Browser compatibility is a natural use case when engine, operating system, viewport, locale, input mode, and account type interact. Model supported values, then add device-specific risks such as permissions, virtual keyboards, and resource limits outside the pairwise array.

Configuration testing can combine database engine, deployment region, feature flag, tenant plan, authentication provider, and cache mode. Constraints prevent unsupported deployments. Because configuration defects can affect every user in a tenant, add higher-strength coverage around recent migrations or incident-prone components.

API testing can combine role, resource state, operation, content type, and conditional header. Be careful: operation and resource state often form a state model, while role and operation form an authorization matrix. A decision table or state transition suite may express these rules more clearly. Combinatorial rows can supplement, not replace, those explicit models.

Forms can combine field values, but do not make every individual input a factor. Ten fields with three levels each create a model that is hard to interpret and may miss within-field boundaries. First test field-level validation. Then model a smaller number of meaningful categories such as customer type, address class, tax state, and delivery selection.

Distributed workflows add timing, retries, duplication, and service availability. These dimensions may require controlled fault injection rather than ordinary parameterization. Pairwise can select fault-location and workflow-state combinations, while targeted tests cover known dangerous triples such as retry plus timeout plus duplicate delivery.

The consistent rule is to preserve semantics. A compact array is useful only when its factors represent real behavior and each row has a reliable oracle.

9. Higher Strength, Constraints, and Coverage Gaps

Three-way or t-wise testing generalizes the same idea. A strength-3 covering array includes every valid triple of factor levels. Row count and setup cost rise, sometimes sharply. Increase strength selectively when architecture, incidents, or domain rules suggest higher-order interactions.

A practical hybrid uses pairwise coverage across the full model and strength 3 for a subset such as region, account type, and payment method. Some generators support variable-strength models. If yours does not, create a second targeted model and merge its distinct rows. Recalculate coverage after deduplication.

Constraints can create subtle gaps. A pair may be valid in isolation but impossible to complete into a full row because of relationships with other factors. Define required pairs as those present in at least one valid full candidate. Review constraints for accidental overrestriction. A rule copied from the current UI may hide an API authorization defect if the backend should still reject direct requests.

Seed known interactions to evaluate a generator or harness. For example, make a local test fixture fail only when Browser is WebKit and Locale is Arabic. Confirm that the suite contains the pair and that reporting shows the actual row. Then add a triple-triggered fixture to demonstrate what strength 2 does not guarantee.

Coverage reports should distinguish generated pair coverage, executed row coverage, passed row coverage, blocked rows, and risk additions. "100% pairwise coverage" is misleading if several generated rows never ran or their oracles checked only page load.

10. A Repeatable Team Workflow

First, state the test objective and system boundary. Second, identify behaviorally meaningful factors and levels. Third, define and review business constraints. Fourth, choose strength based on interaction risk. Fifth, generate and independently verify coverage. Sixth, enrich each row with setup, actions, oracles, and evidence.

Run a pilot against a stable feature. Track setup time, execution time, unique findings, false failures, and diagnostic effort. Compare the compact design with the previous selection, but do not fabricate a defect-detection percentage from a small sample. Ask whether the model revealed gaps and whether teammates can review it.

Store the model and rows in source control. Give each factor and constraint an owner or source. Regenerate through a reproducible command. In the pull request, show added and removed levels, changed constraints, row-count effect, and targeted risk cases.

During release planning, do not report only the number of rows. Report which interactions are covered, which valid configurations remain outside the chosen strength, which rows are blocked, and which risks received higher-strength or dedicated tests.

Retire the design when the objective no longer exists. Archive the model with release evidence if required, but remove stale rows from active regression. A compact obsolete suite is still waste.

Interview Questions and Answers

Q: What is pairwise testing?

Pairwise testing is a combinatorial technique that selects rows so every valid pair of values across any two factors appears at least once. It targets two-way interaction faults while avoiding the full Cartesian product. It does not guarantee boundary coverage, workflow coverage, or every three-way interaction.

Q: How is an orthogonal array different from a pairwise covering array?

A strength-2 orthogonal array includes each two-column level pair an equal number of times. A pairwise covering array only requires each valid pair to appear at least once. Orthogonal arrays provide balance, while covering arrays offer more flexibility for mixed levels and constraints.

Q: Can pairwise testing replace exhaustive testing?

It is an alternative selection strategy when exhaustive combination coverage is not worth its cost, not a proof equivalent to exhaustive testing. I use it with boundary, state, negative, and risk-based tests. Critical higher-order interactions receive dedicated or higher-strength coverage.

Q: How do you handle invalid combinations?

I model them as reviewed constraints and generate against valid full rows. Then I verify coverage against pairs that can occur in at least one valid row. I keep lab limitations separate because an unavailable environment is a coverage gap, not a product constraint.

Q: What is the biggest modeling risk?

Poor factors and levels can produce mathematically complete but behaviorally weak tests. I derive factors from architecture, rules, incidents, and production configurations, then confirm that each level can change behavior. Boundaries and state transitions remain separate explicit models.

Q: When would you use strength 3?

I use it when evidence suggests important three-factor interactions, for example region plus customer class plus tax status. I may apply variable strength to only that high-risk subset. The additional rows must justify their setup and execution cost.

Q: How do you verify a generated suite?

I enumerate all valid pairs from the constrained full model, enumerate pairs present in generated rows, and compare the sets. I also check row validity, executable setup, and actual run status. Generator output alone is not evidence that every row executed or had a meaningful oracle.

Common Mistakes

• Treating pairwise as a synonym for two tests per feature.
• Choosing arbitrary factor values without equivalence or boundary reasoning.
• Calling a covering array orthogonal even though pair frequencies are unequal.
• Deleting invalid orthogonal-array rows without recomputing coverage and balance.
• Encoding a temporary environment outage as an impossible product combination.
• Claiming pairwise testing covers all interactions.
• Counting generated rows as executed and passed coverage.
• Building one giant model across unrelated workflows.
• Ignoring setup cost, order dependence, and cleanup.
• Parameterizing behaviorally different state flows into conditional-heavy tests.
• Failing to save the model, constraints, generator version, and generated artifact.
• Optimizing only for the fewest rows.

Conclusion

Pairwise and orthogonal array testing make combination selection explicit. Pairwise covering arrays are the practical default for mixed-level, constrained software configurations. Orthogonal arrays are valuable when their level structure fits and balanced representation serves the objective.

Model factors carefully, verify valid pair coverage independently, and turn every row into an observable test with a real oracle. Then add boundaries, states, failures, and higher-strength cases where risk demands them. The result is not merely a smaller suite, it is a suite whose interaction coverage the team can explain.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Testing bulk and batch endpoints (2026)
• Testing embeddings and vector search quality (2026)
• Testing pagination and filtering (2026)
• Testing sorting and search APIs (2026)
• Testing token limits and truncation (2026)
• Accessibility testing checklist (2026)