How to Test RAG Metadata Filtering Accuracy (2026)

TL;DR

To test RAG metadata filtering accuracy, query a controlled corpus with deterministic vectors, compare returned IDs with independently calculated eligible IDs, and fail on omissions or leakage. Keep filter tests separate from ranking tests, then add compound, boundary, missing-field, and tenant-isolation cases.

Key Takeaways

• Test filter correctness separately from semantic ranking so a weak embedding cannot hide a filter defect.
• Build expected document IDs from an independent oracle instead of copying production filter construction.
• Measure eligible recall, ineligible leakage, precision, and exact-set agreement for every case.
• Cover arrays, missing fields, dates, tenant boundaries, and compound AND/OR logic with explicit fixtures.
• Use deterministic vectors and an in-memory Qdrant collection for fast, reproducible CI tests.
• Gate releases on zero cross-tenant leakage and case-specific recall thresholds.

To test RAG metadata filtering accuracy, prove two things independently: every returned chunk satisfies the requested constraints, and every eligible chunk that should be retrievable appears within a large enough candidate window. A high-quality answer is not evidence that filtering worked. The generator can sound convincing after the retriever silently leaks another tenant's document or drops an eligible policy.

This tutorial builds a deterministic Python and Qdrant test harness. You will seed a small corpus, express filters through Qdrant's public models, calculate expected IDs with an independent oracle, and enforce accuracy gates with pytest. For the broader evaluation architecture, read the RAG application evaluation guide.

The central design choice is separation. First test metadata eligibility with identical vectors and a generous limit. Then test semantic ranking with meaningful embeddings in a different suite. That split makes failures diagnosable.

What You Will Build

You will create a compact regression suite that:

• stores six chunks with tenant, product, language, status, tags, and timestamp metadata;
• converts a typed search request into a real qdrant_client.models.Filter;
• derives expected IDs from plain Python predicates that do not reuse production filter code;
• reports eligible recall, precision, leakage count, and exact-set agreement;
• tests equality, array membership, date ranges, compound logic, missing fields, and tenant isolation;
• runs locally without a Qdrant server by using Qdrant Client's in-memory mode.

This is an accuracy harness, not a latency benchmark. Once correctness is stable, add timing and concurrency in a separate job using the ideas in load testing an LLM API.

Prerequisites

Use this reproducible baseline:

• Python 3.12.11
• qdrant-client 1.15.1
• pytest 8.4.1

Create an isolated environment and install exact versions:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install qdrant-client==1.15.1 pytest==8.4.1
python --version
python -m pytest --version
python -c "import qdrant_client; print(qdrant_client.__version__)"

Expected verification output contains Python 3.12.11, pytest 8.4.1, and qdrant-client 1.15.1. A later compatible patch can work, but pin the evaluated versions in CI so dependency drift does not change results without review.

Create this layout as you follow the steps:

rag-filter-tests/
  rag_filters.py
  test_rag_filters.py

Step 1: Define a Corpus That Exposes Filter Defects

A useful fixture contains near-duplicates across security boundaries. If every tenant has unrelated text, semantic ranking may hide a missing tenant condition. Identical vectors remove that escape route.

Create rag_filters.py with imports, a typed request, and six records:

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any

from qdrant_client import QdrantClient, models

COLLECTION = "support_chunks"
VECTOR = [1.0, 0.0, 0.0, 0.0]

DOCUMENTS: list[dict[str, Any]] = [
    {"id": 1, "tenant": "acme", "product": "api", "language": "en", "status": "published", "tags": ["auth", "security"], "created_at": "2026-01-10T00:00:00Z"},
    {"id": 2, "tenant": "acme", "product": "web", "language": "en", "status": "published", "tags": ["billing"], "created_at": "2026-02-15T00:00:00Z"},
    {"id": 3, "tenant": "acme", "product": "api", "language": "fr", "status": "draft", "tags": ["auth"], "created_at": "2026-03-01T00:00:00Z"},
    {"id": 4, "tenant": "globex", "product": "api", "language": "en", "status": "published", "tags": ["auth", "security"], "created_at": "2026-01-20T00:00:00Z"},
    {"id": 5, "tenant": "acme", "product": "api", "language": "en", "status": "archived", "tags": [], "created_at": "2025-12-31T23:59:59Z"},
    {"id": 6, "tenant": "acme", "product": "api", "language": "en", "status": "published", "tags": ["auth"]},
]

@dataclass(frozen=True)
class SearchRequest:
    tenant: str
    product: str | None = None
    languages: tuple[str, ...] = ()
    required_tag: str | None = None
    published_since: datetime | None = None

Records 1 and 4 intentionally share all fields except tenant. Record 6 intentionally lacks created_at. Record 5 sits one second before the 2026 boundary.

Verify Step 1: run python -m py_compile rag_filters.py. No output and exit code 0 confirms valid syntax.

Step 2: Seed an In-Memory Vector Collection

Append a factory that creates a fresh collection for every test run:

def make_client() -> QdrantClient:
    client = QdrantClient(location=":memory:")
    client.create_collection(
        collection_name=COLLECTION,
        vectors_config=models.VectorParams(
            size=len(VECTOR),
            distance=models.Distance.COSINE,
        ),
    )
    client.upsert(
        collection_name=COLLECTION,
        points=[
            models.PointStruct(id=doc["id"], vector=VECTOR, payload=doc)
            for doc in DOCUMENTS
        ],
        wait=True,
    )
    return client

Every point receives the same nonzero vector. Cosine similarity is therefore equal, so membership depends on metadata rather than embedding quality. A fresh in-memory database also prevents stale local data from making a test pass accidentally.

For production-scale evaluation, keep a versioned snapshot of realistic payload distributions. The tiny fixture remains valuable because each record has a reason to exist and failures identify one rule. The adversarial RAG evaluation dataset guide explains how to expand controlled cases without turning them into random noise.

Verify Step 2: run this command:

python -c "from rag_filters import make_client, COLLECTION; c=make_client(); print(c.count(COLLECTION, exact=True).count)"

Expected output is 6.

Step 3: Build the Production Metadata Filter

Append the filter builder. Qdrant places AND conditions in must, OR alternatives in should, and exclusions in must_not. A nested Filter inside must preserves the language OR group while requiring tenant and publication status.

def build_filter(request: SearchRequest) -> models.Filter:
    must: list[models.Condition] = [
        models.FieldCondition(
            key="tenant", match=models.MatchValue(value=request.tenant)
        ),
        models.FieldCondition(
            key="status", match=models.MatchValue(value="published")
        ),
    ]

    if request.product is not None:
        must.append(models.FieldCondition(
            key="product", match=models.MatchValue(value=request.product)
        ))

    if request.languages:
        must.append(models.FieldCondition(
            key=\"language\",
            match=models.MatchAny(any=list(request.languages)),
        ))

    if request.required_tag is not None:
        must.append(models.FieldCondition(
            key="tags", match=models.MatchValue(value=request.required_tag)
        ))

    if request.published_since is not None:
        must.append(models.FieldCondition(
            key="created_at",
            range=models.DatetimeRange(gte=request.published_since),
        ))

    return models.Filter(must=must)

MatchValue against an array payload matches when at least one stored array element equals the value. The date condition excludes missing created_at fields because they cannot satisfy the range. Notice that status is a server-side requirement, not a post-query Python check. Post-filtering can return fewer than k results and makes vector database telemetry misleading.

Verify Step 3: run:

python -c "from rag_filters import build_filter, SearchRequest; print(len(build_filter(SearchRequest('acme')).must))"

Expected output is 2, representing tenant and published status.

Step 4: Query Through One Retrieval Boundary

Append the retrieval function:

def retrieve_ids(
    client: QdrantClient, request: SearchRequest, limit: int = 100
) -> list[int]:
    result = client.query_points(
        collection_name=COLLECTION,
        query=VECTOR,
        query_filter=build_filter(request),
        limit=limit,
        with_payload=False,
    )
    return [int(point.id) for point in result.points]

query_points is the public universal query endpoint in qdrant-client. The test limit is 100, larger than the fixture, because filter membership is the target. A production top-k such as 5 would confound an eligible document omitted by ranking with one rejected by filtering.

This boundary is also where integration mistakes appear: sending the filter under the wrong parameter, querying the wrong collection, or applying a default tenant. Keep application code behind a similarly narrow function so tests exercise serialized requests, not only Python model construction.

Verify Step 4: run:

python -c "from rag_filters import *; print(sorted(retrieve_ids(make_client(), SearchRequest('acme'))))"

Expected output is [1, 2, 6]. Draft, archived, and Globex points must be absent.

Step 5: Test RAG Metadata Filtering Accuracy With an Independent Oracle

The oracle must not call build_filter. Reusing the system under test would reproduce the same bug in expected and actual results. Append plain Python eligibility logic:

def parse_utc(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))

def expected_ids(request: SearchRequest) -> set[int]:
    eligible: set[int] = set()
    for doc in DOCUMENTS:
        if doc["tenant"] != request.tenant:
            continue
        if doc["status"] != "published":
            continue
        if request.product is not None and doc["product"] != request.product:
            continue
        if request.languages and doc["language"] not in request.languages:
            continue
        if request.required_tag is not None and request.required_tag not in doc["tags"]:
            continue
        if request.published_since is not None:
            value = doc.get("created_at")
            if value is None or parse_utc(value) < request.published_since:
                continue
        eligible.add(doc["id"])
    return eligible

Create test_rag_filters.py:

from datetime import datetime, timezone

import pytest

from rag_filters import SearchRequest, expected_ids, make_client, retrieve_ids

CASES = [
    SearchRequest(tenant="acme"),
    SearchRequest(tenant="acme", product="api"),
    SearchRequest(tenant="acme", languages=("en", "fr")),
    SearchRequest(tenant="acme", required_tag="security"),
    SearchRequest(
        tenant="acme",
        product="api",
        languages=("en",),
        required_tag="auth",
        published_since=datetime(2026, 1, 1, tzinfo=timezone.utc),
    ),
]

@pytest.mark.parametrize("request", CASES)
def test_filter_returns_exact_eligible_set(request: SearchRequest) -> None:
    actual = set(retrieve_ids(make_client(), request))
    expected = expected_ids(request)
    assert actual == expected, {
        "missing": sorted(expected - actual),
        "leaked": sorted(actual - expected),
    }

Verify Step 5: run python -m pytest -q. Expected output is 5 passed. If one case fails, pytest prints missing and leaked IDs rather than an unhelpful score alone.

Step 6: Calculate Recall, Precision, Leakage, and Exact Agreement

Exact equality is ideal for controlled fixtures, but larger golden datasets need diagnostic metrics. Add this to test_rag_filters.py before the tests:

def membership_metrics(expected: set[int], actual: set[int]) -> dict[str, float | int]:
    true_positive = len(expected & actual)
    false_positive = len(actual - expected)
    false_negative = len(expected - actual)
    recall = true_positive / len(expected) if expected else 1.0
    precision = true_positive / len(actual) if actual else (1.0 if not expected else 0.0)
    return {
        "recall": recall,
        "precision": precision,
        "leakage_count": false_positive,
        "omission_count": false_negative,
        "exact_match": int(expected == actual),
    }

Then append a security-focused gate:

def test_compound_filter_accuracy_gate() -> None:
    request = SearchRequest(
        tenant="acme", product="api", languages=("en",), required_tag="auth"
    )
    expected = expected_ids(request)
    actual = set(retrieve_ids(make_client(), request))
    metrics = membership_metrics(expected, actual)

    assert metrics["leakage_count"] == 0
    assert metrics["recall"] == 1.0
    assert metrics["precision"] == 1.0

The metrics answer different questions:

These are not semantic retrieval metrics. For ranking, calculate recall at k against relevance judgments as shown in testing RAG retrieval recall at k and evaluating RAG retrieval precision.

Verify Step 6: rerun python -m pytest -q. Expected output is 6 passed.

Step 7: Test Boundaries, Missing Fields, and Empty Results

Append targeted tests. Each names the contract it protects:

def test_date_range_is_inclusive_at_lower_boundary() -> None:
    request = SearchRequest(
        tenant="acme",
        published_since=datetime(2026, 1, 10, tzinfo=timezone.utc),
    )
    assert set(retrieve_ids(make_client(), request)) == {1, 2}

def test_missing_date_does_not_satisfy_date_range() -> None:
    request = SearchRequest(
        tenant="acme",
        published_since=datetime(2026, 1, 1, tzinfo=timezone.utc),
    )
    assert 6 not in retrieve_ids(make_client(), request)

def test_array_filter_matches_one_required_tag() -> None:
    request = SearchRequest(tenant="acme", required_tag="security")
    assert retrieve_ids(make_client(), request) == [1]

def test_unknown_tenant_returns_empty_set() -> None:
    request = SearchRequest(tenant="initech")
    assert retrieve_ids(make_client(), request) == []

def test_tenant_filter_never_leaks_near_duplicate() -> None:
    request = SearchRequest(tenant="acme", product="api", required_tag="security")
    actual = set(retrieve_ids(make_client(), request))
    assert actual == {1}
    assert 4 not in actual

The first test confirms gte includes an exact timestamp. The second documents missing-field semantics. The last test deserves its own assertion even though the parameterized suite covers it, because a tenant leak is a release blocker and should be obvious in reports.

Also test invalid requests before they reach the database. An empty tenant should raise a validation error in your API layer rather than becoming a broad or ambiguous query. Filter accuracy begins with a precise request contract.

Verify Step 7: run python -m pytest -q. Expected output is 11 passed.

Step 8: Add Mutation Checks and CI Regression Gates

A passing suite is meaningful only if it fails when logic is broken. Temporarily remove the tenant condition from build_filter; the exact-set and tenant tests must report point 4 as leaked. Change gte to gt; the boundary test must lose point 1. Remove the nested language group; language cases must fail. Revert each mutation after observing the red test.

Run the stable suite in CI with one command:

name: rag-filter-accuracy
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12.11'
          cache: pip
      - run: python -m pip install qdrant-client==1.15.1 pytest==8.4.1
      - run: python -m pytest -q

For a larger golden corpus, write metric JSON as an artifact and compare it with an approved baseline. Never average away tenant leakage. An overall 99.9 percent precision can still conceal one forbidden document. Use a hard zero-leakage gate per tenant and per access-control class, then use aggregate recall for less critical discoverability filters.

When the filter suite passes, connect it to end-to-end checks from testing a RAG chatbot end to end. That second layer verifies that retrieved context is actually sent to the model and citations map back to allowed sources.

Verify Step 8: commit the two example files on a branch and run the workflow. The job must show all 11 tests passing. Perform at least one deliberate mutation in a temporary branch and confirm the job turns red.

How to Test RAG Metadata Filtering Accuracy in Production-Like Data

Small fixtures establish logic. A production-like evaluation catches schema drift and value distributions that unit cases miss. Export payloads without sensitive text, replace identifiers, and preserve the shapes that affect filtering: missing keys, array lengths, casing, timestamp formats, tenant sizes, and status frequencies. Attach a dataset version to every result.

Create stratified cases rather than random queries. Include rare languages, the largest tenant, tenants with one document, empty tag arrays, legacy records without fields, and timestamps on both sides of retention boundaries. For each request, store expected IDs or a reproducible independent predicate. Have a domain owner review authorization-related expectations.

Run membership evaluation with a limit large enough to include all eligible candidates. If the database caps results, paginate or scroll under the same filter. Otherwise, low recall may only mean the test requested too few points. Run a separate top-k ranking suite with realistic embeddings and relevance labels.

Track results by filter family. A single global score can hide that equality filters are perfect while date ranges fail on legacy strings. Report at least tenant, equality, array, range, compound, missing-field, and negative-result slices. If you also evaluate generated answers, keep those scores separate using a method such as measuring RAG faithfulness and relevancy.

Test RAG Metadata Filtering Accuracy Without Misleading Results

Avoid these common mistakes:

• Using the same builder for expected results: a missing clause appears in both paths and the test passes. Use plain predicates, reviewed ID sets, or a separately implemented reference query.
• Judging only the final answer: the model may ignore leaked context once and use it later. Assert retrieved IDs before generation.
• Testing only positive matches: empty results, forbidden tenants, missing fields, and values just outside a range reveal dangerous defaults.
• Using top-k as the filter candidate window: ranking truncation looks like filter omission. Use a generous limit or pagination for membership tests.
• Averaging leakage: authorization failures require a count of zero in every protected slice.
• Generating random fixtures without intent: random data rarely creates exact boundaries or near-duplicate cross-tenant records. Design each row to kill a plausible mutation.
• Ignoring serialization: model-only unit tests can miss an API client parameter or payload mismatch. Retain at least one real client integration test.
• Comparing ordered lists: equal-score vectors may return in an unspecified order. Compare sets for membership, and test order only in the semantic ranking suite.

Troubleshooting

Problem: the same eligible IDs return in a different order -> compare sets in metadata membership tests. Equal vectors intentionally make ranking order irrelevant.

Problem: date filters return no records -> inspect the stored payload type and timezone. Store RFC 3339 timestamps, create timezone-aware Python datetimes, and verify legacy values were not indexed as arbitrary strings.

Problem: an array tag never matches -> confirm the payload is an array of scalar strings and use FieldCondition with MatchValue for one required member. Do not serialize the whole array as one JSON string.

Problem: recall falls when the corpus grows -> raise the query limit or paginate for membership evaluation. If the omitted IDs satisfy the filter but rank below the candidate cutoff, move that failure to the retrieval ranking suite.

Problem: local tests pass but the remote collection leaks data -> compare collection schema, payload keys, client versions, and the serialized filter. Add an integration test against an isolated remote collection populated from the same fixture.

Problem: an empty expected set reports confusing precision -> define the convention before dashboarding. This tutorial assigns precision 1.0 only when expected and actual are both empty, while any unexpected return produces 0.0 and positive leakage.

Interview Questions and Answers

The model answers in the interviewQnA field cover test-oracle independence, filter versus ranking metrics, tenant isolation, date boundaries, missing metadata, and CI gates. In an interview, explain the failure signal and the test architecture, not only the metric formula.

Watch for Silent Filter Drift

The most dangerous metadata-filter failures are the ones that never throw. A renamed field, a type coerced from integer to string, or a tenant id that arrives as null still returns a syntactically valid result set, just the wrong one. Guard against silent drift with three cheap checks. First, assert the count and identity of returned chunks against a fixed fixture, not just that some results came back. Second, add a negative control to every filter test: a document that must be excluded, so a broken filter fails loudly instead of quietly widening recall. Third, log the compiled filter expression your retriever actually sent to the vector store and diff it against the intended one in CI. A filter that matches everything is indistinguishable from no filter at all until a customer sees another tenant's data, so treat exclusion as a first-class assertion rather than an afterthought.

Where To Go Next

Start by adapting the six-record fixture to your real payload schema. Run it from the Resume Studio upload surface when evaluating document ingestion behavior, or practice explaining the design in the QA interview practice area.

Next, broaden the evaluation in this order:

1. Add realistic relevance judgments with testing embeddings and vector search quality.
2. Measure ranked retrieval using RAG context precision with RAGAS.
3. Verify source attribution through RAG citation correctness examples.
4. Test the complete response path with RAG hallucination pipeline testing.

Conclusion

A reliable way to test RAG metadata filtering accuracy is to isolate eligibility from similarity, seed adversarial metadata, query through the real client, and compare returned IDs with an independent oracle. Exact-set assertions expose both omissions and leakage, while sliced metrics make larger regressions understandable.

Keep tenant leakage at zero, document boundary semantics, and prove your tests by mutating each critical clause. Once metadata membership is trustworthy, evaluate ranking, citations, faithfulness, latency, and the final answer as separate layers.

Treat payload schema changes as migrations with compatibility tests. When language becomes locale, seed old-only, new-only, and dual-written records, then state which versions each query must find. Run the old and new filter contracts during the rollout window. This catches a common failure where freshly indexed chunks pass while older knowledge silently disappears. Record the collection schema version beside the dataset version so a historical score can be reproduced after indexes change. Review casing and normalization too: en-US, en-us, and en are distinct scalar values unless ingestion deliberately canonicalizes them.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• How to Add CI to a test framework (2026)
• How to Add logging to a test framework (2026)
• How to Add reporting to a test framework (2026)
• How to Use Gatling to Load Test a GraphQL API (2026)
• How to Use k6 to Load Test HTTP3 Endpoints (2026)
• How to Use k6 to Test gRPC Streaming APIs (2026)