Testing embeddings and vector search quality (2026)

TL;DR

Create a golden relevance dataset, preserve the complete retrieval configuration, and evaluate rankings at the k your product uses. Combine semantic relevance metrics with exact-versus-ANN checks, filter and lifecycle tests, latency measurements, and paired regression analysis.

Key Takeaways

• Evaluate the embedding contract, semantic ranking, vector index, filters, and downstream product behavior separately.
• Use reviewed queries, graded relevance labels, no-answer cases, and hard negatives as the benchmark core.
• Select Recall@k, MRR, precision, and nDCG based on how users consume results.
• Compare approximate neighbors with an exact-search baseline while keeping vectors and distance functions fixed.
• Treat tenant leakage and stale permissions as hard failures, never as small ranking losses.
• Version the complete retrieval configuration and inspect paired per-query deltas before release.

Testing embeddings and vector search quality requires more than confirming that an API returns a list of floating-point numbers. A retrieval system succeeds only when relevant items rank above distractors for realistic queries, across important user segments, at acceptable latency and cost.

This guide gives QA and SDET engineers a measurable 2026 test strategy. You will define relevance labels, calculate Recall@k, precision, mean reciprocal rank, and normalized discounted cumulative gain, compare approximate search with an exact baseline, test chunking and filters, and build regression gates that catch quality loss before it reaches a RAG application.

TL;DR

Build a small, reviewed relevance dataset first. Evaluate the whole retrieval configuration, including text normalization, embedding model, dimensions, chunking, index, similarity function, filters, and reranking. A strong embedding in a weak pipeline still produces weak search.

1. What Testing Embeddings and Vector Search Quality Means

Testing embeddings and vector search quality has three related but distinct layers. The embedding layer maps text to vectors. The index layer retrieves neighbors efficiently. The product layer uses those neighbors to answer a question, recommend an item, or find a duplicate. A single end-to-end score cannot tell you which layer failed.

At the embedding layer, verify contract properties: one vector per input, correct input-to-output order, expected dimension for the configured model, finite numeric values, and safe handling of empty or oversized content. Semantic tests then ask whether relevant pairs are closer than carefully selected negatives.

At the retrieval layer, measure whether labeled relevant documents appear in the top k and in what order. Include metadata filters, tenant isolation, deletions, updates, and approximate nearest-neighbor behavior. Search quality is a property of the complete configuration, not the model name alone.

At the product layer, evaluate downstream usefulness. A RAG answer can fail even when retrieval recall is high because the generator ignores evidence. It can also appear correct despite a retrieval defect because the model memorized the fact. Keep retrieval metrics separate from grounded-answer metrics such as those in measuring RAG faithfulness and relevancy.

Write a version manifest for every experiment. Record the embedding provider and model, requested dimensions if supported, normalization, distance function, chunker revision, index type and parameters, filters, reranker, corpus snapshot, and benchmark version. Without that provenance, a score is not reproducible.

2. Create a Golden Relevance Dataset

A golden dataset contains queries, candidate documents, and reviewed relevance judgments. It should reflect the language, vocabulary, document types, and ambiguity of the actual product. Random sentences are useful for smoke tests, but they are not a search-quality benchmark.

For each query, label at least one relevant document when the product should return an answer. Include graded labels when some documents are ideal and others are merely useful. A simple scheme is 0 for irrelevant, 1 for relevant, and 2 for highly relevant. Also include no-answer queries, where every document is irrelevant. Those cases reveal whether the system confidently returns the nearest bad option.

Build cases from several sources:

• Common production intents after privacy review and de-identification.
• Requirements and domain expert examples.
• Search logs where users reformulated or abandoned a query.
• Known incidents and support tickets.
• Synthetic paraphrases that a reviewer accepts.
• Hard negatives that share keywords but answer a different question.

A password-reset query and a password-policy document may overlap lexically but serve different intents. Such hard negatives are more valuable than obviously unrelated text. Include abbreviations, spelling errors, product names, code identifiers, multilingual input when supported, short queries, long natural-language questions, and temporal wording.

Split the data into a development set and a held-out release set. Teams often overfit chunk sizes and reranker settings to one visible benchmark. Keep label guidelines and reviewer disagreement records. If experts cannot agree on relevance, the system should not be judged against a pretend exact truth. For broader test design, building a QA chatbot with RAG explains how corpus and retrieval choices affect user-visible behavior.

3. Choose Metrics That Match the Search Experience

No single metric describes all ranking behavior. Select metrics based on how users consume results and how many relevant items can exist.

Recall@k is often the first retrieval metric for RAG because the generator cannot use a document it never receives. If each query has one canonical target, MRR exposes whether that target is first or buried. nDCG is useful when multiple chunks have different relevance grades.

Always report per-query results in addition to averages. Averages hide catastrophic slices and make debugging difficult. Break results down by intent, language, document age, query length, tenant, and difficulty. Use a hard zero-leakage assertion for tenant boundaries rather than treating cross-tenant retrieval as a small precision loss.

Choose k from the application contract. If the retriever passes five chunks to a generator, Recall@20 can look excellent while Recall@5 is poor. Report several diagnostic cutoffs, but gate the cutoff the product actually uses. Avoid copying threshold values from another company. Baseline your current system, inspect the distribution, and select risk-based targets with domain owners.

4. Build a Runnable Exact-Search Test Harness

An exact cosine search over a small corpus is an excellent reference implementation. It removes approximate-index behavior so you can test the embedding and ranking logic directly. The official OpenAI embeddings endpoint accepts a batch of strings and returns embeddings in input order.

# retrieval_eval.py
import os
from dataclasses import dataclass
from math import sqrt
from openai import OpenAI

MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")
client = OpenAI()

@dataclass(frozen=True)
class Document:
    doc_id: str
    text: str
    tenant: str

DOCUMENTS = [
    Document("reset", "Reset a forgotten password from the sign-in page.", "acme"),
    Document("policy", "Passwords require at least fourteen characters.", "acme"),
    Document("invoice", "Download invoices from Billing, then Documents.", "acme"),
    Document("other", "Private roadmap for another customer.", "globex"),
]

def embed(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model=MODEL,
        input=texts,
        encoding_format="float",
    )
    ordered = sorted(response.data, key=lambda item: item.index)
    vectors = [item.embedding for item in ordered]
    if len(vectors) != len(texts):
        raise AssertionError("Embedding count does not match input count")
    if any(not vector for vector in vectors):
        raise AssertionError("Embedding must not be empty")
    return vectors

def cosine(a: list[float], b: list[float]) -> float:
    if len(a) != len(b):
        raise ValueError("Vector dimensions differ")
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sqrt(sum(x * x for x in a))
    norm_b = sqrt(sum(y * y for y in b))
    if norm_a == 0 or norm_b == 0:
        raise ValueError("Zero vector cannot be compared")
    return dot / (norm_a * norm_b)

def search(query: str, tenant: str, k: int = 2) -> list[str]:
    scoped = [doc for doc in DOCUMENTS if doc.tenant == tenant]
    vectors = embed([query] + [doc.text for doc in scoped])
    query_vector, document_vectors = vectors[0], vectors[1:]
    ranked = sorted(
        zip(scoped, document_vectors),
        key=lambda pair: cosine(query_vector, pair[1]),
        reverse=True,
    )
    return [doc.doc_id for doc, _ in ranked[:k]]

if __name__ == "__main__":
    result = search("I cannot remember my login password", "acme", k=2)
    print(result)
    assert "other" not in result
    assert result[0] == "reset"

Run it with python retrieval_eval.py after installing the current openai package and setting OPENAI_API_KEY. The last assertion is a deliberately small integration check, not proof of production quality.

Keep query and document preprocessing identical to production. A benchmark that embeds clean source text while production adds navigation, markup, or access-control headers measures a different system. Batch requests reduce overhead, but preserve the returned index when rebuilding input order. Never compare vectors generated by different models or dimensions as though they occupied the same space.

5. Calculate Recall, MRR, and nDCG Correctly

Metric code should be deterministic, independently tested, and explicit about unlabeled documents. The following functions accept ranked document IDs and relevance judgments. They require no model call and can be covered with ordinary unit tests.

# metrics.py
from math import log2

def recall_at_k(ranked: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        raise ValueError("Recall is undefined when no documents are relevant")
    found = len(set(ranked[:k]) & relevant)
    return found / len(relevant)

def reciprocal_rank(ranked: list[str], relevant: set[str]) -> float:
    for rank, doc_id in enumerate(ranked, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(ranked: list[str], grades: dict[str, int], k: int) -> float:
    def dcg(values: list[int]) -> float:
        return sum(
            (2 ** grade - 1) / log2(index + 2)
            for index, grade in enumerate(values)
        )

    actual = [grades.get(doc_id, 0) for doc_id in ranked[:k]]
    ideal = sorted(grades.values(), reverse=True)[:k]
    ideal_score = dcg(ideal)
    return 0.0 if ideal_score == 0 else dcg(actual) / ideal_score

def test_metrics() -> None:
    ranked = ["d2", "d1", "d4", "d3"]
    relevant = {"d1", "d3"}
    assert recall_at_k(ranked, relevant, 2) == 0.5
    assert reciprocal_rank(ranked, relevant) == 0.5

    grades = {"d1": 2, "d3": 1}
    score = ndcg_at_k(ranked, grades, 4)
    assert 0.0 < score < 1.0

if __name__ == "__main__":
    test_metrics()
    print("metric tests passed")

Average query metrics only over the population for which each metric is defined. No-answer queries need separate measures, such as the rate of returning no results above a calibrated similarity threshold, or the rate of correctly abstaining after reranking. Do not silently assign perfect recall to a query with no relevant labels.

Test the metric implementation with hand-calculated tiny rankings. Cover k larger than the result list, duplicate IDs, missing grades, no relevant result, and an ideal ranking. A metric bug can make every model experiment look scientific while reversing the actual conclusion.

Store raw rankings with experiment results. When a score moves, engineers need to see which documents entered or left the top k. A delta without a ranking is difficult to diagnose.

6. Add Negative, Boundary, and Metamorphic Tests

Golden queries show known behavior, while property-oriented tests reveal broader defects. Embeddings are probabilistic representations, so choose relationships that should usually hold for the product rather than demanding universal linguistic laws.

Useful metamorphic cases include:

• A harmless punctuation change should not radically replace all top results.
• A reviewed paraphrase should retrieve the same canonical document.
• Adding a tenant filter must never introduce an out-of-tenant document.
• Reordering a batch must preserve the text-to-vector association after using response indexes.
• Re-embedding unchanged text with the same pinned configuration should preserve retrieval behavior within the team's accepted tolerance.
• Appending an irrelevant boilerplate footer should not consistently overpower the core topic.

Boundary cases include an empty string, whitespace, one token, very long input near the application's limit, unsupported language, invalid Unicode handling, duplicated documents, identical vectors, zero-result filters, and k greater than corpus size. Validate the application's own limit before sending input to a provider. Provider limits and model availability can change, so the product contract should be configurable.

Test hard negatives deliberately. For a query about deleting an account, documents about deleting a project and deactivating an account are close but operationally different. Keyword overlap can fool both lexical and semantic systems. These cases often provide more diagnostic value than random negatives.

Avoid asserting an exact floating-point vector. Provider implementations and model revisions can change, and an exact vector is not the user requirement. Assert the vector contract and ranked behavior. If a regulated application requires a pinned artifact, verify the provider's versioning guarantees and store a controlled baseline according to policy.

7. Test Chunking, Metadata, Hybrid Search, and Reranking

A vector search test that ignores chunking tests only half the retrieval system. Chunk size, overlap, headings, tables, code blocks, and document boundaries determine what the model embeds. Evaluate these settings as part of the candidate configuration.

Create fixtures where the answer falls at a chunk boundary, appears under a heading far from the relevant sentence, or spans two adjacent paragraphs. Check that chunk IDs preserve source document, section, offsets, version, and access metadata. Duplicate overlap can fill the top k with near-identical chunks from one document, reducing evidence diversity. Add a result-diversity measure or document-level deduplication check if that harms the product.

Metadata filters require both correctness and security coverage. Test missing fields, null values, case normalization, multiple allowed groups, revoked access, document updates, and tenant isolation. Apply authorization filtering in a trusted retrieval layer, not in a prompt that asks the model to ignore forbidden chunks.

If the product uses hybrid retrieval, compare lexical-only, vector-only, and combined candidates on the same benchmark. Hybrid search can rescue exact identifiers, error codes, and rare names that dense embeddings underweight. Reranking may improve top positions while adding latency, so measure both relevance gain and operational cost. A reranker must never receive documents the user was not authorized to view.

Treat each pipeline variation as a complete candidate. Changing chunking requires re-indexing. Changing the embedding model requires re-embedding the corpus. Mixing old and new vectors in one index invalidates a fair comparison unless the migration design explicitly supports it.

8. Validate Approximate Nearest-Neighbor Indexes

Production vector databases usually use approximate nearest-neighbor, or ANN, indexes because exact comparison against every vector is expensive. ANN quality should be measured against an exact search over the same vectors, distance function, filters, and query set.

For each query, collect the exact top k IDs and the ANN top k IDs. Calculate the fraction of exact neighbors recovered. This is ANN recall@k, which evaluates index approximation, not business relevance. A system can have high ANN recall and poor semantic search because the embeddings are weak. It can also have strong embeddings but lose relevant neighbors through aggressive index settings.

Test index lifecycle behavior:

1. Insert a known document and verify it becomes searchable after the documented consistency window.
2. Update its text and metadata, then verify stale content and permissions disappear.
3. Delete it and confirm it cannot be retrieved.
4. Rebuild or compact the index and compare rankings.
5. Restart services and verify persistence and index readiness.
6. Run concurrent reads and writes if the product allows them.

Measure quality and latency at realistic corpus size and filter selectivity. An index setting that performs well on ten thousand vectors may behave differently at production scale. Use synthetic filler for scale only after preserving a reviewed semantic core. Report p50 and tail latency, timeouts, and error rates alongside ANN recall.

When tuning index parameters, change one dimension at a time and preserve the exact-vector snapshot. Otherwise you may attribute an embedding change to the index or an index change to chunking.

9. Analyze Regressions and Statistical Uncertainty

A candidate comparison should be paired by query. For each benchmark query, calculate the new metric minus the baseline metric. This shows which cases improved, regressed, or stayed unchanged and prevents corpus composition from shifting between runs.

Do not rely only on a tiny change in the overall mean. Inspect effect size, the number of changed queries, critical-slice movement, and confidence intervals when the dataset supports them. Bootstrap intervals over queries can describe uncertainty without assuming normal metric values, but the benchmark still needs representative sampling. Statistical technique cannot repair biased labels.

Define three categories of release criteria:

• Hard invariants: no cross-tenant results, no malformed IDs, no missing required metadata.
• Quality floors: accepted minimum for the business-critical slice at the product's k.
• Non-regression rules: the candidate must not materially degrade against the approved baseline.

The numeric values should be owned by the product and engineering team. Mark early thresholds as provisional and revise them through a documented process, not after seeing a favored candidate fail.

Investigate every large per-query delta. Common causes include a changed text normalizer, truncated content, wrong distance direction, stale index data, inconsistent filters, duplicated chunks, and label errors. Keep an error taxonomy so repeated failure patterns guide the next dataset additions and engineering work.

10. Automate Testing Embeddings and Vector Search Quality in CI

Testing embeddings and vector search quality in CI should separate fast deterministic checks from provider-backed benchmarks. Run metric unit tests, text normalization tests, filter tests, and exact-search logic on every commit. Run a focused embedding integration set on relevant pull requests, then schedule the larger benchmark or run it before release.

A simple pytest gate can make thresholds explicit:

# test_retrieval_gate.py
from metrics import recall_at_k, reciprocal_rank

CASES = [
    {"ranked": ["reset", "policy"], "relevant": {"reset"}, "slice": "account"},
    {"ranked": ["invoice", "policy"], "relevant": {"invoice"}, "slice": "billing"},
]

def test_critical_retrieval_floor() -> None:
    recalls = [recall_at_k(case["ranked"], case["relevant"], 2) for case in CASES]
    mrrs = [reciprocal_rank(case["ranked"], case["relevant"]) for case in CASES]

    # Illustrative project thresholds, replace with approved product targets.
    assert sum(recalls) / len(recalls) >= 0.95
    assert sum(mrrs) / len(mrrs) >= 0.90

In a real gate, populate ranked from the candidate retriever and save the complete results as an artifact. Cache embeddings for unchanged corpus text when the experiment permits it, using a cache key that includes model, requested dimensions, normalization, and exact text hash. Never reuse a cache across incompatible configurations.

Fail separately for infrastructure errors. A provider timeout is not evidence that search relevance regressed, but it is still a pipeline failure that needs visibility. Limit retries and show the original error.

The CI summary should include configuration manifest, corpus and benchmark versions, metric deltas, slice results, hard-invariant failures, latency, and a link to changed rankings. That makes a quality gate auditable instead of a mysterious red check.

Interview Questions and Answers

Q: What is the difference between Recall@k and precision@k?

Recall@k measures how many labeled relevant documents appear in the first k results. Precision@k measures what fraction of those k results are relevant. I prioritize based on the experience: RAG often needs recall, while a short search panel may care strongly about precision.

Q: Why is cosine similarity not itself a quality metric?

Cosine similarity is a scoring function between vectors. A high number has no universal semantic meaning across models or datasets. Quality comes from whether the induced ranking satisfies reviewed relevance expectations.

Q: How would you test an ANN index?

I would hold vectors constant, compute exact top k neighbors, query the ANN index, and measure overlap as ANN recall@k. I would also test insert, update, delete, persistence, filters, scale, and tail latency. Business relevance remains a separate evaluation.

Q: What makes a good hard negative?

A hard negative looks similar in vocabulary or topic but does not satisfy the query intent. It reveals whether the retriever distinguishes nearby concepts instead of matching surface terms. Domain experts are valuable when curating these cases.

Q: How do you test a vector search migration?

I version the old and new complete configurations, re-embed and re-index as required, and run a paired benchmark on the same corpus snapshot. I compare hard invariants, per-query deltas, important slices, relevance metrics, latency, and cost. I keep a rollback-ready baseline.

Q: Why should tenant leakage be a hard assertion?

A cross-tenant result is an authorization defect, not a minor ranking error. Averaging it into precision can hide a serious exposure. I test filtering before retrieval where possible and assert zero forbidden IDs at every returned rank.

Common Mistakes

• Checking only vector shape: Valid arrays can still produce useless rankings. Add reviewed relevance tests.
• Using random negatives: Easy distractors inflate scores. Curate keyword-rich and semantically close negatives.
• Reporting only an average: Inspect per-query changes and critical slices.
• Comparing different corpora: Candidate and baseline must use the same snapshot and labels.
• Mixing vector generations: Re-embed consistently when model or dimensions change.
• Treating similarity as probability: Calibrate decisions on product data instead of assuming a universal cutoff.
• Ignoring filters and lifecycle: Search must remain correct after permission changes, updates, and deletes.
• Optimizing Recall@20 for a product that uses five results: Gate the cutoff that the downstream system consumes.
• Letting generated labels grade generated data: Include independent domain review and a held-out set.

Conclusion

Testing embeddings and vector search quality is a ranking and systems problem. Build a reviewed relevance dataset, verify embedding contracts, measure rankings with metrics that match the product, compare ANN results with exact neighbors, and test chunking, filters, index lifecycle, latency, and cost as one versioned configuration.

Begin with a compact benchmark of critical queries and hard negatives. Save raw rankings, establish an approved baseline, and automate paired regression checks. The result is evidence that a retrieval change helps users, not merely evidence that a vector endpoint returned numbers.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Testing semantic search quality (2026)
• Testing sorting and search APIs (2026)
• Pairwise and orthogonal array testing (2026)
• Testing an AI search feature (2026)
• Testing bulk and batch endpoints (2026)
• Testing pagination and filtering (2026)