AI test data generation with Faker and LLMs (2026)

TL;DR

The reliable pattern for AI test data generation with Faker and LLMs is schema first, rules second, Faker for exact fields, and an LLM for bounded semantic content. Validate every record, keep sensitive production data out, save generation lineage, and commit a reviewed fixture or seed for repeatable regression tests.

Key Takeaways

• Use Faker for deterministic identities and fields, and use an LLM only for semantic variation that rules cannot express economically.
• Define a strict schema and business constraints before generating any records.
• Parse LLM responses into typed models, then validate domain rules independently of the model.
• Keep seeds, prompts, model identifiers, schema versions, and generated artifacts traceable in CI.
• Never transform production personal data through an unapproved model or assume synthetic means automatically private.
• Separate valid baseline data, boundary data, invalid negative data, and realistic narrative data into explicit datasets.
• Measure duplicate rate, constraint failures, coverage, review effort, reproducibility, and defects found rather than raw record count.

AI test data generation with Faker and LLMs works best as a hybrid pipeline, not as a prompt that asks for a thousand realistic customers. Faker is fast, local, seedable, and good at schema-shaped identities. An LLM is useful for bounded semantic variation such as support narratives, ambiguous product descriptions, or combinations of business intent that would be tedious to enumerate.

This guide builds a runnable Python approach for 2026. It uses Faker for deterministic fields, Pydantic for validation, the current OpenAI Responses API for structured semantic output, and pytest for executable examples. It also covers privacy, negative data, databases, CI reproducibility, and the point where simpler rule-based generation is the better choice.

TL;DR

The core pipeline is schema -> constraints -> deterministic generator -> optional LLM enrichment -> validation -> deduplication -> reviewed artifact -> test execution.

1. Design AI Test Data Generation With Faker and LLMs

Start by classifying the data, because not every field should be generated the same way. Structural fields have types and formats: UUID, email, ISO country code, timestamp, enum, and integer range. Relational fields must agree with other records: order customer IDs must exist, refunds cannot exceed captured payment, and a cancelled subscription cannot renew. Semantic fields carry meaning: a complaint narrative, an ambiguous search query, or a product description with a specific policy conflict.

Faker handles structural variety well. Rule-based factories handle boundaries and relationships. An LLM can help with semantic variety, but it should receive a narrow job and return a strict type. Human-curated fixtures remain best for high-value canonical scenarios where the exact meaning and expected result must be stable. A mature dataset uses all four.

Define dataset partitions explicitly:

• Valid baseline records for ordinary workflows.
• Boundary records at and around limits.
• Invalid records for validation and error handling.
• Stateful scenarios spanning related entities.
• Semantic cases with labeled intent or expected properties.
• Security and abuse cases from an approved curated source.

Do not mix invalid data into a positive dataset without a validity label. The test needs to know whether rejection is the expected outcome. Every generated artifact should have a schema version, generator version, seed or run ID, creation time, and purpose.

2. Set Up the Python Project

Create an isolated environment and install the required packages. This guide uses Python 3.11 or later syntax, but the APIs shown are not tied to a single patch version.

python -m venv .venv
source .venv/bin/activate
python -m pip install Faker openai pydantic pytest requests
export OPENAI_API_KEY='replace-with-your-test-key'

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 and set the environment variable according to your shell. Keep .venv, generated secrets, and local .env files out of source control. Do not hardcode the API key in examples, fixtures, CI logs, or committed configuration.

Use a small project layout:

test-data-lab/
  data/
    reviewed/
  src/
    models.py
    identity_factory.py
    semantic_factory.py
    pipeline.py
  tests/
    test_pipeline.py
  pyproject.toml

Pin compatible dependency versions through the project's normal lock process once the proof of concept works. Provider SDKs and model availability change, so record the tested environment and review official documentation during upgrades. Keep a recorded-output path for unit tests so ordinary CI does not require network access or consume model budget.

Before using an external model, confirm that the provider and data class are approved. The examples use invented product requirements and create synthetic text from scratch. They do not send production rows for rewriting.

3. Define the Schema and Business Constraints First

A schema makes generated data machine-checkable, but business rules provide the real oracle. Define both before writing the generator. The example customer model uses Pydantic for types and field constraints, then applies a model-level relationship check.

from datetime import date
from typing import Literal
from pydantic import BaseModel, EmailStr, Field, model_validator

class Customer(BaseModel):
    customer_id: str = Field(pattern=r'^cus_[0-9a-f]{12}
#39;)
    full_name: str = Field(min_length=2, max_length=100)
    email: EmailStr
    locale: Literal['en_US', 'en_GB', 'en_IN']
    birth_date: date
    plan: Literal['free', 'pro', 'team']
    seats: int = Field(ge=1, le=500)
    marketing_opt_in: bool

    @model_validator(mode='after')
    def plan_matches_seats(self):
        if self.plan != 'team' and self.seats != 1:
            raise ValueError('Only team plans may have multiple seats')
        return self

Install email-validator if the local Pydantic setup requests it for EmailStr: python -m pip install email-validator. The regular expression controls the generated internal ID format, and the model validator enforces a relationship that types alone cannot express.

Keep transport schema and domain rules separate when that makes ownership clearer. JSON Schema or an API specification can validate the wire contract. Domain validation can check effective dates, account state, inventory, currency compatibility, or tenant ownership. Database constraints provide another layer, not a replacement for earlier validation.

Version the schema. If team later requires at least two seats, old fixtures may become intentionally historical or invalid. A migration should state which behavior is expected rather than silently regenerating data until tests pass.

4. Generate Deterministic Identities With Faker

Faker supports local providers and deterministic seeding. Seed an instance for an isolated factory rather than relying only on global state. Use a separate random generator when you need deterministic business choices. The following code produces repeatable customers and validates each through the schema.

import hashlib
import random
from datetime import date
from faker import Faker

from models import Customer

class CustomerFactory:
    def __init__(self, seed: int = 20260713):
        self.fake = Faker(['en_US', 'en_GB', 'en_IN'])
        self.fake.seed_instance(seed)
        self.random = random.Random(seed)

    def build(self, index: int) -> Customer:
        plan = self.random.choice(['free', 'pro', 'team'])
        seats = self.random.randint(2, 25) if plan == 'team' else 1
        raw_id = f'{index}:{self.fake.uuid4()}'.encode()
        customer_id = 'cus_' + hashlib.sha256(raw_id).hexdigest()[:12]
        locale = self.random.choice(['en_US', 'en_GB', 'en_IN'])

        return Customer(
            customer_id=customer_id,
            full_name=self.fake.name(),
            email=self.fake.unique.safe_email(),
            locale=locale,
            birth_date=self.fake.date_between(
                start_date=date(1940, 1, 1),
                end_date=date(2008, 7, 13),
            ),
            plan=plan,
            seats=seats,
            marketing_opt_in=self.fake.boolean(),
        )

factory = CustomerFactory(seed=42)
customers = [factory.build(index) for index in range(10)]
print(customers[0].model_dump_json(indent=2))

The seed makes the local sequence repeatable for the same compatible environment and generator code. Do not assume that a seed guarantees identical values across every future Faker upgrade or provider-data change. For long-lived regression fixtures, generate once, validate, review, and commit the approved artifact. Use seeds to reproduce generation within a known environment.

The unique proxy tracks previously returned values and can exhaust a small domain. Clear it with fake.unique.clear() between independent batches when appropriate. Uniqueness in memory does not replace a database unique constraint under concurrent tests.

5. Create Boundary and Invalid Data With Rules

Faker's realism can distract from the cases most likely to find defects. Boundaries should be explicit. If a coupon percentage supports 1 through 50, generate 0, 1, 2, 49, 50, and 51 with labels showing the expected validity. Do not hope that random generation eventually hits them.

from dataclasses import dataclass

@dataclass(frozen=True)
class PercentCase:
    value: int
    valid: bool
    reason: str

def percent_cases() -> list[PercentCase]:
    return [
        PercentCase(0, False, 'below minimum'),
        PercentCase(1, True, 'minimum'),
        PercentCase(2, True, 'just above minimum'),
        PercentCase(49, True, 'just below maximum'),
        PercentCase(50, True, 'maximum'),
        PercentCase(51, False, 'above maximum'),
    ]

Build invalid records deliberately. Examples include missing required fields, wrong types at the transport layer, invalid enum members, impossible dates, duplicate keys, broken references, and state conflicts. Store the expected error code or rule, not just valid: false. That turns the record into an executable test case.

Separate syntactically invalid JSON from schema-invalid JSON. A Python dictionary cannot represent malformed JSON text, so keep raw strings for parser tests. Likewise, Pydantic will reject an invalid object before an API receives it. That is useful for factories, but negative API tests sometimes need to bypass the domain model and send raw payloads.

Use pairwise or combinatorial techniques for many independent fields, and state-machine models for dependent workflows. Pure random combinations often create mostly invalid noise and poor diagnostic value.

6. Generate Bounded Semantic Data With an LLM

Use an LLM where meaning matters and simple rules become costly. Good examples include varied support messages with a known intent, search queries containing ambiguity, accessibility descriptions, or product reviews that express defined themes. Do not use an LLM merely to generate UUIDs, dates, or enum values that Faker and rules handle better.

The current OpenAI Python SDK supports client.responses.parse with a Pydantic output model. This runnable example generates invented support scenarios. The prompt defines allowed labels and asks for no personal data. Pydantic enforces the shape.

from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, Field

class SupportScenario(BaseModel):
    case_id: str = Field(pattern=r'^sem_[0-9]{3}
#39;)
    intent: Literal['duplicate_charge', 'password_reset', 'cancel_plan']
    message: str = Field(min_length=20, max_length=300)
    expected_action: Literal['open_billing_case', 'send_reset_link', 'confirm_cancellation']
    ambiguity: str = Field(max_length=160)

class ScenarioBatch(BaseModel):
    scenarios: list[SupportScenario] = Field(min_length=3, max_length=12)

client = OpenAI()

def generate_scenarios(count: int = 6) -> list[SupportScenario]:
    response = client.responses.parse(
        model='gpt-5.6',
        input=[
            {
                'role': 'system',
                'content': (
                    'Create fictional software support messages. Use no real people, '
                    'companies, account numbers, addresses, or contact details. '
                    'Match the expected action to the intent exactly.'
                ),
            },
            {
                'role': 'user',
                'content': f'Generate {count} varied scenarios across all allowed intents.',
            },
        ],
        text_format=ScenarioBatch,
    )
    if response.output_parsed is None:
        raise RuntimeError('No structured scenario batch returned')
    return response.output_parsed.scenarios

Structured output guarantees schema conformance when parsing succeeds, not factual or domain correctness. Validate that intent and action match, scan for prohibited data patterns, deduplicate messages, and review a sample. Save the prompt, model identifier, schema version, and raw run ID.

7. Compose a Hybrid Record Pipeline

Keep semantic generation independent from identity generation. The LLM does not need a person's name or email to write a support message about a duplicate charge. Generate the narrative first, validate it, then attach a Faker customer locally if the downstream test needs an authenticated record. This minimizes data exposure and makes each part replaceable.

from pydantic import BaseModel

from identity_factory import CustomerFactory
from semantic_factory import SupportScenario, generate_scenarios

class SupportTicket(BaseModel):
    ticket_id: str
    customer_id: str
    message: str
    expected_action: str
    intent: str

def build_tickets(seed: int = 42) -> list[SupportTicket]:
    identity_factory = CustomerFactory(seed=seed)
    scenarios = generate_scenarios(count=6)
    tickets = []

    for index, scenario in enumerate(scenarios):
        customer = identity_factory.build(index)
        tickets.append(
            SupportTicket(
                ticket_id=f'tkt_{index:04d}',
                customer_id=customer.customer_id,
                message=scenario.message,
                expected_action=scenario.expected_action,
                intent=scenario.intent,
            )
        )
    return tickets

In production-quality tooling, inject the semantic generator behind an interface. One implementation calls the model, another loads reviewed JSON, and a third returns small fixtures for unit tests. This prevents external calls from leaking into every test and makes failures attributable.

Use a run manifest containing the identity seed, generator commit, Faker version, prompt version, model identifier, schema version, and output checksum. An LLM response may not reproduce byte for byte even with the same request, so the saved reviewed artifact is the regression input. Regeneration is a controlled content update, not an invisible test setup step.

8. Validate, Deduplicate, and Review the Dataset

Validation should be layered. First parse the schema. Then apply business rules, referential integrity, prohibited-pattern checks, uniqueness, distribution checks, and semantic labels. Finally, review a risk-based sample or all high-risk cases. A dataset can pass its schema while containing six paraphrases of the same easy scenario.

Create exact and normalized duplicate detection. Normalize case, whitespace, and punctuation for simple text. Use token or embedding similarity only when necessary, and calibrate its threshold with reviewed examples. Similar messages are not always duplicates if their expected actions differ. Never delete a record automatically without retaining the decision evidence.

Check coverage against a declared target matrix. For the support example, count each intent, ambiguity type, length band, locale if supported, and expected action. Fail generation if a required category is absent. Distribution checks should use product goals or privacy-safe measured properties, not a fabricated assumption that equal representation is always realistic.

Add provenance to every record: rule, Faker, LLM, or curated; generator and schema version; review status; and intended suite. For semantic data, store the label source and reviewer where policy permits. A test failure should be able to distinguish an application regression from a bad generated label.

For more schema and contract patterns, read the API test data management guide.

9. Protect Privacy, Security, and Fairness

Synthetic does not automatically mean anonymous. A model asked to imitate real support tickets may reproduce unique phrases or personal details from the examples. Combining individually fake fields can also accidentally create a real person's contact information. Use reserved domains such as example.com or provider methods intended for safe examples, avoid live phone and payment routing, and prevent automated messages from reaching external recipients.

Do not send raw production rows to an LLM for anonymization unless an approved privacy process explicitly permits it. Redaction can miss indirect identifiers, and the provider relationship may create a new data boundary. Prefer generating from schema, allowed value ranges, aggregate distributions that passed privacy review, and invented domain rules. High-fidelity statistical synthetic data is a specialized discipline, not a side effect of good prompting.

Scan generated text for secrets, email addresses, phone-like patterns, account identifiers, and prohibited names before persistence. Pattern scans are imperfect, so combine them with narrow prompts, schemas, review, and access controls. Log only the identifiers needed for traceability. Protect generated datasets if they encode confidential business rules even when they contain no personal data.

Fairness and representation require explicit goals. Locale-aware names do not prove realistic or fair coverage. Define supported populations and behaviors with product and domain teams, review language quality with qualified speakers, and test outcomes by relevant slice. Avoid sensitive-attribute inference. If sensitive attributes are necessary for approved fairness evaluation, handle them under dedicated governance.

Use the test data privacy checklist before moving a prototype into shared CI.

10. Seed Databases and APIs Safely

Insert data through the same public API when the contract, authorization, and side effects are part of the test. Use a dedicated seeding API or direct database setup when the goal is to create a precondition quickly and the team owns that boundary. Keep test setup explicit so failures can distinguish seeding from the behavior under examination.

The following standard-library SQLite example creates customers transactionally and rolls back automatically on error. It is runnable with the earlier Customer objects.

import sqlite3
from collections.abc import Iterable

from models import Customer

def seed_customers(connection: sqlite3.Connection, customers: Iterable[Customer]) -> None:
    rows = [
        (
            item.customer_id,
            item.full_name,
            str(item.email),
            item.locale,
            item.birth_date.isoformat(),
            item.plan,
            item.seats,
            int(item.marketing_opt_in),
        )
        for item in customers
    ]
    with connection:
        connection.executemany(
            '''
            INSERT INTO customers (
                customer_id, full_name, email, locale, birth_date,
                plan, seats, marketing_opt_in
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
            ''',
            rows,
        )

Use a test namespace or run ID in entities so cleanup is targeted. Make cleanup idempotent and ensure it cannot touch non-test records. For parallel execution, allocate distinct accounts, schemas, or tenant identifiers rather than relying only on unique emails. Database constraints remain enabled so bad factory output fails early.

Avoid shared mutable fixtures that tests update in different ways. Generate or reset the smallest state needed per test. Expensive reference data can be session-scoped if it is immutable and safe to share.

11. Integrate Generated Data With pytest and CI

Parameterize tests from reviewed records rather than generating new LLM output inside every test function. Model calls introduce network, service, cost, and behavioral variability that make ordinary functional failures harder to diagnose. Run generation as a controlled job, validate and approve the artifact, then consume it deterministically.

import json
from pathlib import Path
import pytest

CASES = [
    pytest.param(row, id=row['case_id'])
    for row in (
        json.loads(line)
        for line in Path('data/reviewed/support_cases.jsonl')
            .read_text(encoding='utf-8')
            .splitlines()
        if line.strip()
    )
]

@pytest.mark.parametrize('case', CASES)
def test_support_intent_classifier(api_client, case):
    response = api_client.post('/v1/classify', json={'message': case['message']})
    assert response.status_code == 200
    assert response.json()['intent'] == case['intent']

Give every case a stable ID so test reports identify the failing record. Check the fixture checksum or manifest in CI if accidental regeneration is a risk. Store validation reports as artifacts. A separate scheduled generation job can propose new data, but it should create a reviewed change rather than mutating the dataset used by a running pipeline.

Cache carefully. Caching a model response by prompt hash can control cost during development, but the key must include model, prompt, schema, and relevant configuration. Never let cache hits obscure which artifact a test consumed. Expire or deliberately migrate caches when dependencies change.

12. Maintain AI Test Data Generation With Faker and LLMs

Treat the generator and dataset as test products. Assign owners, review dependency updates, and track why cases are added or removed. When the API schema changes, migrate intentionally and preserve fixtures for backward-compatibility tests if required. When a real defect escapes, create the smallest synthetic reproduction and add it to the correct partition.

Measure useful outcomes:

• Schema and business-rule rejection rate during generation.
• Duplicate and near-duplicate rate.
• Required risk and category coverage.
• Human review acceptance and correction effort.
• Reproducibility of Faker-based batches in the locked environment.
• Number and importance of defects found by new semantic cases.
• Generation latency and provider usage where relevant.
• Stale cases after requirement or source changes.

Raw record count is a poor target. Ten traceable boundary cases can be more valuable than ten thousand fluent narratives with unclear labels. Periodically remove redundant cases, but preserve coverage and historical incident value.

Know when not to use an LLM. Use a table, factory, grammar, property-based generator, or curated corpus when it creates a stronger oracle at lower cost. The hybrid approach succeeds because each generator receives the task it handles best.

Interview Questions and Answers

These questions help you explain the design in a QA, SDET, or test platform interview.

Q: Why combine Faker with an LLM?

Faker provides fast, local, seedable structural fields. An LLM provides bounded semantic variation for narratives and intents. Keeping them separate reduces data exposure, cost, and nondeterminism while making each output easier to validate.

Q: Does a Faker seed guarantee the same data forever?

It helps reproduce a sequence for the same generator code and compatible dependency environment. Provider datasets or algorithms can change across upgrades. Long-lived regression data should be generated, validated, reviewed, and stored as an artifact.

Q: Why use structured LLM output?

A typed schema prevents missing fields and invalid shapes from entering the pipeline. It does not prove semantic correctness, privacy, uniqueness, or business validity. Independent validation and review remain required.

Q: Should an LLM generate negative test data?

It can suggest semantic invalid cases, but exact boundaries and malformed payloads are usually better generated by rules. Every negative record must state the expected rejection rule. This prevents invalid noise from being mistaken for a useful test.

Q: How do you prevent synthetic test data from reaching real people?

Use reserved domains, non-routable or approved test contacts, sandbox integrations, and outbound guards. Avoid random numbers that could contact real recipients. Test the guard and monitor test environments for unexpected egress.

Q: Should model generation run inside every CI test?

Usually no. Generate through a controlled job, validate and approve the artifact, then let tests consume it deterministically. Live generation belongs in dedicated evaluation or generator tests with explicit budget, retry, and artifact handling.

Q: How do you test the generator itself?

I test schema conformance, business rules, uniqueness, relationships, coverage, prohibited patterns, deterministic behavior where promised, and failure handling. I also review semantic label accuracy and compare generator versions on a fixed validation set.

Q: How do you measure whether LLM-generated data is valuable?

I measure accepted unique scenarios, review correction effort, target coverage, defects found, and maintenance cost. I also track rejection, duplication, privacy flags, latency, and usage. More records alone do not indicate better testing.

Q: Can synthetic data replace all production-like testing?

No. Synthetic data is excellent for control, privacy, and boundaries, but it may miss real distributions, dependencies, and rare correlations. Use approved aggregate insights, production monitoring, and carefully governed validation where necessary.

Q: What belongs in a generation run manifest?

Include schema and generator versions, code commit, Faker version, seed, prompt version, model identifier, configuration, timestamp, output checksum, validation result, and review status. This lineage makes failures and regeneration decisions auditable.

Common Mistakes

• Asking an LLM to generate fields that Faker or simple rules can create more reliably.
• Generating data before defining schema, constraints, validity labels, and expected outcomes.
• Assuming structured output proves business correctness.
• Assuming a Faker seed reproduces identical values across every future upgrade.
• Calling the model from every regression test and creating cost, latency, and flake noise.
• Sending production records to an unapproved model for rewriting or anonymization.
• Using fake phone numbers or email addresses that can reach real recipients.
• Mixing valid and invalid records without expected error metadata.
• Optimizing record count while ignoring duplicates, coverage, review effort, and defect signal.
• Regenerating fixtures silently when a test fails.
• Ignoring relationships, cleanup, concurrency, and database constraints.

Conclusion

AI test data generation with Faker and LLMs is reliable when generation is divided by responsibility. Let schemas and rules define truth, let Faker create deterministic structural variety, and let an LLM add only the semantic variation that earns its extra uncertainty. Validate every layer and keep a reviewed artifact for repeatable tests.

Start with one dataset of 20 to 50 cases. Add a schema, a seeded identity factory, three explicit boundary groups, and one structured semantic generator. Run privacy and coverage checks, commit the reviewed output, and measure whether it finds behavior your existing fixtures missed.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• AI assisted API test generation (2026)
• AI powered test data masking (2026)
• Generating test cases from a PRD with AI (2026)
• Measuring cost per test with LLMs (2026)
• SQL for test data setup and teardown (2026)
• AI for exploratory test charters (2026)