Using Ollama for offline test generation (2026)

TL;DR

Using Ollama for offline test generation requires more than calling a localhost endpoint. Pre-pull an approved local model, disable cloud features, pass a strict JSON Schema to the chat API, validate the response, and review candidates against traceability, coverage, privacy, and feasibility gates.

Key Takeaways

• Offline means pre-staging every model, package, document, and dependency before network isolation.
• Enable Ollama local-only mode and keep the API bound to loopback unless an approved network design requires more.
• Pass a JSON Schema through the documented format field and validate the returned content again with Pydantic.
• Generate from a coverage model and requirement evidence, not from a one-line request for all edge cases.
• Apply deterministic schema, duplicate, traceability, safety, and feasibility checks after model generation.
• Record model name, digest, prompt version, schema version, parameters, and source requirement version.
• Use local generation as a candidate-production step, with a qualified tester owning final coverage.

Using Ollama for offline test generation lets a QA team turn private requirements into candidate scenarios without sending those requirements to a hosted model, provided the runtime is genuinely configured and verified for local-only use. The practical pattern is to pre-stage an approved model, call Ollama's local /api/chat interface, require a JSON Schema, validate the response, and keep human review between generation and execution.

Offline generation is a data-control choice, not a guarantee of good testing. A local model can still misunderstand a rule, duplicate cases, invent a field, or propose destructive data. This guide separates privacy controls from quality controls and builds a runnable Python generator around Ollama's documented structured-output API. The result is a reviewable artifact, not an automatically trusted test suite.

TL;DR

A useful pipeline is requirement -> coverage model -> structured candidates -> deterministic validation -> tester review -> selected tests. Do not connect the model output directly to production execution.

1. What Using Ollama for Offline Test Generation Actually Means

Ollama runs local models behind a local HTTP service, and its documentation states that prompts and answers are not sent back to Ollama when running locally. Ollama also supports cloud-hosted models and web search, so a serious offline design must disable cloud features rather than assume every Ollama command is local. Current Ollama supports OLLAMA_NO_CLOUD=1 or disable_ollama_cloud in ~/.ollama/server.json for local-only mode.

Define offline in your threat model. A common definition is: requirement content, prompts, generated cases, model inference, logs, embeddings, and review artifacts remain on approved equipment, and the generation environment has no outbound route during use. Another organization may permit an internal package mirror but forbid public egress. Write the exact boundary and prove it.

The model must already exist locally. ollama pull needs network access, so perform that step in an approved staging phase, verify the artifact, and then move or isolate the system according to policy. Python wheels, system packages, source documents, and certificates also need staging. An air-gapped laptop with a missing Pydantic wheel is not a functioning offline pipeline.

Offline does not automatically mean private. Prompts can leak into shell history, application logs, crash reports, backups, screenshots, swap, or generated JSON committed to Git. Apply storage encryption, access control, data minimization, redaction, retention, and secure deletion. The local LLM setup for private test data provides a broader infrastructure checklist.

Finally, separate two claims: the inference stayed local and the output is correct. Network evidence supports the first. Requirements traceability, schema validation, and expert review support the second.

2. Choose a Local Model by Constraints, Not Hype

Model selection is an engineering experiment. The largest model you can download may be too slow, too memory-hungry, or impossible to operate on a CI worker. A small model may follow a strict schema but miss subtle state transitions. Choose candidates from approved licenses and artifacts, then evaluate them on a representative local benchmark.

This guide uses qwen3:8b because it is an available Ollama library tag and is small enough for many development machines, but it is an example, not a universal recommendation. You can set OLLAMA_MODEL to another approved, already-pulled text model. Keep the exact tag and digest in your manifest because mutable tags can resolve differently over time.

Build a small evaluation set before standardizing. Include a form with clear boundaries, a stateful workflow, a permissions matrix, a negative security rule, ambiguous acceptance criteria, and a deliberately impossible request. Have experienced testers label useful unique cases, unsupported assumptions, and critical omissions. Compare candidates on those review outcomes, latency, resource use, and schema success.

Do not publish a percentage from a handful of prompts as a general model ranking. Your benchmark describes your requirements, rubric, prompt, hardware, and model artifact. Re-run it when any of those changes.

3. Stage Ollama and Prove Local-Only Operation

Install Ollama through the approved platform process. During a network-enabled staging window, pull the selected model and inspect the local inventory:

ollama pull qwen3:8b
ollama list
curl http://localhost:11434/api/tags

The documented GET /api/tags endpoint returns installed model names, sizes, digests, families, parameter sizes, and quantization details. Capture the selected digest in a controlled manifest. Do not treat the friendly tag alone as immutable provenance.

Where you run ollama serve directly, start a loopback-only, local-only process:

OLLAMA_HOST=127.0.0.1:11434 OLLAMA_NO_CLOUD=1 ollama serve

Platform-managed Ollama applications and services receive environment variables differently. Apply the documented macOS, Linux systemd, or Windows configuration, restart the service, and inspect logs for the local-only setting. Ollama binds to 127.0.0.1:11434 by default. Do not change it to 0.0.0.0 merely to solve a client connection problem. A network bind requires authentication, segmentation, authorization, TLS termination, and an explicit threat review because the native local endpoint should not be exposed casually.

Test readiness without sending sensitive text:

curl http://localhost:11434/api/tags
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Reply with OK"}],
  "stream": false
}'

Then enforce the network boundary with the operating system, container, or isolated network, and observe connection attempts during a synthetic generation. Checking the URL in source code is insufficient. DNS, proxy, plugin, update, telemetry, or fallback behavior can create other routes. Record the proof method and date.

4. Define a Test-Case Contract Before Prompting

Free-form prose is easy to read but difficult to validate, diff, import, and govern. Define the minimum artifact that downstream QA work needs. Avoid trying to encode every test management field in the first schema. A focused case can include a stable candidate ID, title, priority, requirement references, preconditions, steps, expected result, technique, and tags.

The contract should distinguish facts from model suggestions. requirement_ids trace candidates to supplied source identifiers. It does not prove the scenario is required. assumptions makes missing facts visible instead of letting the model bury guesses inside steps. A review_status field should be added by the workflow after generation, not chosen by the model.

Use enumerations where the organization has a controlled vocabulary. Constrain empty strings and minimum list sizes. Require exactly one observable expected result per case unless the data model explicitly supports step-level results. Do not let a schema imply semantic correctness. JSON can be perfectly shaped and still assert the wrong rule.

A practical contract asks the generator to produce a suite:

• source_requirement_id identifies the source.
• coverage_notes summarizes partitions, boundaries, states, permissions, and failure paths considered.
• cases contains individual candidates.
• each case has id, title, priority, requirement_ids, preconditions, steps, expected_result, technique, tags, and assumptions.

The contract also creates a review interface. A tester can filter all candidates with assumptions, check whether every source rule has at least one candidate, and compare duplicate titles. The prompt engineering for test case generation guide explains how explicit input and output contracts improve candidate quality.

Version the schema independently from the prompt. A schema change can break importers even when generation quality stays the same.

5. Build a Runnable Structured Test Generator in Python

Install the Python client and Pydantic during staging:

python -m pip install ollama pydantic

Create generate_tests.py. The code uses Ollama's documented chat function, passes model_json_schema() through format, lowers temperature, and validates the returned message with Pydantic:

import json
import os
import sys
from typing import Literal

from ollama import chat
from pydantic import BaseModel, Field, model_validator


class CandidateTest(BaseModel):
    id: str = Field(min_length=1)
    title: str = Field(min_length=1)
    priority: Literal["critical", "high", "medium", "low"]
    requirement_ids: list[str] = Field(min_length=1)
    preconditions: list[str]
    steps: list[str] = Field(min_length=1)
    expected_result: str = Field(min_length=1)
    technique: Literal[
        "equivalence_partitioning",
        "boundary_value",
        "decision_table",
        "state_transition",
        "negative",
        "exploratory"
    ]
    tags: list[str]
    assumptions: list[str]

    @model_validator(mode="after")
    def reject_blank_steps(self):
        if any(not step.strip() for step in self.steps):
            raise ValueError("steps cannot contain blank strings")
        return self


class GeneratedSuite(BaseModel):
    source_requirement_id: str = Field(min_length=1)
    coverage_notes: list[str] = Field(min_length=1)
    cases: list[CandidateTest] = Field(min_length=1)


def generate(requirement_id: str, requirement: str) -> GeneratedSuite:
    schema = GeneratedSuite.model_json_schema()
    model = os.environ.get("OLLAMA_MODEL", "qwen3:8b")

    system_prompt = """
You are a senior software test designer.
Generate candidate tests only from the supplied requirement.
Expose missing facts in assumptions. Never invent credentials or production data.
Make each case atomic, observable, and distinct.
Use boundary, partition, decision-table, state, negative, or exploratory reasoning.
Return only data matching the supplied JSON Schema.
""".strip()

    user_prompt = f"""
Requirement ID: {requirement_id}
Requirement:
{requirement}

Required JSON Schema:
{json.dumps(schema)}
""".strip()

    response = chat(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        format=schema,
        options={"temperature": 0},
    )

    suite = GeneratedSuite.model_validate_json(response.message.content)
    if suite.source_requirement_id != requirement_id:
        raise ValueError("source_requirement_id does not match the request")
    return suite


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python generate_tests.py REQ_ID REQUIREMENT_TEXT")

    result = generate(sys.argv[1], sys.argv[2])
    print(result.model_dump_json(indent=2))

Run it with synthetic input first:

OLLAMA_MODEL=qwen3:8b python generate_tests.py   PAY-17   "The amount field accepts whole USD values from 1 through 500 inclusive."

The schema is passed both through format and in the prompt, which Ollama recommends for grounding. Pydantic validation is still required after generation.

6. Prompt From a Coverage Model, Not an Edge-Case Slogan

Generate all edge cases is not a test strategy. Give the model named techniques and product evidence. For the amount requirement, identify the valid partition 1 through 500, invalid partitions below 1 and above 500, boundary neighbors, non-integer types if the interface can receive them, blank input, and persistence or downstream behavior. Ask the model to state which technique produced each case.

For a workflow, provide states and allowed transitions. For permissions, provide roles, resources, actions, and ownership rules. For an API, provide the OpenAPI contract plus business constraints that the schema cannot express. For a UI, provide semantic behavior and relevant accessibility or DOM contracts, not a full unrelated page dump.

Use a coverage request with explicit exclusions:

Generate candidates for REQ-PAY-17 only.
Cover valid and invalid equivalence partitions, values at and directly beside
both numeric boundaries, blank input, and decimal input.
Do not invent currency conversion, fees, roles, or database behavior.
If type coercion is unspecified, record it as an assumption.
Each candidate must have one observable expected result.
Do not duplicate a boundary case as a generic negative case.

This instruction helps a smaller local model allocate attention. It also makes output review possible because the expected coverage dimensions were known before generation.

Separate generation passes when the requirement is large. First ask for a decision table or state model. Have a tester correct it. Then generate cases from the approved model. The intermediate representation is valuable evidence and reduces hidden reasoning errors.

For requirements-heavy workflows, see generating test cases from a PRD with AI. Do not paste an entire confidential PRD when a minimized, approved extract is enough.

7. Add Deterministic Validation After Model Generation

Schema validation catches shape errors, not duplicates, unsafe instructions, or missing coverage. Build a deterministic gate before human review. The gate should fail closed and preserve the raw generation artifact separately from the accepted artifact.

Useful checks include:

1. every candidate ID is unique and matches your naming pattern,
2. every requirement_id exists in the supplied source set,
3. titles are not exact duplicates after normalization,
4. steps and expected results contain no empty values,
5. no secrets, real customer identifiers, or forbidden environments appear,
6. destructive terms trigger an explicit safety review,
7. required techniques and boundary values are represented,
8. assumptions are visible and block automatic approval,
9. case count stays within a reviewable limit,
10. output contains no unknown imports or executable instructions.

A small duplicate and traceability validator can be added after Pydantic parsing:

def validate_suite(suite: GeneratedSuite, allowed_requirements: set[str]) -> None:
    ids = [case.id for case in suite.cases]
    if len(ids) != len(set(ids)):
        raise ValueError("candidate IDs must be unique")

    titles = [" ".join(case.title.lower().split()) for case in suite.cases]
    if len(titles) != len(set(titles)):
        raise ValueError("candidate titles must be unique")

    unknown = {
        req
        for case in suite.cases
        for req in case.requirement_ids
        if req not in allowed_requirements
    }
    if unknown:
        raise ValueError(f"unknown requirement IDs: {sorted(unknown)}")

Near-duplicate detection can assist review, but avoid an opaque similarity threshold that silently deletes cases. Present suspected pairs to a tester with their differing preconditions and oracles.

A model can also follow malicious instructions embedded in source documents. Treat requirements, tickets, and retrieved text as untrusted data. The system prompt should state that source content is evidence, not instructions to alter the generator or access files. Keep the generator process minimally privileged and unable to execute model output.

8. Turn Candidates Into a Human Review Workflow

Generated cases need a lifecycle. Store immutable raw output with a generation ID, then create a review copy with fields such as review_status, reviewer, review_notes, and supersedes. Do not let the model mark its own work approved.

Review in this order:

• Traceability: Can the reviewer point to the exact source rule for the expected result?
• Correctness: Does the oracle match the current product contract?
• Uniqueness: Does the candidate detect a failure not already covered?
• Feasibility: Can the precondition and data be created safely?
• Layer: Is UI, API, component, contract, or manual exploration cheapest and reliable?
• Safety: Could execution alter real users, payments, messages, or protected data?
• Maintainability: Will a failure show what product rule broke?

Use four decisions: accept, revise, reject, and clarify requirement. The last is essential. If the model exposes an ambiguity around decimal coercion, inventing an expected result would create false authority. Route the question to the product owner and generate or revise cases after the rule is decided.

Measure the process without rewarding volume. Track schema failure rate, duplicate suggestions, accepted unique candidates, critical reviewer corrections, time to review, and escaped requirement misunderstandings. A team that generates 100 cases and accepts 8 does not necessarily outperform a team that generates 15 focused candidates and accepts 10.

Preserve rejected candidates and reasons for a limited period if they help prompt improvement, subject to data retention. They are useful examples for a local benchmark. Do not feed private review content into a model training process without separate approval.

9. Scale Using Ollama for Offline Test Generation Reproducibly

Local generation has performance and variance constraints. Record the inputs needed to explain a run: Ollama version, model name, model digest, prompt version, schema version, generation options, requirement version or hash, host class, start time, duration, and validation results. Avoid storing sensitive requirement text twice in the manifest if a controlled source reference is enough.

Temperature zero reduces sampling variation but does not create mathematical determinism across model artifacts, runtime versions, hardware paths, or prompt changes. Treat regeneration as a new run. Diff normalized structured candidates by ID and semantic purpose, not only raw JSON order.

Batch carefully. One requirement per request improves traceability and isolates failures. Huge prompts can exceed useful context, encourage cross-requirement contamination, and create long review artifacts. If related rules must be evaluated together, include their identifiers and ask for explicit cross-rule cases.

Control concurrency based on memory and latency evidence. Multiple parallel requests can overload a local GPU or force model churn. Ollama exposes duration and token-related fields in API responses, and ollama ps shows loaded models and processor placement. Capture operational metrics without logging prompt bodies. Warm-up can reduce first-request variability, but document whether warm or cold latency is being compared.

For CI, decide whether generation belongs in the gate. Dynamic generation on every pull request can create variable, slow feedback. A more stable pattern is to generate candidates in a controlled review workflow, approve and version the selected deterministic tests, then execute those tests in CI. Use scheduled regeneration when requirements or the approved model changes.

Local inference saves hosted transfer but still consumes hardware, energy, storage, maintenance, and reviewer time. Evaluate total workflow value rather than calling offline generation free.

10. Secure and Govern the Offline Test-Generation Service

Keep Ollama on loopback for a single-user workstation. If multiple users need a service, place it behind an approved internal gateway that adds identity, authorization, TLS, rate limits, request size limits, logging policy, and network segmentation. Do not expose port 11434 directly to a shared network or the internet.

Run the generator with least privilege. It needs requirement input and an output directory, not source-control credentials, production database access, browser cookies, or shell execution rights. Sanitize error logs because Pydantic errors can include fragments of rejected output. Protect model files against unapproved replacement and record digests.

Create a data classification policy. Some inputs should remain prohibited even offline, such as secrets that are unnecessary for test design, unminimized regulated records, or live authentication material. Use synthetic examples and placeholders. Align backups and crash dumps with the same classification.

Govern the supply chain. Approve the Ollama binary, model license, model artifact, Python packages, and update path. Scan staged artifacts. Test an upgrade against the local benchmark before changing the shared digest. Rollback should restore both runtime and model, not only prompt text.

Finally, audit the claim. During a representative synthetic run, enforce denied egress and monitor attempted connections. Confirm local-only configuration after restarts and upgrades. Verify that logs, output, terminal history, and temporary files follow retention. A policy that was true on installation day can drift.

The AI test data generation guide offers complementary techniques for combining deterministic generators with model assistance. Prefer Faker or rule-based code when the desired data can be generated exactly. Use a local LLM where language and semantic variety add value.

Interview Questions and Answers

Q: What does offline test generation with Ollama require?

It requires a local model artifact, local runtime, staged dependencies, cloud features disabled, and an enforced network boundary during generation. I keep the API on loopback unless an approved shared-service design exists. I also inspect logs, storage, backups, and temporary files because local inference alone does not prevent data leakage. Network evidence proves locality, while test review proves quality.

Q: Why use structured outputs for generated tests?

A JSON Schema makes required fields, types, and controlled vocabularies machine-checkable. It enables deterministic validation, imports, diffs, and review filters. I still validate the returned content because transport success does not prove schema or semantic correctness. A well-shaped case can still contain a wrong oracle.

Q: How would you select an Ollama model for QA generation?

I would shortlist license-approved models that fit the target hardware and context needs. Then I would run a representative benchmark covering boundaries, state, permissions, ambiguity, and safety. I would compare schema success, useful unique candidates, critical omissions, latency, and reviewer effort. The chosen model name and digest would be pinned in a manifest.

Q: Can temperature zero make offline test generation deterministic?

No. It reduces sampling variation but runtime versions, model artifacts, hardware, prompt changes, and implementation details can still affect output. I treat every regeneration as a new run and record its provenance. Approved executable tests remain versioned separately from dynamic candidates.

Q: How do you prevent prompt injection from requirement documents?

I treat requirement text as untrusted data and state that it cannot override the generator's instructions. The process has no shell, production credentials, or unnecessary file access, and model output is never executed. I minimize retrieved context and validate output against an allowlisted schema. Suspicious source instructions are surfaced for review.

Q: What validation belongs after an Ollama response?

I validate schema, unique IDs, known requirement references, nonblank steps, duplicate titles, safety terms, assumptions, coverage dimensions, and reviewable size. I scan for secrets and forbidden environments. Human reviewers then verify correctness, feasibility, layer, and product traceability.

Q: Should test generation run dynamically in CI?

Usually I prefer a controlled generation and approval workflow, then CI executes versioned deterministic tests. Dynamic generation can add latency and output variance to pull request feedback. It can be appropriate for a specialized experiment if the model artifact, inputs, validation, budgets, and review policy are tightly controlled.

Q: How would you prove Ollama did not send prompts externally?

I would enable local-only mode, keep the service bound to loopback, deny outbound traffic at the operating system or network layer, and observe connection attempts during a synthetic run. I would also inspect Ollama and client configuration for cloud models, web search, proxies, and fallback paths. The proof is recorded and repeated after upgrades.

Common Mistakes

• Calling localhost and assuming the workflow is offline. Cloud features, web search, proxies, dependencies, and other software can still use the network. Disable and verify.
• Pulling a model after entering an air gap. Models and packages must be staged before isolation, with provenance and license review.
• Binding Ollama to every interface for convenience. 0.0.0.0 expands exposure. Keep loopback or add a designed gateway.
• Sending a whole confidential repository into the prompt. Minimize the source to the rules needed for the current generation.
• Requesting all edge cases. Name partitions, boundaries, states, decisions, permissions, and exclusions.
• Trusting JSON because it parses. Schema-valid output can still be duplicated, unsafe, infeasible, or factually wrong.
• Letting the model approve its own cases. Approval is a separate human-controlled workflow state.
• Using generated IDs as durable identity without governance. Detect collisions and distinguish candidate run identity from approved case identity.
• Assuming temperature zero guarantees repeatability. Record the complete generation provenance and diff new runs.
• Logging full prompts and responses by default. Operational telemetry should avoid sensitive bodies and follow retention policy.
• Executing model-produced commands or code automatically. Treat output as untrusted content and keep the process minimally privileged.
• Choosing a model only by size. Hardware fit, schema behavior, review yield, license, and operational cost all matter.

Conclusion

Using Ollama for offline test generation is valuable when privacy controls and test-quality controls are designed separately and verified together. Pre-stage an approved model, disable cloud paths, bind locally, use the documented structured-output interface, validate every field, and route candidates through qualified review.

Start with one synthetic boundary requirement and the runnable generator in this guide. Capture the model digest and prompt version, inspect the JSON, run deterministic gates, and have a tester record accept, revise, reject, or clarify decisions before introducing private requirements.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Java for Testers: Builder pattern for test data (2026)
• Prompt engineering for test case generation (2026)
• A/B test validation: A Complete Guide for QA (2026)
• AI assisted API test generation (2026)
• AI for exploratory test charters (2026)
• AI for flaky test root cause analysis (2026)