Resource library

QA How-To

k6 Distributed Load Testing with the Kubernetes Operator (2026)

Learn k6 distributed load testing kubernetes operator setup, run a multi-runner TestRun, verify thresholds, and troubleshoot Kubernetes runner jobs safely.

18 min read | 2,689 words

TL;DR

Install the k6 Operator, store a tested k6 script in a ConfigMap, and apply a TestRun with parallelism greater than one. The operator creates coordinated Kubernetes jobs, divides the execution across runner pods, and reports threshold success or failure through runner output and TestRun status.

Key Takeaways

  • Use the TestRun custom resource to coordinate initializer, starter, and runner jobs.
  • Set parallelism to the number of runner pods, while k6 execution segments divide the workload.
  • Validate the script locally before packaging it in a ConfigMap.
  • Set requests and limits so Kubernetes can schedule runners predictably.
  • Use thresholds as machine-readable pass or fail criteria for the distributed run.
  • Inspect TestRun status, jobs, pod events, and logs in that order when diagnosing failures.

k6 distributed load testing kubernetes operator workflows let you generate load from several coordinated pods without building your own sharding system. You define one TestRun, and the operator prepares the script, starts the runners together, and assigns each runner an execution segment.

This tutorial takes you from an empty namespace to a verified four-runner test with thresholds and resource controls. For the larger test-design context, read the k6 performance engineering complete guide before you turn this example into a production workload.

You will use a safe public demo endpoint in the example. Replace it with a system you own or are explicitly authorized to test, then start with low traffic and increase it deliberately.

TL;DR

Component Purpose Created by
ConfigMap Stores the k6 JavaScript file You
TestRun Declares the distributed execution You
Initializer job Validates and prepares the test Operator
Starter job Coordinates runner start Operator
Runner jobs Generate the load Operator

The shortest path is: install the operator, create a namespace and ConfigMap, apply a TestRun with parallelism: 4, then inspect its status and runner logs. A parallelism value of four means four runner jobs, not four total virtual users. k6 divides scenario work across the execution segments, so keep test options in the script and treat the aggregate metrics as the result.

What You Will Build

By the end, you will have:

  • A dedicated k6-tests namespace.
  • The Grafana k6 Operator and its TestRun CRD.
  • A k6 script with a ramping workload, checks, tags, and thresholds.
  • A TestRun that distributes the script across four runner pods.
  • Repeatable commands to watch jobs, read logs, verify thresholds, and clean up.

The example is intentionally observable. A failed HTTP check affects checks, slow requests affect http_req_duration, and server errors affect http_req_failed. That separation helps you distinguish correctness, latency, and reliability failures.

Prerequisites

Use a conformant Kubernetes cluster and a kubectl client supported by that cluster. This tutorial assumes Kubernetes 1.30 or newer, kubectl within one minor version of the server, curl, and enough capacity for four small runner pods plus the operator. The manifests use k6.io/v1alpha1, which is the current TestRun API in the k6 Operator.

Check your tools and access:

kubectl version
kubectl cluster-info
kubectl auth can-i create customresourcedefinitions.apiextensions.k8s.io
kubectl auth can-i create jobs.batch --namespace k6-tests

You need cluster-level permission to install CRDs and the operator. If a platform administrator installs the operator for you, namespace-level permission to create ConfigMaps and TestRuns is enough for the remaining steps. For production, pin the reviewed operator release in GitOps instead of blindly tracking a mutable branch. The official release available when this tutorial was prepared is k6 Operator v1.4.0, and the operator bundle selects a tagged controller image.

Also install k6 locally if possible. Local validation provides the fastest feedback for JavaScript syntax and threshold behavior. The distributed jobs run k6 inside containers, so a local installation is helpful but not required to execute the Kubernetes test.

Step 1: Install the k6 Operator

Install the official bundle. It includes the CRDs, service account, RBAC, controller deployment, and k6-operator-system namespace.

curl -fsSLo k6-operator-bundle.yaml \
  https://raw.githubusercontent.com/grafana/k6-operator/main/bundle.yaml
kubectl apply -f k6-operator-bundle.yaml

Saving the manifest before applying it gives you an auditable artifact. In a long-lived environment, retrieve the bundle from a reviewed release tag or use the Grafana Helm chart with a pinned chart version. Do not grant the runner application credentials through the operator deployment. Runner secrets belong in the test namespace and should be referenced only by the runner specification.

Wait for the controller and confirm that Kubernetes recognizes TestRun:

kubectl rollout status deployment/k6-operator-controller-manager \
  --namespace k6-operator-system --timeout=120s
kubectl get crd testruns.k6.io
kubectl api-resources --api-group=k6.io

Verify: The rollout command reports successfully rolled out. The CRD command returns testruns.k6.io, and the API resources list includes TestRun. If the deployment never becomes ready, inspect it with kubectl describe deployment -n k6-operator-system k6-operator-controller-manager.

Step 2: Create and Validate the k6 Script

Create a directory named distributed-k6, then save this as distributed-k6/test.js:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    browse: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 20 },
        { duration: '1m', target: 20 },
        { duration: '30s', target: 0 },
      ],
      gracefulRampDown: '10s',
    },
  },
  thresholds: {
    checks: ['rate>0.99'],
    http_req_failed: ['rate<0.01'],
    'http_req_duration{endpoint:home}': ['p(95)<750'],
  },
  tags: {
    test_type: 'operator-tutorial',
  },
};

const baseUrl = __ENV.BASE_URL || 'https://quickpizza.grafana.com';

export default function () {
  const response = http.get(baseUrl, {
    tags: { endpoint: 'home' },
    timeout: '10s',
  });

  check(response, {
    'home status is 200': (res) => res.status === 200,
    'home body is not empty': (res) => res.body.length > 0,
  });

  sleep(1);
}

This is a complete k6 module. The ramping-vus executor raises the aggregate target to 20 VUs, holds it, then ramps down. In a distributed execution, k6 segments that scenario across the runners. Do not multiply the script target by parallelism; the target is the logical test target.

Run a small local check before sending the file to Kubernetes:

k6 run --vus 1 --duration 10s distributed-k6/test.js

Command-line shortcuts override compatible script options for this quick validation.

Verify: k6 parses the file, sends requests, prints both named checks, and ends without a JavaScript exception. The public endpoint can vary in latency, so investigate a threshold miss, but focus first on confirming valid syntax. Never continue to distributed execution when local parsing fails.

Step 3: Package the Script in a ConfigMap

Create an isolated namespace and package the exact tested file. Using --from-file preserves the filename that the TestRun will reference.

kubectl create namespace k6-tests --dry-run=client -o yaml | kubectl apply -f -
kubectl create configmap distributed-api-test \
  --namespace k6-tests \
  --from-file=test.js=distributed-k6/test.js \
  --dry-run=client -o yaml | kubectl apply -f -

ConfigMaps are convenient for small, single-file or modest multi-file tests. Kubernetes limits an individual ConfigMap to about 1 MiB. Use a persistent volume, a k6 archive, or a custom runner image for larger test suites and binary fixtures. Never put passwords or tokens in the script or ConfigMap because ConfigMaps are not secret storage.

Inspect the key without printing unrelated namespace data:

kubectl get configmap distributed-api-test \
  --namespace k6-tests \
  -o jsonpath='{.data.test\.js}' | head -n 5

Verify: The output begins with the http and check imports. If the JSONPath prints nothing, check that the ConfigMap contains a test.js data key with kubectl describe configmap -n k6-tests distributed-api-test.

Step 4: Define k6 Distributed Load Testing Kubernetes Operator Resources

Save the following as distributed-k6/testrun.yaml:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: distributed-api
  namespace: k6-tests
  labels:
    app.kubernetes.io/name: distributed-api-load-test
spec:
  parallelism: 4
  script:
    configMap:
      name: distributed-api-test
      file: test.js
  arguments: --tag environment=tutorial
  runner:
    env:
      - name: BASE_URL
        value: https://quickpizza.grafana.com
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi

parallelism: 4 requests four k6 runners. The operator also creates coordination jobs, so expect more than four jobs and pods during the lifecycle. Resource requests help the scheduler reserve capacity; limits protect the cluster from one runner consuming unbounded resources. They are starting values, not universal sizing advice. Watch CPU, memory, network, and k6 dropped iterations, then tune with evidence.

Keep non-secret environment values under runner.env. For credentials, create a Kubernetes Secret and use the TestRun runner environment mechanisms supported by your installed CRD. Confirm fields before deployment with kubectl explain testrun.spec.runner --api-version=k6.io/v1alpha1.

Validate the resource on the API server without creating it:

kubectl apply --dry-run=server -f distributed-k6/testrun.yaml
kubectl explain testrun.spec.parallelism --api-version=k6.io/v1alpha1

Verify: Server-side dry run reports testrun.k6.io/distributed-api created (server dry run), and kubectl explain describes the parallelism field. An unknown-field error means your installed CRD differs from the manifest, so inspect the CRD rather than guessing.

Step 5: Start the Distributed Test

Apply the resource and immediately watch the objects created in its namespace:

kubectl apply -f distributed-k6/testrun.yaml
kubectl get testrun --namespace k6-tests
kubectl get jobs,pods --namespace k6-tests --watch

The initializer prepares the script and execution plan. Runner jobs then start k6 processes, and the starter coordinates their launch. For parallelism n, a normal run has n + 2 jobs: one initializer, one starter, and n runners. Pod names include the TestRun name and role, but generated suffixes vary.

Do not apply another resource with the same name while the current execution is active. A TestRun represents an execution, and Kubernetes will treat another apply as an update, not a clean new run. Use a unique name per CI execution, or delete the completed resource before repeating this tutorial.

Stop watching with Ctrl+C after runners reach Running or Completed.

Verify: kubectl get jobs -n k6-tests shows an initializer, starter, and four runner jobs associated with distributed-api. Pods should progress from Pending or ContainerCreating to Running, then Completed. Pending pods are not a k6 failure until you inspect scheduling events.

Step 6: Monitor Logs and TestRun Status

Describe the custom resource first. It provides the controller-level view and preserves useful conditions when a job fails.

kubectl describe testrun distributed-api --namespace k6-tests
kubectl get testrun distributed-api --namespace k6-tests -o yaml

Next, list runner pods and stream logs from all pods matching the operator's runner label. Labels can evolve, so inspect labels when the selector returns nothing.

kubectl get pods --namespace k6-tests --show-labels
kubectl logs --namespace k6-tests \
  --selector=k6_cr=distributed-api \
  --all-containers=true \
  --prefix=true \
  --tail=100

If your operator version uses a different generated label, get runner pod names from kubectl get pods -n k6-tests and run kubectl logs -n k6-tests <pod-name>. Prefixing output matters because several runners emit similar summaries. In centralized observability, attach the TestRun name, namespace, scenario, and environment as dimensions so results from concurrent tests remain distinguishable.

For deeper telemetry, continue with the k6 OpenTelemetry trace export tutorial. Load-generator metrics tell you what the client observed, while application traces help explain which downstream operation consumed the time.

Verify: Runner logs show scenario progress and HTTP metric lines. The TestRun status advances rather than remaining indefinitely in initialization. If logs show connection errors from every runner, confirm cluster DNS, egress policy, TLS trust, and the target URL.

Step 7: Verify k6 Distributed Load Testing Kubernetes Operator Results

A test is not successful merely because every pod completed. Read the k6 summary and require all declared thresholds to pass. The relevant summary resembles this shape:

checks.........................: 100.00% ✓ ... ✗ 0
http_req_duration..............: avg=... p(95)=...
http_req_failed................: 0.00%   ✓ ... ✗ 0

Threshold markers and exact formatting can vary by k6 release, but a breached threshold causes k6 to exit unsuccessfully. Check the Kubernetes jobs after completion:

kubectl get jobs --namespace k6-tests
kubectl get pods --namespace k6-tests \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName'
kubectl get events --namespace k6-tests --sort-by=.lastTimestamp | tail -n 20

The node column confirms placement, not true geographic distribution. Several pods can land on one node unless you configure scheduling constraints. That is acceptable for learning, but a high-volume run should avoid saturating a single node or network interface. Add affinity or topology spread only after confirming your cluster has enough matching capacity.

Thresholds are the automation contract. In CI, fail the pipeline when a runner job fails or the TestRun reaches an error stage. Also retain raw metrics in an output backend when you need trend analysis; terminal summaries alone are poor evidence for regressions across releases.

Verify: Four runner jobs ran, runner logs contain request activity, all thresholds pass, and completed jobs have successful status. If checks pass but latency fails, treat it as a performance failure, not a flaky assertion.

Step 8: Repeat Safely and Clean Up

Delete the TestRun before repeating it with the same name. The operator owns the generated jobs and normally removes dependent resources according to its lifecycle behavior.

kubectl delete testrun distributed-api --namespace k6-tests
kubectl wait --for=delete testrun/distributed-api \
  --namespace k6-tests --timeout=60s
kubectl apply -f distributed-k6/testrun.yaml

For CI, generate a DNS-safe unique resource name, apply it, wait with a bounded timeout, collect logs and status even on failure, then delete it. Avoid concurrent load tests against the same target unless overlap is part of the test plan. Two individually reasonable tests can combine into unsafe traffic.

When you have finished the tutorial, remove its namespace:

kubectl delete namespace k6-tests

Keep the operator if other teams use it. If this was an isolated cluster and you installed from the saved bundle, uninstall with kubectl delete -f k6-operator-bundle.yaml. Confirm ownership before removing cluster-scoped CRDs because deleting a CRD also deletes its custom resources.

Verify: After namespace deletion, kubectl get namespace k6-tests returns NotFound. If you repeated instead, a fresh set of jobs appears and the new execution begins from initialization.

Troubleshooting

Problem: no matches for kind TestRun -> The CRD is missing or discovery is stale. Run kubectl get crd testruns.k6.io, reinstall the reviewed operator bundle if it is absent, then retry after kubectl api-resources --api-group=k6.io lists TestRun.

Problem: initializer fails or reports that test.js is missing -> Confirm the TestRun namespace, ConfigMap name, and file key are identical. Inspect kubectl get configmap -n k6-tests distributed-api-test -o yaml. Recreate the ConfigMap from the locally validated file.

Problem: runner pods remain Pending -> Run kubectl describe pod -n k6-tests <pod-name> and read Events. Insufficient CPU or memory, an impossible node selector, taints, quotas, and unbound volumes are scheduler issues. Reduce parallelism or requests only if the resulting generator capacity is still adequate.

Problem: pods finish with Error -> Read the failed runner log, then the TestRun description. Use kubectl logs -n k6-operator-system deployment/k6-operator-controller-manager -c manager for controller errors. Add arguments: --verbose temporarily when ordinary k6 output is insufficient.

Problem: all requests time out inside Kubernetes but work locally -> Test DNS and egress from an allowed diagnostic pod. Review NetworkPolicies, proxies, service names, certificate authorities, and cloud firewall rules. A laptop's route to a private endpoint does not prove runner pods have the same route.

Problem: throughput is lower than planned -> Inspect runner CPU throttling, memory, node network capacity, dropped_iterations, target response time, and scenario configuration. Increasing VUs or parallelism cannot fix a saturated target or constrained generator. Establish generator headroom with a controlled calibration run.

Best Practices

  • Prove script correctness locally, then prove distribution at low load before scaling.
  • Pin and review the operator, runner image, and manifests in production workflows.
  • Put thresholds in code and treat breaches as test failures.
  • Label every run with environment, service, test type, and a unique execution identifier.
  • Store secrets in Kubernetes Secrets or an approved external secret system, never ConfigMaps.
  • Set resource requests from observed consumption and preserve headroom on generator nodes.
  • Separate load generators from the target workload when co-location would distort results.
  • Archive manifests, logs, k6 version, test revision, and raw metrics for reproducibility.
  • Obtain authorization, define abort criteria, and coordinate high-volume tests with service owners.

Where To Go Next

Use the complete k6 performance engineering guide to add workload modeling, capacity interpretation, CI quality gates, and a mature observability strategy. The Operator solves orchestration, but it cannot decide whether your traffic model represents real users.

Expand carefully based on protocol and evidence needs:

Your next practical change should be replacing the demo URL with an authorized staging service, lowering the first workload, and defining thresholds from an agreed service-level objective. Run that baseline repeatedly before increasing parallelism.

Interview Questions and Answers

Q: What does the k6 Operator add beyond running a k6 Kubernetes Job?

It provides a TestRun abstraction and coordinates initialization, distributed execution segments, synchronized startup, and runner jobs. A plain Job can run k6, but you would need to implement distribution and lifecycle coordination yourself.

Q: Does parallelism: 4 multiply a 20 VU scenario into 80 VUs?

No. The runners execute segmented portions of the logical test. Treat the scenario target as the aggregate plan and verify the aggregate result, rather than multiplying VUs by runner count.

Q: Why validate a script locally before creating a TestRun?

Local validation isolates syntax, imports, options, and basic target access from Kubernetes concerns. It prevents you from debugging a JavaScript error through several layers of jobs and controller status.

Q: How do thresholds behave in a distributed run?

Thresholds evaluate the test metrics used by the distributed execution and provide pass or fail criteria. Checks validate individual conditions, while thresholds enforce aggregate rates or percentiles. A clean pod phase is not a substitute for a passing threshold.

Q: How would you troubleshoot runners stuck in Pending?

Describe a pending pod and read scheduler events first. Check resource requests, quotas, taints, selectors, affinity, volume binding, and cluster autoscaler limits. Operator logs are secondary when Kubernetes never schedules the pod.

Q: When should a script use a volume instead of a ConfigMap?

Use a volume for a large or complex suite that approaches the ConfigMap size limit, contains many assets, or is produced by another workflow. A ConfigMap remains the simplest option for small text scripts. A custom runner image is another reproducible option for versioned dependencies.

Conclusion

k6 distributed load testing kubernetes operator execution is straightforward when you separate the responsibilities: k6 defines the workload and assertions, the TestRun defines orchestration, and Kubernetes supplies schedulable capacity. Start with a locally valid script, use explicit resources and thresholds, then verify status, jobs, logs, and aggregate results.

Keep the first real run small. Once it is repeatable and observable, scale parallelism only when generator measurements show that more runners are necessary and the target owners approve the traffic.

Interview Questions and Answers

What resources does the k6 Operator create for a distributed TestRun?

For parallelism n, a typical TestRun creates n runner jobs plus an initializer and a starter job. The initializer prepares the execution, the starter coordinates launch, and the runners generate load. The exact generated names contain the TestRun name and Kubernetes suffixes.

What is the difference between k6 VUs and TestRun parallelism?

VUs describe concurrency in the k6 workload model. Parallelism describes how many runner processes execute segmented parts of that logical workload. Increase parallelism to add generator capacity or distribution, not as a substitute for designing the intended VU scenario.

Why should resource requests be set on k6 runners?

Requests make scheduling capacity explicit and reduce unpredictable contention. Limits can protect the cluster, but overly tight CPU limits may throttle the load generator and distort results. Size both from calibration measurements and retain headroom.

How would you make a k6 Operator test reproducible in CI?

Pin reviewed operator and runner versions, version the script and TestRun manifest, and use a unique TestRun name. Capture status, runner logs, thresholds, raw metrics, target build, and environment labels. Always collect diagnostics before cleanup, including on failure.

How do you distinguish a generator bottleneck from a target bottleneck?

Monitor runner CPU, throttling, memory, network, and dropped iterations alongside target metrics and response time. If runners exhaust capacity, add or resize generators before interpreting the target result. If generators have headroom while target latency and saturation rise, investigate the application path.

What should you inspect when a TestRun fails before runners start?

Inspect TestRun conditions, initializer job status, pod events, and initializer logs. Verify the ConfigMap name and file key, CRD schema, service account permissions, image pulls, and scheduling capacity. Consult controller logs after checking the failing Kubernetes object.

Frequently Asked Questions

What is the k6 Operator?

The k6 Operator is a Kubernetes operator for running distributed k6 tests through custom resources such as TestRun. It creates and coordinates initializer, starter, and runner jobs from a declarative manifest.

How does k6 Operator parallelism work?

The TestRun parallelism field controls the number of k6 runner jobs. k6 divides the logical execution into segments across those runners, so parallelism should not be treated as a multiplier for the VU target in the script.

Can the k6 Operator run a multi-file test?

Yes. A ConfigMap can contain several source files, and the script file points to the entry point. For larger suites, use a k6 archive, persistent volume, or custom runner image instead of approaching the ConfigMap size limit.

How do I pass environment variables to k6 runner pods?

Declare non-secret variables under the TestRun runner environment configuration. Store credentials in a Kubernetes Secret or approved external secret manager and reference them through fields supported by your installed TestRun CRD.

How do I know whether a distributed k6 test passed?

Require the declared k6 thresholds to pass, then confirm successful runner jobs and a healthy TestRun status. Pod completion alone is insufficient because a test can execute fully and still breach a latency or error-rate threshold.

Why are k6 Operator runner pods stuck in Pending?

The most common causes are insufficient cluster capacity, quotas, taints, impossible scheduling constraints, or unbound storage. Describe a pending pod and read its Events section to get the scheduler's specific reason.

Related Guides