Load Test Kubernetes Horizontal Pod Autoscaling

TL;DR

Deploy a CPU-bound service with resource requests, configure an autoscaling/v2 HPA, and drive it with k6. Pass the test only when replicas increase within your allowed window, latency and errors stay inside thresholds, and replicas return to minimum after the scale-down stabilization period.

Key Takeaways

• Define an autoscaling hypothesis before generating traffic.
• Set CPU requests because HPA utilization is calculated against requested CPU.
• Use autoscaling/v2 behavior rules to make scale timing testable.
• Capture HPA conditions, events, pod counts, and request latency on one timeline.
• Assert service quality as well as replica growth.
• Keep load running long enough to observe multiple metrics and reconciliation cycles.
• Verify scale-down separately because stabilization intentionally delays it.

To load test Kubernetes Horizontal Pod Autoscaling, generate controlled traffic, record the HPA metrics and replica decisions, and assert both application quality and scaling time. A graph that eventually shows more pods is useful evidence, but it is not a complete autoscaling test.

This tutorial builds one repeatable CPU autoscaling experiment. Place it in the wider strategy from the Cloud-Native Performance Testing Complete Guide, which covers capacity, observability, infrastructure limits, and distributed-system bottlenecks.

You will deploy a small CPU-bound web service, apply an autoscaling/v2 HPA, run k6 from inside the cluster, and verify scale-up, steady behavior, and delayed scale-down. Use an isolated test cluster because sustained load consumes real compute and may affect neighboring workloads.

What You Will Build

You will build a test that can:

• deploy a dedicated hpa-lab namespace and CPU-bound sample service;
• expose the service only inside the cluster;
• scale from 1 to 8 replicas at a 50 percent average CPU target;
• generate constant request arrival with k6;
• prove that the HPA increases replicas within an explicit test window;
• correlate k6 latency and failures with HPA metrics, conditions, and events;
• prove that replicas return to the minimum after traffic stops.

The commands use plain manifests, kubectl, and k6. They work on a conformant Kubernetes cluster without requiring a vendor-specific dashboard.

Prerequisites

Use Kubernetes 1.30 or newer, kubectl within one supported minor version of the cluster, and Metrics Server compatible with that cluster. You also need permission to create a namespace, Deployment, Service, ConfigMap, Job, and HPA. The load Job pulls a public k6 container image, so cluster nodes need registry access.

Check the client and server versions:

kubectl version
kubectl auth can-i create deployments --namespace default
kubectl auth can-i create horizontalpodautoscalers.autoscaling --namespace default

Confirm the resource metrics API is available:

kubectl get --raw '/apis/metrics.k8s.io/v1beta1/nodes' >/dev/null
kubectl top nodes

If the raw API request fails, install or repair Metrics Server before continuing. A healthy HPA needs fresh resource metrics. Also confirm cluster autoscaling or spare node capacity can schedule up to eight sample pods. HPA creates pod demand, but it does not create nodes itself.

For a local cluster, allocate at least 4 CPUs and enough memory for the control plane plus the test pods. Commands below assume a POSIX shell.

TL;DR

Treat every numeric limit in this tutorial as a starting hypothesis, not a universal benchmark. Calibrate rates and thresholds against your service-level objective, pod cost, startup time, and cluster capacity.

Step 1: Write the Autoscaling Test Hypothesis

Before creating resources, define a falsifiable contract. For this lab, use the following hypothesis:

At 40 requests per second for 6 minutes, average pod CPU will cross 50 percent, desired replicas will exceed 1 within 120 seconds, all desired replicas will become Ready, failed requests will remain below 1 percent, and replicas will return to 1 after the configured 60-second scale-down stabilization window plus metrics and reconciliation delay.

The traffic rate is illustrative. If your cluster reaches no CPU pressure, increase it gradually. If the service saturates before another pod becomes Ready, lower the rate and investigate startup or scheduling latency.

Create a test directory and record a baseline:

mkdir -p hpa-load-test/results
cd hpa-load-test
kubectl get nodes -o wide > results/nodes-before.txt
kubectl top nodes > results/node-usage-before.txt

Save the hypothesis beside the run results in your test management system or CI metadata. Record the cluster identity, application image digest, manifest revision, k6 version, and start time. Those details distinguish a regression from a different environment.

Verification: open both baseline files. Every schedulable node should appear, and kubectl top nodes should return recent CPU and memory values rather than Metrics API not available. Do not proceed with missing metrics.

Step 2: Deploy a CPU-Bound Test Service

Create app.yaml. CPU requests are essential because a utilization-based HPA compares measured CPU consumption with requested CPU. A missing request can leave utilization undefined for a pod.

apiVersion: v1
kind: Namespace
metadata:
  name: hpa-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-app
  namespace: hpa-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cpu-app
  template:
    metadata:
      labels:
        app: cpu-app
    spec:
      containers:
        - name: server
          image: registry.k8s.io/hpa-example:1.0
          ports:
            - name: http
              containerPort: 80
          resources:
            requests:
              cpu: 200m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 2
            periodSeconds: 3
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: cpu-app
  namespace: hpa-lab
spec:
  selector:
    app: cpu-app
  ports:
    - name: http
      port: 80
      targetPort: http

Apply it and wait for readiness:

kubectl apply -f app.yaml
kubectl -n hpa-lab rollout status deployment/cpu-app --timeout=120s
kubectl -n hpa-lab get pods -l app=cpu-app -o wide

The service is intentionally ClusterIP. The load generator will run in the same namespace, avoiding ingress, external load balancers, and internet variability. Test those layers separately when they are part of the production request path.

Verification: rollout status must report a successful rollout, and one pod must show 1/1 Running. Run kubectl -n hpa-lab run curl-check --rm -i --restart=Never --image=curlimages/curl -- curl --fail --silent http://cpu-app/. The command should print OK!.

Step 3: Configure Testable HPA Behavior

Create hpa.yaml with the stable autoscaling/v2 API. The policies allow rapid scale-up while giving scale-down a 60-second stabilization window. Production often uses a longer scale-down window to avoid oscillation. The shorter lab value keeps the tutorial practical.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-app
  namespace: hpa-lab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: cpu-app
  minReplicas: 1
  maxReplicas: 8
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 2
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 60
      selectPolicy: Max
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Apply and inspect it:

kubectl apply -f hpa.yaml
kubectl -n hpa-lab get hpa cpu-app
kubectl -n hpa-lab describe hpa cpu-app

At idle, TARGETS should eventually show a numeric current percentage followed by /50%. A value such as <unknown>/50% for several collection cycles means the HPA cannot calculate current utilization. Inspect Metrics Server, pod CPU requests, and HPA conditions.

The controller calculates a desired replica count from the relationship between current and target metrics, then applies tolerance, missing-metric handling, stabilization, and behavior policies. Therefore, do not assert an exact one-step replica sequence. Assert bounded outcomes and explain decisions from recorded conditions.

Verification: kubectl -n hpa-lab get hpa cpu-app -o jsonpath='{.spec.metrics[0].resource.target.averageUtilization}{"\n"}' must print 50. The status should name one current replica, one desired replica, and a valid CPU target after metrics arrive.

Step 4: Create the k6 Load Job

Create load.yaml. The first object stores a complete k6 script. The second runs it inside the cluster and writes the summary to the Job log. Pin the image by an approved version or digest in your own repository. The explicit tag below makes the tutorial reproducible at publication time.

apiVersion: v1
kind: ConfigMap
metadata:
  name: k6-script
  namespace: hpa-lab
data:
  test.js: |
    import http from 'k6/http';
    import { check } from 'k6';

    export const options = {
      scenarios: {
        steady: {
          executor: 'constant-arrival-rate',
          rate: 40,
          timeUnit: '1s',
          duration: '6m',
          preAllocatedVUs: 20,
          maxVUs: 100,
        },
      },
      thresholds: {
        http_req_failed: ['rate<0.01'],
        http_req_duration: ['p(95)<1000'],
        checks: ['rate>0.99'],
        dropped_iterations: ['count==0'],
      },
    };

    export default function () {
      const response = http.get('http://cpu-app.hpa-lab.svc.cluster.local/');
      check(response, {
        'status is 200': (r) => r.status === 200,
        'body is OK': (r) => r.body.includes('OK'),
      });
    }
---
apiVersion: batch/v1
kind: Job
metadata:
  name: hpa-load
  namespace: hpa-lab
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: hpa-load
    spec:
      restartPolicy: Never
      containers:
        - name: k6
          image: grafana/k6:1.2.3
          args: ["run", "/scripts/test.js"]
          resources:
            requests:
              cpu: 200m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 512Mi
          volumeMounts:
            - name: script
              mountPath: /scripts
      volumes:
        - name: script
          configMap:
            name: k6-script

Constant arrival rate is useful here because it expresses demand independently of response time. A closed-model test with a fixed number of virtual users can reduce achieved throughput when latency rises, hiding the load that should trigger scaling. dropped_iterations detects when k6 lacks enough virtual users to maintain the configured rate.

Apply only the ConfigMap first so you can validate the script without starting the experiment:

kubectl apply -f load.yaml --dry-run=server
kubectl -n hpa-lab apply -f load.yaml --prune=false
kubectl -n hpa-lab get job hpa-load

The second command starts the Job, so have the observer from Step 5 ready when running a formal test. Delete and recreate the Job for each run because a completed Job template is immutable.

Verification: kubectl -n hpa-lab logs job/hpa-load --tail=20 should show k6 scenario initialization or active progress. If the Job has not started, inspect kubectl -n hpa-lab describe job hpa-load and the pod events.

Step 5: Load Test Kubernetes Horizontal Pod Autoscaling

Open a second terminal immediately after starting the Job. Record a timestamped sample every 10 seconds:

while kubectl -n hpa-lab get job hpa-load -o jsonpath='{.status.completionTime}' | grep -q '^
#39;; do
  date -u +%FT%TZ
  kubectl -n hpa-lab get hpa cpu-app
  kubectl -n hpa-lab get deployment cpu-app \
    -o custom-columns=DESIRED:.spec.replicas,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas
  kubectl -n hpa-lab top pods -l app=cpu-app
  sleep 10
done | tee results/hpa-timeline.txt

In a third terminal, use a bounded assertion for scale-up:

deadline=$((SECONDS + 120))
scaled=false
while [ $SECONDS -lt $deadline ]; do
  desired=$(kubectl -n hpa-lab get hpa cpu-app -o jsonpath='{.status.desiredReplicas}')
  if [ "${desired:-0}" -gt 1 ]; then
    scaled=true
    echo "PASS: HPA desired replicas reached $desired"
    break
  fi
  sleep 5
done
[ "$scaled" = true ]

Then verify usable capacity catches requested capacity:

kubectl -n hpa-lab wait --for=condition=Available deployment/cpu-app --timeout=180s
kubectl -n hpa-lab get hpa,pods -o wide

The Available condition alone does not prove every desired replica is Ready, so compare Deployment spec.replicas with status.readyReplicas in the timeline. Also inspect pending pods. A desired count of eight with only three Ready pods indicates scheduling, image-pull, startup, quota, or node capacity trouble, not a successful autoscaling outcome.

Verification: the assertion must exit zero within 120 seconds, results/hpa-timeline.txt must show desired replicas greater than one, and Ready replicas must catch Desired during sustained load. If k6 finishes before the assertion passes, the trigger or observation window is inadequate.

Step 6: Validate Service Quality and Explain the Decision

Wait for k6 to finish, then capture its exit result and evidence:

kubectl -n hpa-lab wait --for=condition=complete job/hpa-load --timeout=10m
kubectl -n hpa-lab logs job/hpa-load | tee results/k6-summary.txt
kubectl -n hpa-lab describe hpa cpu-app > results/hpa-describe.txt
kubectl -n hpa-lab get events --sort-by=.metadata.creationTimestamp > results/events.txt
kubectl -n hpa-lab get hpa cpu-app -o yaml > results/hpa-final.yaml

A successful Job means the k6 process exited zero, including its threshold evaluation. Read the summary anyway. Confirm the achieved request rate, dropped_iterations, failure rate, check rate, and p95 duration. A scale-up can be functionally correct while requests fail during cold start. That result fails the overall hypothesis and points to minimum replicas, pod startup, readiness, capacity, or scaling policy.

Use these artifacts together:

Do not infer causality from two dashboard lines alone. Align UTC timestamps and identify the order: load began, CPU rose, HPA changed desired replicas, pods scheduled, readiness passed, and latency recovered. If your platform uses a service mesh, measure the proxy separately with the service mesh latency overhead testing tutorial. Sidecar CPU requests and startup behavior can change the autoscaling result.

Verification: the Job must have Complete=True, the log must show all four k6 thresholds passed, and HPA conditions must not contain persistent FailedGetResourceMetric or FailedComputeMetricsReplicas reasons. Archive all five result files under one run ID.

Step 7: Verify Scale-Down and Repeatability

After load stops, the HPA should retain recent recommendations during its 60-second scale-down stabilization window. Metrics collection and controller reconciliation add more time, so assert a generous but explicit bound of 5 minutes:

deadline=$((SECONDS + 300))
scaled_down=false
while [ $SECONDS -lt $deadline ]; do
  current=$(kubectl -n hpa-lab get deployment cpu-app -o jsonpath='{.spec.replicas}')
  printf '%s replicas=%s\n' "$(date -u +%FT%TZ)" "$current" | tee -a results/scale-down.txt
  if [ "$current" -eq 1 ]; then
    scaled_down=true
    break
  fi
  sleep 10
done
[ "$scaled_down" = true ]

Scale-down tests deserve their own assertion. Aggressive reduction can cause oscillation when traffic is bursty, while no reduction wastes capacity. Re-run the experiment at least three times with the same cluster state before treating one timing difference as a regression. Compare distributions of scale-up delay, pod-ready delay, error rate, and latency, not only one peak value.

Clean up after evidence collection:

kubectl delete namespace hpa-lab
kubectl get namespace hpa-lab --ignore-not-found

For soak testing, keep the namespace and vary traffic in controlled stages. Watch memory even when CPU is the scaling metric. A workload can pass a brief CPU test and degrade over hours. The long-running load test memory leak guide shows how to separate retained memory growth from expected cache behavior.

Verification: the assertion exits zero and results/scale-down.txt ends with replicas=1. Namespace cleanup should return no object. Before a second formal run, recreate from the same manifests and capture a fresh baseline.

Troubleshooting

HPA shows <unknown> CPU -> Confirm Metrics Server is healthy, wait for at least one metrics collection cycle, and verify every relevant container has a CPU request. Run kubectl describe hpa and inspect conditions before changing the load.

CPU stays below target -> Confirm k6 reaches the intended request rate and has no dropped iterations. Increase the arrival rate gradually or reduce the sample pod CPU request for the lab. Do not lower production requests solely to force scaling because requests also affect scheduling and capacity planning.

Desired replicas rise but pods stay Pending -> Inspect pod events, ResourceQuota, LimitRange, taints, affinities, and node allocatable resources. HPA has created demand, but the scheduler or cluster capacity cannot fulfill it.

Latency spikes even though scaling works -> Compare HPA decision time with image pull, scheduling, startup, and readiness durations. Increase minimum replicas, improve startup, pre-pull large images, or scale on an earlier leading indicator when appropriate.

k6 reports dropped iterations -> Increase preAllocatedVUs and maxVUs, give the load pod enough CPU, or distribute load across multiple generators. A constrained generator invalidates the intended arrival model.

Replicas do not scale down -> Wait beyond the configured stabilization window, confirm CPU returned below target, and inspect ScalingLimited and AbleToScale conditions. Recent high recommendations, another metric, or a policy can legitimately delay reduction.

Where To Go Next

Return to the complete cloud-native performance testing guide to add node capacity, observability, failure injection, and workload-specific test layers. Then extend this lab based on your protocol and runtime risks:

• Measure service mesh latency overhead under load before attributing all tail latency to the application.
• Follow the gRPC streaming service performance test tutorial when long-lived streams make request-rate models misleading.
• Run memory leak detection during long load tests to catch failures that a six-minute CPU experiment cannot expose.
• Add an external or custom metric HPA test only after validating metric freshness, adapter failure behavior, and fallback expectations.

In CI, run a short autoscaling smoke test on infrastructure changes and schedule longer capacity experiments in an isolated environment. Use the k6 performance testing guide to expand workload design, and the Kubernetes testing complete guide to cover scheduling, resilience, and deployment checks. Store manifests, immutable image identities, load scripts, raw outputs, and pass or fail assertions as one versioned test artifact.

Interview Questions and Answers

Q: Why are CPU requests required for a utilization-based HPA?

The resource HPA expresses average CPU utilization as usage divided by requested CPU. Without a usable request, the controller cannot calculate that pod's utilization normally. Requests also influence scheduling, so they must reflect a defensible capacity model rather than being tuned only to trigger scaling.

Q: What is the difference between desired replicas and Ready replicas?

Desired replicas represent the Deployment size requested after the HPA decision. Ready replicas have passed readiness checks and can normally receive service traffic. The delay between them measures scheduling and startup capacity, which directly affects user experience during scale-up.

Q: Why use a constant arrival rate for this test?

It holds offered demand near a target rate independently of response time. A fixed virtual-user model can send fewer iterations when the service slows, unintentionally reducing pressure. The generator must still have enough virtual users and CPU, which is why dropped iterations are a failure signal.

Q: How do you choose an HPA performance pass criterion?

Start from the service-level objective and expected burst. Define bounds for scale decision time, pod readiness, error rate, tail latency, and capacity recovery. Validate them across repeated runs and realistic cluster contention rather than copying a universal replica or time number.

Q: Why might an HPA request more pods without improving latency?

Pods may be Pending, pulling images, failing readiness, or waiting on the same downstream bottleneck. The scaling metric may also be unrelated to the true constraint. Correlate HPA status with scheduler events, readiness time, dependency telemetry, and load-generator results.

Q: How do stabilization windows affect tests?

A scale-down stabilization window retains recent recommendations to reduce flapping. Tests must wait beyond that window plus metrics and reconciliation delay before declaring a failure. Scale-up and scale-down should have separate timing assertions because their policies often differ.

Best Practices

• Do state the workload, trigger, timing bound, and service-level assertions before the run. Do not call any replica increase a pass.
• Do pin application and load-generator images for formal runs. Do not compare results across silently changing images.
• Do test in an isolated namespace and control neighboring load. Do not create noisy-neighbor risk in a shared production cluster.
• Do record desired, current, Ready, and available replicas. Do not rely on one HPA screenshot.
• Do validate the generator achieved the intended arrival rate. Do not interpret an underpowered generator as application capacity.
• Do examine HPA conditions and Kubernetes events. Do not guess why the controller acted.
• Do test both scaling directions and repeat the run. Do not generalize from one scale-up event.

Conclusion

To load test Kubernetes Horizontal Pod Autoscaling well, treat scaling as a timed system behavior, not a visual effect. Drive a declared workload, prove the metric crosses its target, assert desired and Ready replica changes, and keep latency and failures inside explicit limits.

Start with this CPU lab, preserve its evidence, and repeat it until the result is stable. Then replace the sample service and illustrative thresholds with your immutable build, production-shaped traffic, real service objectives, and known cluster constraints.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Java Semaphore Control Parallel Test Load Tutorial (2026)
• k6 Distributed Load Testing with the Kubernetes Operator (2026)
• k6 Test Server Sent Events Streams: Load Testing Tutorial
• Selenium Test Web Components Slots Tutorial: Test Web Component Slots with Selenium
• Test scenarios vs test cases (2026)
• TypeScript Decorators Test Metadata Tutorial: Create Test Metadata