Autoscale Selenium Grid on Kubernetes Step by Step

TL;DR

Deploy one Selenium Hub and a scalable Chrome node Deployment, install Metrics Server, and attach an autoscaling/v2 HPA to the node Deployment. Generate parallel sessions, watch replicas grow, and confirm the Grid returns to its minimum after the load ends.

Key Takeaways

• Run the Selenium Hub as a stable control-plane Deployment and browser nodes as a separately scalable Deployment.
• Install Metrics Server before creating an HPA that uses CPU or memory utilization.
• Set CPU requests on every browser node because HPA utilization cannot be calculated without them.
• Keep one browser session per pod first, then tune concurrency only after measuring isolation and cost.
• Verify registration, session execution, scale-out, and scale-in as separate acceptance checks.
• Use CPU-based HPA as a dependable baseline, then consider queue-aware scaling for bursty test traffic.

The selenium grid kubernetes autoscaling step by step process is straightforward when you separate the stable Grid control plane from disposable browser capacity. You will deploy a Selenium Hub, register Chrome node pods, and use a Kubernetes HorizontalPodAutoscaler (HPA) to add nodes when their average CPU utilization rises.

This tutorial produces a working baseline that you can paste into a local or managed cluster. Read the complete Selenium Grid cloud scaling guide first if you need help choosing Kubernetes, Docker, or a hosted grid. Here, the goal is narrower: build Kubernetes autoscaling and prove it under parallel WebDriver load.

The example pins a known Selenium 4 image instead of using latest. Treat the tag as a controlled dependency: test a newer Selenium release in staging, update both Hub and node images together, and keep the last working tag available for rollback.

What You Will Build

By the end, you will have:

• A dedicated selenium-grid namespace.
• One Selenium Hub exposed inside the cluster and through a local port-forward.
• Chrome node pods that register through the Grid event bus.
• Resource requests and limits suitable for CPU-based autoscaling.
• An HPA that maintains 1 to 8 Chrome node replicas.
• A Python load test that creates parallel remote sessions.
• Repeatable checks for registration, scale-out, test completion, and scale-in.

The topology deliberately keeps state simple. The Hub handles session routing and the node pods run browsers. Kubernetes replaces failed pods, while HPA changes only the Chrome node replica count.

Prerequisites

Use Kubernetes 1.30 or newer and a matching kubectl minor version when possible. The manifests use the stable autoscaling/v2 API. You also need Helm 3, Python 3.11 or newer, and Docker if you create a local cluster with kind or minikube.

Check your tools:

kubectl version --client
helm version
python3 --version

You need permission to create namespaces, Deployments, Services, HPAs, and the Metrics Server components. For a local example, create a minikube cluster with enough memory for several browsers:

minikube start --cpus=6 --memory=12288
kubectl get nodes

A managed cluster works too. Ensure its worker nodes can pull public images and have spare CPU and memory. Eight Chrome pods at the limits in this tutorial could request substantial capacity, so lower maxReplicas on a small laptop.

Create a clean working directory outside your application repository if desired:

mkdir -p selenium-grid-k8s
cd selenium-grid-k8s

Verification: kubectl get nodes must show at least one node with STATUS equal to Ready. Do not continue while the cluster is still provisioning.

Step 1: Create the Grid Namespace

Put all tutorial resources in one namespace so you can inspect, quota, and remove them together. Create namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: selenium-grid
  labels:
    app.kubernetes.io/part-of: selenium-grid

Apply it and set it as the namespace for the current kubectl context:

kubectl apply -f namespace.yaml
kubectl config set-context --current --namespace=selenium-grid

The second command prevents accidental creation of later resources in default. Explicit -n selenium-grid flags still work and are preferable in CI scripts because they do not depend on workstation context.

Add a modest namespace limit only after you understand your cluster policy. Browser containers are memory-heavy, and an overly strict quota can leave HPA-created pods in Pending. Production teams usually pair a maximum replica count with namespace quotas and cluster autoscaling.

Verification: Run kubectl get namespace selenium-grid. The status must be Active. Then run kubectl config view --minify --output 'jsonpath={..namespace}'; it should print selenium-grid.

Step 2: Deploy the Selenium Hub

Create hub.yaml. The Service exposes WebDriver on port 4444 and the event bus publish and subscribe ports on 4442 and 4443. Nodes use the Service DNS name selenium-hub.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-hub
  labels:
    app: selenium-hub
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selenium-hub
  template:
    metadata:
      labels:
        app: selenium-hub
    spec:
      containers:
        - name: hub
          image: selenium/hub:4.29.0
          imagePullPolicy: IfNotPresent
          ports:
            - { name: webdriver, containerPort: 4444 }
            - { name: events-pub, containerPort: 4442 }
            - { name: events-sub, containerPort: 4443 }
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          readinessProbe:
            tcpSocket:
              port: webdriver
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: webdriver
            initialDelaySeconds: 20
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: selenium-hub
spec:
  selector:
    app: selenium-hub
  ports:
    - { name: webdriver, port: 4444, targetPort: 4444 }
    - { name: events-pub, port: 4442, targetPort: 4442 }
    - { name: events-sub, port: 4443, targetPort: 4443 }

Apply it and wait for readiness:

kubectl apply -f hub.yaml
kubectl rollout status deployment/selenium-hub --timeout=120s

A TCP probe checks that the server is listening. Your later /status request provides the stronger functional check. Keep one Hub replica for this tutorial because multiplying Grid control-plane components requires a deliberate distributed Grid configuration, not a simple replica change.

Verification: Start kubectl port-forward service/selenium-hub 4444:4444 in another terminal. Run curl -fsS http://localhost:4444/status; the JSON response should contain "ready": true. Open http://localhost:4444/ui to see the Grid console.

Step 3: Deploy Scalable Chrome Nodes

Create chrome-nodes.yaml. Every pod registers with the Hub event bus and advertises one session slot. One session per pod gives predictable isolation and makes replica count closely match browser capacity.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-node-chrome
  labels:
    app: selenium-node-chrome
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selenium-node-chrome
  template:
    metadata:
      labels:
        app: selenium-node-chrome
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: chrome
          image: selenium/node-chrome:4.29.0
          imagePullPolicy: IfNotPresent
          env:
            - { name: SE_EVENT_BUS_HOST, value: selenium-hub }
            - { name: SE_EVENT_BUS_PUBLISH_PORT, value: "4442" }
            - { name: SE_EVENT_BUS_SUBSCRIBE_PORT, value: "4443" }
            - { name: SE_NODE_MAX_SESSIONS, value: "1" }
            - { name: SE_NODE_OVERRIDE_MAX_SESSIONS, value: "true" }
            - { name: SE_NODE_SESSION_TIMEOUT, value: "300" }
            - name: SE_NODE_GRID_URL
              value: http://selenium-hub:4444
          ports:
            - { name: node, containerPort: 5555 }
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 1500m
              memory: 2Gi
          readinessProbe:
            tcpSocket:
              port: node
            initialDelaySeconds: 8
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: node
            initialDelaySeconds: 30
            periodSeconds: 10

Apply the Deployment:

kubectl apply -f chrome-nodes.yaml
kubectl rollout status deployment/selenium-node-chrome --timeout=180s

CPU requests are essential. HPA computes utilization as usage divided by requested CPU. If a selected pod lacks a CPU request, Kubernetes cannot calculate its CPU utilization correctly for this policy. Limits protect neighboring workloads, but limits that are too low can throttle Chrome and inflate test duration.

Verification: Run kubectl get pods -l app=selenium-node-chrome. One pod should be 1/1 Running. With the port-forward still active, curl -fsS http://localhost:4444/status should show one registered Chrome slot in the returned Grid data. The Grid UI should also display one Chrome node.

Step 4: Install and Verify Metrics Server

HPA reads resource metrics through Kubernetes Metrics API. Many managed clusters provide Metrics Server, but local clusters may not. First test the API:

kubectl top pods

If that command returns CPU and memory values, continue to Step 5. On minikube, enable the bundled addon:

minikube addons enable metrics-server
kubectl rollout status deployment/metrics-server -n kube-system --timeout=180s

For another cluster, install the current upstream release through its official components manifest, subject to your platform policy:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl rollout status deployment/metrics-server -n kube-system --timeout=180s

Production environments should pin and review third-party manifests rather than applying a moving URL. The command is convenient for a disposable tutorial cluster. Some local clusters use certificates that Metrics Server cannot validate; prefer the platform-specific addon before changing TLS arguments.

Metrics arrive on a sampling interval, so a newly started pod may display <unknown> briefly. HPA also uses stabilization behavior, which means replicas do not jump on every short spike. Those delays are healthy control-loop behavior.

Verification: Repeat kubectl top pods -n selenium-grid until both Hub and Chrome pod values appear. Also run kubectl get apiservice v1beta1.metrics.k8s.io; its AVAILABLE column must be True.

Step 5: Configure selenium grid kubernetes autoscaling step by step with HPA

Create chrome-hpa.yaml. This policy targets 60 percent average CPU utilization, permits fast scale-up, and delays aggressive scale-down so short pauses between tests do not churn pods.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: selenium-node-chrome
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: selenium-node-chrome
  minReplicas: 1
  maxReplicas: 8
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 4
          periodSeconds: 30
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Apply it:

kubectl apply -f chrome-hpa.yaml
kubectl get hpa selenium-node-chrome

The HPA formula compares current average utilization with the target and chooses a desired replica count, bounded by 1 and 8. CPU is a practical baseline but not a direct measure of waiting Selenium sessions. A quiet browser can leave queued work while CPU remains below target, particularly when every existing node is already occupied but the pages under test are light.

Start with CPU, measure behavior, then add a queue-aware signal if session wait time is the real bottleneck.

Verification: kubectl describe hpa selenium-node-chrome should show a valid CPU target rather than <unknown>. The AbleToScale, ScalingActive, and ScalingLimited conditions should not report configuration errors.

Step 6: Generate Parallel WebDriver Load

Keep the Hub port-forward running. In the working directory, create requirements.txt:

selenium==4.29.0

Install it in a virtual environment:

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

Create load_test.py. It launches 16 tasks. Each task requests a remote Chrome session, loads a data URL that performs repeated JavaScript work, checks the title, waits briefly, and always quits the driver.

from concurrent.futures import ThreadPoolExecutor, as_completed
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

GRID_URL = "http://localhost:4444"
WORKERS = 16

def run_session(index: int) -> str:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1280,720")
    driver = webdriver.Remote(command_executor=GRID_URL, options=options)
    try:
        html = """<title>Grid Load</title><script>
        const end = Date.now() + 15000;
        while (Date.now() < end) { Math.sqrt(Math.random() * 1000000); }
        </script><h1>Autoscaling check</h1>"""
        driver.get("data:text/html;charset=utf-8," + html)
        assert driver.title == "Grid Load"
        time.sleep(15)
        return f"session {index}: passed"
    finally:
        driver.quit()

with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    futures = [pool.submit(run_session, i) for i in range(WORKERS)]
    for future in as_completed(futures):
        print(future.result())

Run the script:

python load_test.py

In another terminal, watch the control loop and pods:

kubectl get hpa,pods -w

Requests beyond current Grid capacity wait in the new-session queue. As active Chrome pods consume CPU, HPA requests more replicas. Newly ready nodes register and accept queued sessions. The exact peak depends on cluster resources and observed utilization, so do not assert a fixed replica number.

Verification: The watcher should show the HPA desired replica count rise above 1, followed by additional Chrome pods moving from Pending or ContainerCreating to Running. The Python process should eventually print 16 passed lines without leaking sessions.

Step 7: Verify selenium grid kubernetes autoscaling step by step recovery

After the load stops, HPA waits through the configured 300-second stabilization window before reducing replicas. Watch its events and current metrics:

kubectl describe hpa selenium-node-chrome
kubectl get deployment selenium-node-chrome -w

Do not manually change the Deployment replica count while HPA owns it. A manual kubectl scale can be overwritten on the next reconciliation. Change minReplicas, disable the HPA, or use a deployment process that clearly transfers ownership.

Once the Deployment returns to one replica, run a small health test to prove the remaining capacity still works:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
with webdriver.Remote(
    command_executor="http://localhost:4444", options=options
) as driver:
    driver.get("data:text/html,<title>Recovered</title>")
    assert driver.title == "Recovered"
    print("post-scale test: passed")

Save it as health_test.py, then run python health_test.py. This final check catches a common false success: replicas scaled, but nodes failed to re-register or the one remaining pod became unhealthy.

For production, also test pod termination during a live session. Kubernetes sends SIGTERM and observes the grace period, but application-aware draining needs deliberate validation. Avoid scaling down so quickly that active sessions are destroyed. Long stabilization and one session per pod reduce risk, but they do not replace workload-specific shutdown tests.

Verification: kubectl get deployment selenium-node-chrome should eventually show 1/1 ready. python health_test.py must print post-scale test: passed, and the Grid status must remain ready.

Step 8: Add Operational Guardrails

Autoscaling is only useful when the cluster can schedule the replicas. Compare requested resources with node capacity:

kubectl top nodes
kubectl describe nodes
kubectl get events --sort-by=.lastTimestamp

Add a PodDisruptionBudget for the Hub so voluntary disruptions do not remove it without respecting availability. Create hub-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: selenium-hub
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: selenium-hub

Apply it with kubectl apply -f hub-pdb.yaml. A PDB does not protect against node failure, container crashes, or involuntary eviction. It only constrains voluntary disruptions such as a node drain.

Use labels and taints if browsers share a cluster with latency-sensitive services. Browser pods often have distinctive CPU, memory, and shared-memory behavior. Dedicated node pools make capacity and costs easier to reason about. Connect Kubernetes cluster autoscaling as a separate layer: HPA creates pod demand, and the infrastructure autoscaler adds worker nodes when those pods cannot be scheduled.

For deeper visibility, add traces and metrics using the Selenium Grid OpenTelemetry monitoring setup. Observability should distinguish queue delay, pod startup, browser startup, test runtime, and application response time. A single end-to-end duration cannot tell you which layer caused the slowdown.

Verification: kubectl get pdb selenium-hub should show ALLOWED DISRUPTIONS as 0 when only one healthy Hub exists. kubectl get events should contain no repeated FailedScheduling, OOM kill, or probe failure events.

Troubleshooting

HPA shows CPU as <unknown> -> Confirm kubectl top pods works, inspect v1beta1.metrics.k8s.io, and verify the Chrome container has a CPU request. Wait for at least one metrics sampling interval after a pod starts.

Chrome pods run but do not appear in Grid -> Check kubectl logs deployment/selenium-node-chrome. Verify SE_EVENT_BUS_HOST=selenium-hub, Service ports 4442 and 4443, namespace DNS, and matching Hub and node versions. Test DNS from a temporary diagnostic pod if policy permits.

New replicas stay Pending -> Run kubectl describe pod <pod-name> and read Events. Insufficient CPU or memory means you must lower requests, lower maxReplicas, add worker capacity, or enable the provider's cluster autoscaler. Quota, affinity, taints, and image pull errors can cause the same symptom.

Parallel sessions queue but HPA does not scale -> CPU might be below the target even though all slots are occupied. Lower the CPU target only if measurements justify it, increase the minimum before scheduled CI peaks, or implement queue-aware autoscaling from an exported Grid metric. Do not treat CPU as a perfect proxy for demand.

Pods scale down during active sessions -> Increase the scale-down stabilization window, reduce the scale-down rate, and validate graceful node draining. Inspect termination events and test logs. Never assume a PDB on the Hub protects browser sessions in node pods.

Chrome crashes with shared-memory or OOM errors -> Inspect kubectl describe pod and previous container logs. Increase memory only after measurement, reduce per-node session concurrency, and consider an appropriately sized memory-backed /dev/shm volume according to the browser image guidance and your cluster security policy.

Common Mistakes and Best Practices

• Do pin Hub and node images to the same tested Selenium version. Do not deploy latest in a reproducible CI platform.
• Do set CPU and memory requests from observed workloads. Do not copy limits blindly from a laptop into production.
• Do begin with one session per pod for isolation. Increase concurrency only after profiling browser memory, CPU, and test stability.
• Do cap replicas according to worker capacity and budget. An HPA maximum is a safety boundary, not a capacity promise.
• Do monitor the new-session queue and pod startup time. CPU alone can miss idle-looking but fully occupied nodes.
• Do call quit() in a finally block. Leaked sessions consume slots and make autoscaling behavior misleading.
• Do separate test failures from infrastructure failures. Capture the Grid session ID, pod name, browser logs, and timestamps.
• Do test scale-in with live and completed sessions. A successful scale-out test covers only half the lifecycle.

If failed-session diagnosis is slow, add artifacts with the Selenium Grid video recording for failed sessions guide. Video is most useful when it is correlated with WebDriver logs and retained selectively, not recorded indefinitely without ownership or expiry.

Interview Questions and Answers

Q: Why scale Selenium nodes instead of the Hub Deployment?

Browser nodes provide session capacity and consume most workload resources. The Hub is the routing and coordination layer. Scaling its replica count without configuring a distributed Grid does not safely create more browser slots.

Q: Why are CPU requests required for a utilization-based HPA?

Kubernetes calculates CPU utilization relative to each selected pod's requested CPU. Missing requests prevent a meaningful percentage calculation. Requests also help the scheduler reserve enough node capacity for browser pods.

Q: What is the weakness of CPU-based Selenium Grid autoscaling?

CPU is an indirect demand signal. All browser slots may be occupied while CPU stays modest, leaving sessions queued without triggering enough scale-out. Queue depth or wait time is usually a better advanced signal.

Q: How do HPA and cluster autoscaling work together?

HPA changes the desired number of browser pods. If worker nodes lack room, those pods remain Pending. A cluster autoscaler can then add infrastructure capacity, after which the scheduler places the pods.

Q: How do you reduce session loss during scale-in?

Use a conservative stabilization window and rate limit, keep sessions bounded, and test termination behavior. Add application-aware draining where supported by your deployment design. A PodDisruptionBudget alone does not guarantee active node sessions finish.

Q: Why use one browser session per pod?

It improves isolation and makes one replica equal one slot. Failures and resource spikes affect fewer tests. Higher concurrency can reduce overhead but requires careful profiling and makes capacity less transparent.

Where To Go Next

You now have a working CPU-based baseline. Use the Selenium Grid cloud scaling complete guide to compare this design with Docker and managed services, then harden the option that matches your workload.

Next, follow the dynamic Selenium Grid nodes with Docker tutorial if Kubernetes is more operational machinery than your team needs. Add the Selenium Grid OpenTelemetry monitoring setup before tuning thresholds, because queue delay and pod startup measurements should drive the policy. Finally, use video recording for failed Selenium sessions when you need visual evidence from transient browser pods.

Conclusion

A reliable selenium grid kubernetes autoscaling step by step implementation separates the Hub from Chrome capacity, gives every node measurable resource requests, and attaches an autoscaling/v2 HPA only to the node Deployment. The job is not finished when replicas increase. Prove registration, parallel execution, controlled scale-in, and a successful test after recovery.

CPU scaling is a sound first control loop, not the final answer for every suite. Measure session queue time, browser startup, pod scheduling, and test duration. When CPU does not track waiting work, graduate to queue-aware metrics while keeping the same verification discipline.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• Docker for Selenium Grid: Step by Step (2026)
• Monitor Selenium Grid with OpenTelemetry: Selenium Grid OpenTelemetry Monitoring Setup
• Running tests on a Selenium Grid in Kubernetes (2026)
• Allure report in CI: Step by Step (2026)
• Azure DevOps test pipelines: Step by Step (2026)
• CircleCI for test automation: Step by Step (2026)