Detect Memory Leaks During Long-Running Load Tests

TL;DR

Run a constant, realistic workload long enough to cross several garbage-collection cycles. Detect a leak from sustained post-warm-up memory growth, failure to recover after traffic stops, and matching heap or profile evidence, then repeat the identical test after the fix.

Key Takeaways

• Establish an idle and warm-load baseline before judging memory growth.
• Hold a realistic arrival rate steady so elapsed time, not changing demand, explains the trend.
• Monitor process RSS, runtime heap, container working set, restarts, and request quality together.
• Compare equal steady-state windows and calculate a sustained slope instead of reacting to one peak.
• Stop traffic and observe recovery because retained memory after garbage collection is stronger leak evidence.
• Use two heap snapshots and allocation retainers to locate the objects keeping memory alive.
• Repeat the same run after the fix and require both stable memory and passing service thresholds.

To detect memory leaks long running load tests must keep demand controlled while you observe memory across many allocation and garbage-collection cycles. A rising memory chart alone is not proof. You need a stable workload, an early baseline, equal comparison windows, recovery observation, and runtime evidence showing which objects remain reachable.

This tutorial gives you that complete workflow. It fits into the broader Cloud-Native Performance Testing Complete Guide, where soak testing is one part of capacity, reliability, and observability engineering.

You will run k6 against an intentionally leaky Node.js service in Docker, sample process and container memory, calculate a post-warm-up slope, stop traffic, and compare heap snapshots. You can then apply the same method to Java, Go, .NET, Python, or a Kubernetes service by replacing the runtime-specific profiler.

What You Will Build

You will build a small memory-leak laboratory that can:

• start a bounded Node.js API with an optional controlled leak;
• apply constant arrival rate with a configurable k6 soak test;
• reject an invalid run when errors, latency, or dropped iterations exceed limits;
• capture RSS, heap used, external memory, container memory, CPU, and restart evidence;
• compare early and late steady-state windows with a simple slope calculation;
• capture two heap snapshots and find the retaining path;
• disable the leak and prove the fix with the same workload.

The sample deliberately stores request data in a process-level array. Never expose diagnostic endpoints or Node inspector ports publicly. Use an isolated test environment and remove the fixture when you finish.

Prerequisites

Install Docker Engine 26 or newer with Docker Compose v2, Node.js 22 LTS or newer, and k6 1.x. The application runs in Docker, while k6 and the analysis script run on your host. A POSIX shell with curl is assumed.

Verify the tools:

docker version
docker compose version
node --version
k6 version
curl --version

Give Docker at least 2 GB of memory. The sample container has a 384 MB limit and aborts before consuming the host. For a real system, obtain authorization, define a maximum duration and request rate, estimate test-data growth, and agree on abort conditions for memory, errors, latency, restarts, and dependency health.

Use at least 30 minutes for the tutorial run. A production soak commonly needs hours, but duration must come from the suspected accumulation rate and operational cycle, not a generic rule. If a leak appears only after token rotation, compaction, or a daily job, the test must cross that boundary.

Step 1: Create a Controlled Leaky Service

Create a directory and configure package.json:

mkdir -p memory-leak-lab/results
cd memory-leak-lab
npm init -y
npm install express@5
npm pkg set type=module scripts.start='node server.js'

Create server.js:

import express from 'express';
import crypto from 'node:crypto';
import v8 from 'node:v8';

const app = express();
const retained = [];
const leakEnabled = process.env.LEAK_ENABLED === 'true';
const startedAt = Date.now();

app.get('/work', (req, res) => {
  const record = {
    id: crypto.randomUUID(),
    createdAt: Date.now(),
    payload: crypto.randomBytes(4096).toString('hex'),
  };
  if (leakEnabled) retained.push(record);
  res.json({ ok: true, id: record.id });
});

app.get('/health', (req, res) => res.json({ ok: true }));

app.get('/debug/memory', (req, res) => {
  const m = process.memoryUsage();
  res.json({
    timestamp: new Date().toISOString(),
    uptimeSeconds: Math.round((Date.now() - startedAt) / 1000),
    leakEnabled,
    retainedRecords: retained.length,
    rssBytes: m.rss,
    heapUsedBytes: m.heapUsed,
    heapTotalBytes: m.heapTotal,
    externalBytes: m.external,
    arrayBuffersBytes: m.arrayBuffers,
  });
});

app.post('/debug/snapshot', (req, res) => {
  const filename = v8.writeHeapSnapshot();
  res.json({ filename });
});

app.listen(3000, '0.0.0.0', () => {
  console.log(`listening on 3000 leakEnabled=${leakEnabled}`);
});

The leak is explicit: each request adds an object containing about 8 KB of hexadecimal text to an array that remains reachable for the life of the process. The health and debug routes do not add records.

Verification: run LEAK_ENABLED=true npm start and call curl -s http://localhost:3000/work five times. Then run curl -s http://localhost:3000/debug/memory. The JSON must show leakEnabled as true and retainedRecords as 5. Stop the process before continuing.

Step 2: Put the Service Under a Safe Memory Limit

Create Dockerfile:

FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY server.js ./
USER node
EXPOSE 3000
CMD ["node", "server.js"]

Create compose.yaml:

services:
  api:
    build: .
    environment:
      LEAK_ENABLED: ${LEAK_ENABLED:-true}
    ports:
      - "3000:3000"
    mem_limit: 384m
    cpus: 1.0
    init: true
    volumes:
      - ./results:/app/results

Build and start it:

LEAK_ENABLED=true docker compose up --build -d
docker compose ps
curl --fail --silent http://localhost:3000/health

The cgroup limit bounds the experiment. It is not a target for the application. Abort well below it because snapshot creation needs temporary memory and an out-of-memory kill can lose evidence. In a shared environment, set service-specific resource limits and protect dependencies too.

Capture configuration with every run:

docker compose config > results/compose-config.yaml
docker image inspect memory-leak-lab-api --format '{{.Id}}' > results/image-id.txt

If Compose generates a different image name, obtain it from docker compose images. Recording the digest, code revision, runtime flags, host capacity, and limits makes the comparison defensible.

Verification: docker compose ps must show the API as running. docker inspect $(docker compose ps -q api) --format '{{.HostConfig.Memory}}' must print 402653184, which is 384 MiB in bytes.

Step 3: Establish Idle and Warm-Load Baselines

A service often grows during startup because modules load, pools initialize, code compiles, and caches fill. Do not call that a leak. First, record five idle samples at ten-second intervals:

: > results/idle.jsonl
for i in 1 2 3 4 5; do
  curl --fail --silent http://localhost:3000/debug/memory >> results/idle.jsonl
  printf '\n' >> results/idle.jsonl
  sleep 10
done

Warm the request path with a bounded smoke run:

k6 run --vus 2 --duration 30s - <<'EOF'
import http from 'k6/http';
import { check, sleep } from 'k6';
export default function () {
  const r = http.get('http://localhost:3000/work');
  check(r, { 'status 200': (x) => x.status === 200 });
  sleep(0.2);
}
EOF

Then record five warm samples using the same loop and write them to results/warm.jsonl. Keep traffic and monitoring routes separate. A monitoring request that allocates heavily can distort a small test.

Compare metrics, not just one number. RSS reflects resident pages, heapUsed reflects the JavaScript heap, external and arrayBuffers help expose native buffers, and container memory shows cgroup pressure. A Java service would add used heap after full GC, metaspace, direct buffers, thread count, and GC pause data.

Verification: both JSONL files must contain five valid JSON objects. Run wc -l results/idle.jsonl results/warm.jsonl, then node -e "JSON.parse(require('fs').readFileSync('results/warm.jsonl','utf8').trim().split('\n').at(-1)); console.log('valid')". Expect 5 for each file and valid.

Step 4: Run a Long-Running k6 Load Test

Create soak.js:

import http from 'k6/http';
import { check } from 'k6';

const baseUrl = __ENV.BASE_URL || 'http://localhost:3000';

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 20),
      timeUnit: '1s',
      duration: __ENV.DURATION || '30m',
      preAllocatedVUs: 20,
      maxVUs: 60,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<500'],
    checks: ['rate>0.99'],
    dropped_iterations: ['count==0'],
  },
};

export default function () {
  const response = http.get(`${baseUrl}/work`, { tags: { operation: 'work' } });
  check(response, {
    'work returns 200': (r) => r.status === 200,
    'business response is valid': (r) => r.json('ok') === true,
  });
}

Use constant arrival rate because a slowdown must not silently reduce offered demand. Start a 30-minute run and save its machine-readable summary:

DURATION=30m RATE=20 k6 run --summary-export=results/k6-summary.json soak.js \
  | tee results/k6-console.txt

For a real service, derive operation mix, arrival rate, data, pacing, and duration from production evidence. Keep the rate below known capacity because this experiment tests accumulation at a supported load, not overload behavior. If the service autos-scales, either hold replicas constant to study one process or collect per-pod identity and memory. The Kubernetes HPA load testing tutorial explains how scaling changes the experiment.

Do not wait for an OOM to declare success. Set a stop condition, such as RSS above 320 MiB, repeated container restarts, error rate above 5 percent, or dependency distress. Stop k6 with Ctrl+C, preserve evidence, and record why the run ended.

Verification: k6 must sustain the requested iteration rate with zero dropped iterations, greater than 99 percent checks, less than 1 percent failed requests, and p95 below 500 ms. A threshold failure invalidates the simple memory comparison until you explain the changed workload or failure mode.

Step 5: Sample Memory During the Same Steady Workload

While Step 4 runs, open a second terminal in the lab directory. Create monitor.sh:

#!/usr/bin/env sh
set -eu
output=results/memory.jsonl
: > "$output"
while curl --fail --silent http://localhost:3000/health >/dev/null; do
  app=$(curl --fail --silent http://localhost:3000/debug/memory)
  container=$(docker stats --no-stream --format '{{json .}}' "$(docker compose ps -q api)")
  printf '{"app":%s,"container":%s}\n' "$app" "$container" >> "$output"
  sleep 15
done

Run it and leave it active through the load phase and for ten minutes after k6 stops:

chmod +x monitor.sh
./monitor.sh

In a third terminal, record restarts and inspect pressure periodically:

watch -n 30 'docker compose ps; docker stats --no-stream'

The monitor combines application metrics and Docker statistics in one timestamped line. Docker's formatted fields vary slightly by platform, so treat application byte values as the analysis source and the container record as supporting evidence. In Kubernetes, collect container_memory_working_set_bytes, container_memory_rss, pod restarts, limits, and process-runtime metrics with pod labels. Aggregate graphs can hide one leaking pod behind newly started replicas.

Watch the generator too. A growing k6 process, excessive response logging, or unbounded test data can be mistaken for target leakage. Keep response bodies out of custom arrays and metric tags. If the request path includes a mesh proxy, use the service mesh latency overhead tutorial to separate application and sidecar resource behavior.

Verification: after two minutes, wc -l results/memory.jsonl should show at least eight samples. Validate the latest line with tail -1 results/memory.jsonl | node -e "let s='';process.stdin.on('data',d=>s+=d).on('end',()=>{const x=JSON.parse(s);console.log(x.app.rssBytes,x.app.retainedRecords)})". Both values must be positive.

Step 6: Calculate Growth and Test Recovery

Create analyze.js:

import fs from 'node:fs';

const rows = fs.readFileSync('results/memory.jsonl', 'utf8')
  .trim().split('\n').map(JSON.parse).map((x) => x.app);
if (rows.length < 12) throw new Error('Need at least 12 samples');

const windowSize = Math.max(5, Math.floor(rows.length * 0.2));
const early = rows.slice(windowSize, windowSize * 2);
const late = rows.slice(-windowSize);
const mean = (xs, key) => xs.reduce((n, x) => n + x[key], 0) / xs.length;
const minutes = (late.at(-1).uptimeSeconds - early[0].uptimeSeconds) / 60;
const rssDelta = mean(late, 'rssBytes') - mean(early, 'rssBytes');
const heapDelta = mean(late, 'heapUsedBytes') - mean(early, 'heapUsedBytes');

console.log(JSON.stringify({
  samples: rows.length, windowSize, minutes,
  rssDeltaMiB: rssDelta / 1048576,
  heapDeltaMiB: heapDelta / 1048576,
  rssSlopeMiBPerMinute: rssDelta / 1048576 / minutes,
  heapSlopeMiBPerMinute: heapDelta / 1048576 / minutes,
  retainedRecordsStart: early[0].retainedRecords,
  retainedRecordsEnd: late.at(-1).retainedRecords,
}, null, 2));

Run node analyze.js | tee results/analysis.json. The script skips the first window as warm-up, averages windows to reduce sawtooth noise, and reports a rate. Do not copy its slope threshold blindly into another system. Establish normal variance from repeated leak-free runs, then set a project limit that detects harmful accumulation before the memory budget is exhausted.

After k6 ends, keep monitoring for ten minutes at idle. For managed test environments, trigger a normal full collection only if the runtime exposes an approved mechanism. Do not make forced GC part of production behavior. If RSS stays high but heap used falls, investigate allocator retention, native memory, buffers, memory mapping, or fragmentation before blaming live application objects.

Compare equal windows at equal load. A rising cache that reaches a documented bound is different from a leak. Cache entries should stabilize, evict, or expire, and the plateau should fit the memory budget. A leak continues to raise the retained floor for the same steady demand.

Verification: the intentionally leaky run should show a positive heap and RSS slope plus a large increase in retainedRecords. The final idle samples should not return near the warm baseline because the global array still owns every record. This is detection evidence, not yet the root cause.

Step 7: Confirm the Retaining Path and Prove the Fix

Capture a snapshot early in a fresh run, then another after ten minutes of load:

curl --fail --silent -X POST http://localhost:3000/debug/snapshot
k6 run --duration 10m --vus 10 soak.js
curl --fail --silent -X POST http://localhost:3000/debug/snapshot
docker compose exec api sh -c 'ls -lh *.heapsnapshot'
docker compose cp api:/app/Heap.00000001.001.heapsnapshot results/early.heapsnapshot
docker compose cp api:/app/Heap.00000002.002.heapsnapshot results/late.heapsnapshot

Actual snapshot names include sequence values and can differ. Use the filenames returned by the endpoint, then substitute them in docker compose cp. Snapshot creation pauses the process and may temporarily require substantial memory. Capture outside the measured latency window, use a disposable replica when possible, and stop if memory is near its limit.

Open Chrome DevTools, select Memory, load both snapshots, and use Comparison view. Sort by positive retained-size delta. Inspect the retaining path for the growing strings and records. You should find the module-level retained array keeping request objects reachable. A dominator or retaining path is stronger evidence than a large constructor count because it explains ownership.

Now prove the fix without changing the workload. Restart with leakage disabled, then start ./monitor.sh in one terminal. In a second terminal, run:

docker compose down
rm -f results/memory.jsonl
LEAK_ENABLED=false docker compose up -d
./monitor.sh

DURATION=30m RATE=20 k6 run --summary-export=results/k6-fixed.json soak.js
sleep 600
node analyze.js | tee results/analysis-fixed.json

The ten-minute sleep preserves the idle recovery window before analysis. Stop the monitor after analyze.js finishes. In a real fix, remove the unintended reference, bound the cache, close the resource, or correct listener cleanup. Do not hide growth by raising the memory limit or scheduling restarts.

Verification: retainedRecords must remain zero, k6 thresholds must pass, and the late-window heap slope must fall inside the normal band established by repeated leak-free runs. Repeat at least three comparable runs. Preserve the before and after summaries, snapshots, configuration, image IDs, and analysis under unique run IDs.

Detect Memory Leaks Long Running Load Tests Can Expose

Different leak classes need different evidence. Use this decision table before choosing a profiler:

For Java, compare old-generation occupancy after full GC and use Java Flight Recorder or a heap dump. For Go, inspect pprof heap profiles and in-use space. For .NET, use dotnet-counters, GC heap metrics, and dotnet-gcdump or a dump. For Python, use tracemalloc, object counts, and an allocation profiler. The scientific method stays the same even when tooling changes.

Time is an experimental variable. Hold request mix, data cardinality, build, replicas, limits, flags, dependencies, and observability configuration constant. If memory grows only when unique IDs grow, repeat with a bounded recycled dataset to distinguish per-request leakage from expected data-dependent caching.

Troubleshooting

Problem: RSS rises but heap used stays flat -> Inspect external buffers, native libraries, allocator arenas, thread stacks, memory maps, and page cache. A managed heap snapshot cannot explain every resident byte.

Problem: Memory drops sharply, then rises again -> Normal garbage collection can create a sawtooth. Compare post-collection floors or averaged equal windows. A ratcheting floor is more important than the peak.

Problem: k6 shows dropped iterations -> Increase preallocated VUs only after checking generator CPU, memory, network, and response time. Until delivered demand matches requested demand, the leak rate comparison is invalid.

Problem: The container restarts before a snapshot -> Lower the load or duration, add an earlier evidence trigger, and leave headroom for profiling. Record OOM and restart evidence instead of rerunning blindly.

Problem: A second snapshot freezes the API -> Snapshot capture is stop-the-world and memory-intensive. Use a disposable replica, capture outside the scored interval, or switch to lower-overhead sampling and allocation profiling.

Problem: Memory stabilizes only after several hours -> Check cache bounds, TTLs, scheduled jobs, token refresh, data growth, and runtime adaptive behavior. Extend the observation window only with approved cost and abort limits.

Where To Go Next

Move the sampler into your normal metrics system, label every series with a low-cardinality run ID, and build a dashboard that aligns demand, latency, errors, garbage collection, heap, RSS, restarts, and dependency pools. Use the Cloud-Native Performance Testing Complete Guide to place this soak test in a wider performance strategy.

If replica changes affect the result, follow the Kubernetes Horizontal Pod Autoscaling load test. For proxy-heavy services, measure service mesh latency overhead. For persistent connections, adapt the same accumulation workflow with the gRPC streaming performance testing tutorial. You can also strengthen the setup with a practical test data strategy so unique records do not create misleading growth.

Interview Questions and Answers

Q: How do you distinguish a memory leak from normal cache growth? A cache has a documented bound, observable entry count, eviction or expiry behavior, and a stable plateau within budget. A leak shows continuing retained growth under the same steady workload and does not recover as designed.

Q: Why is RSS alone insufficient? RSS includes managed heap, native allocations, stacks, code, mapped pages, and allocator behavior. Correlate it with runtime heap, container working set, object counts, garbage collection, and profiles.

Q: Why use constant arrival rate for a soak test? It keeps offered demand independent of target slowdown. That makes elapsed time and accumulation easier to interpret, provided dropped iterations remain zero and the generator stays healthy.

Q: What makes a memory slope trustworthy? Exclude warm-up, compare equal steady-state windows, average several samples, preserve workload and environment, and repeat the experiment. Define the acceptable band from known-good variance and the service memory budget.

Q: When should you capture a heap snapshot? Capture at least two controlled points after the symptom is reproducible and while enough memory headroom remains. Because snapshots can pause and enlarge a process, use an isolated environment or disposable replica.

Q: What proves that a fix worked? The same build context except for the fix must pass the same correctness, load, and latency thresholds, show stable post-warm-up memory across repeated runs, recover as expected, and remove the growing retaining path.

Best Practices

• Define steady demand, duration, memory budget, recovery window, and abort conditions before execution.
• Warm the runtime and major request paths before measuring the leak slope.
• Collect application, runtime, container, operating-system, and generator evidence on synchronized clocks.
• Preserve per-instance identity because aggregate memory can hide an aging pod.
• Track business correctness and achieved load so low traffic does not look like stability.
• Keep profiling outside the scored latency interval and document profiler overhead.
• Repeat before and after tests with the same data model, limits, replicas, and dependencies.
• Fix ownership or lifecycle, not the symptom created by a small memory limit.

Common Mistakes

• Declaring a leak from one rising dashboard line during warm-up.
• Comparing a low-load early window with a high-load late window.
• Watching only heap for a native or buffer leak.
• Ignoring k6 memory, CPU, dropped iterations, or network saturation.
• Capturing a dump only after the process is already near OOM.
• Treating an autoscaling aggregate as if it described every pod.
• Changing workload, environment, and code together between comparison runs.
• Accepting periodic restarts or larger limits as the permanent fix.

Detect Memory Leaks Long Running Load Tests Conclusion

To detect memory leaks long running load tests must create controlled time under realistic demand and preserve evidence from several layers. Start after warm-up, validate delivered traffic, compare equal windows, observe idle recovery, and use a runtime profiler to identify the retaining owner.

Run the same experiment after the code change. The job is complete only when request quality still passes, the memory trend stays inside a known-good band, recovery behaves as designed, and repeated runs no longer show the suspect retaining path.

Interview Questions and Answers

Frequently Asked Questions

Related Guides

• AI code review for Playwright tests (2026)
• Designing a load model (2026)
• Detect Missing Accessible Names with Playwright
• Detecting flaky tests with machine learning (2026)
• Generating API tests from OpenAPI with AI (2026)
• Generating Playwright tests from a user story (2026)