JVM Heap Sizing — OOM Incident, Root Cause & Safe Rollout Procedure

Sequence	6 of 6
Context	The day after removing swap (see previous page), all three master nodes hit an out-of-memory kill within hours of each other
Skills	JVM memory management, kernel OOM analysis, shard allocation control, zero-downtime production changes

Part A — Root Cause Investigation

Step 1 — Confirm the kernel's own account of what happened

grep -i -E "oom|out of memory|killed process" /var/log/messages

Real log evidence:

oom-kill: constraint=CONSTRAINT_NONE, task_memcg=/system.slice/elasticsearch.service, task=java
Out of memory: Killed process (java) total-vm:5196996kB, anon-rss:2733980kB

constraint=CONSTRAINT_NONE confirms a genuine whole-system RAM exhaustion — not a cgroup/systemd memory limit being hit.

Step 2 — Check configured heap size vs. live heap size

grep -E '^-Xm[sx]' /etc/elasticsearch/jvm.options
grep -R "Xms\|Xmx" /etc/elasticsearch/jvm.options.d/
ps aux | grep elasticsearch | grep -oE '\-Xmx[0-9]+[a-zA-Z]?'
free -m

Finding: no explicit heap configured anywhere. Elasticsearch auto-sizes to roughly 50% of total RAM when unset — on a small 3.66GB master node, that computed to -Xmx2193m, leaving almost no headroom for the OS.

Step 3 — Understand why swap removal exposed this now

Before swap was removed, a transient memory spike had somewhere to spill into. Removing swap (a separate, correct decision — see the swap configuration page) removed that safety net, so the very next spike had nowhere to go, and the kernel OOM-killer had no choice but to act. The auto-sizing gap had existed the whole time; swap had simply been masking it.

Step 4 — Compare against the proven reference cluster's value

# On the stable legacy cluster, for reference:
-Xms1g
-Xmx1g

Deliberately conservative on a similarly small node (3.79GB total) — matching this value directly was the fix, not picking an arbitrary number.

Part B — Immediate Fix on the Masters

sudo cp /etc/elasticsearch/jvm.options /etc/elasticsearch/jvm.options.bak-20260801
echo "-Xms1g" | sudo tee -a /etc/elasticsearch/jvm.options
echo "-Xmx1g" | sudo tee -a /etc/elasticsearch/jvm.options
sudo systemctl restart elasticsearch

# Verify
ps aux | grep elasticsearch | grep -oE '\-Xmx[0-9]+[a-zA-Z]?'
systemctl status elasticsearch | grep Active

Applied one master at a time. With 3 master-eligible nodes, quorum requires 2 of 3 — restarting them sequentially, verifying each before moving to the next, avoided ever dropping below quorum unnecessarily.

Part C — Extending the Fix: Data & Client Nodes

Checking data and client nodes revealed the same underlying gap (no explicit heap anywhere) — but auto-sizing happened to land safely there purely because of much larger absolute RAM (32GB / 15.7GB vs. the masters' 3.66GB). Same root cause, different outcome, purely due to node size. Fixed proactively rather than waiting for a second incident.

Data nodes — full drain-and-restore cycle (they hold shard data)

# 1. Check current shard load first
curl -s -u elastic --cacert  "/_cat/allocation?v" | grep <node-name>

# 2. "Lock" the node — exclude it from shard allocation
curl -X PUT "/_cluster/settings" -u elastic --cacert    -H 'Content-Type: application/json' -d '{
    "transient": { "cluster.routing.allocation.exclude._name": "<node-name>" }
  }'

# 3. Wait for shards to reach zero — confirm directly, don't infer from cluster health alone
curl -s -u elastic --cacert  "/_cat/allocation?v" | grep <node-name>
curl -s -u elastic --cacert  "/_cat/recovery?v&active_only=true"   # progress detail

# 4. Set heap (~50% of node RAM, matching the legacy cluster's convention) and restart
echo "-Xms16g" | sudo tee -a /etc/elasticsearch/jvm.options
echo "-Xmx16g" | sudo tee -a /etc/elasticsearch/jvm.options
sudo systemctl restart elasticsearch

# 5. Verify live heap
ps aux | grep elasticsearch | grep -oE '\-Xmx[0-9]+[a-zA-Z]?'

# 6. "Unlock" — remove the exclusion
curl -X PUT "/_cluster/settings" -u elastic --cacert    -H 'Content-Type: application/json' -d '{
    "transient": { "cluster.routing.allocation.exclude._name": null }
  }'

# 7. Wait for cluster green + zero relocating shards before the next node
curl -s -u elastic --cacert  "/_cat/health?v"

Client nodes — simpler (no shard data), but never both at once

echo "-Xms8g" | sudo tee -a /etc/elasticsearch/jvm.options
echo "-Xmx8g" | sudo tee -a /etc/elasticsearch/jvm.options
sudo systemctl restart elasticsearch
ps aux | grep elasticsearch | grep -oE '\-Xmx[0-9]+[a-zA-Z]?'
# confirm cluster healthy before touching the second client node

Part D — Full-Cluster Final Verification

# Every node's role and heap, one table
curl -s -u elastic --cacert    "/_cat/nodes?v&h=name,node.role,heap.max,ram.percent,disk.avail"

# Confirm nothing left locked
curl -s -u elastic --cacert  "/_cluster/settings?pretty"
# "transient" should be {}

Real Operational Issues Encountered (and how they were handled)

What happened	Resolution
A drain-verification command returned empty output	Turned out to be an authentication failure (wrong password at the prompt), not a genuine zero — `curl -s` silently swallows error responses when piped through `grep`. Re-ran without the filter to see the raw `401` response and confirm.
Cluster stayed `green` during active shard relocation	Expected — Elasticsearch keeps the original shard copy serving until the new copy finishes, so relocation alone doesn't degrade health status.
One data node showed far less disk usage than its peers, with a nonzero `shards.undesired` count	Elasticsearch's own balancer reporting it intends to rebalance data onto that node — self-correcting in the background, not a fault requiring manual action.

Key Lesson

Fixing an active incident on the node type that broke is necessary but not sufficient — the same root cause (missing heap sizing) existed cluster-wide and only hadn't caused a second incident yet due to node size. Auditing every related node type after an incident, not just the one that failed, closes the gap before it becomes the next 2 a.m. page.