Elasticsearch Production Swap Configuration

Sequence	5 of 6
Context	A new cluster's nodes had a 1GB swap partition (a stable legacy cluster had none); a Zabbix "low free swap" alert triggered the investigation
Skills	Linux kernel memory management, Elasticsearch official guidance, phased production change planning

Step 1 — Verify the reference environment directly, don't assume

free -g
cat /etc/fstab
grep bootstrap.memory_lock /etc/elasticsearch/elasticsearch.yml
systemctl show elasticsearch | grep LimitMEMLOCK

Real findings on the stable legacy cluster: Swap: 0 0 0, no swap entry in fstab at all (never provisioned, not merely disabled), bootstrap.memory_lock: false, and LimitMEMLOCK=65536 (the unmodified ~64KB OS default — confirming memory locking had never actually been configured either).

Conclusion: the legacy cluster's stability came entirely from having no swap device to begin with — not from any additional memory-locking safeguard. This simplified the plan for the new cluster: match the proven configuration exactly, rather than combine multiple theoretical best practices that weren't actually validated by real uptime evidence.

Step 2 — Confirm official guidance, don't rely on memory

Elastic's own documentation ("Disable swapping") states plainly that swapping "can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster," and lists three options in order of preference:

1. Disable swap entirely (most complete)
2. vm.swappiness = 1 — not 0, since kernel 3.5-rc1+ can trigger the OOM killer instead of light swapping at exactly 0
3. bootstrap.memory_lock: true — locks the JVM heap into RAM specifically

Step 3 — Disable swap (no restart required)

# Check headroom first — swapoff moves everything currently swapped back into RAM immediately
free -h
swapon --show

sudo swapoff -a
sudo cp /etc/fstab /etc/fstab.bak-20260801
sudo sed -i '/swap/s/^/#/' /etc/fstab

free -h   # confirm Swap: 0 total, 0 used

Step 4 — Roll out safely across a cluster

1. Start with whichever node is already alerting — known baseline, lowest-risk validation.
2. Confirm the alert clears and the node stays stable for a day before continuing.
3. Roll to the remaining nodes one at a time, off-peak.

Optional Hardening (only if pursued deliberately)

# Grant the OS-level permission BEFORE enabling the setting, or the service won't start
sudo systemctl edit elasticsearch
# add: [Service]
#      LimitMEMLOCK=infinity
sudo systemctl daemon-reload

echo "bootstrap.memory_lock: true" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl restart elasticsearch

# Verify the lock actually took effect — don't just trust the config file
curl -s "http://<node>:9200/_nodes?filter_path=**.mlockall&pretty"
# should show "mlockall": true

Production bootstrap checks mean Elasticsearch will refuse to start with memory_lock: true if the OS-level permission wasn't granted first — this order matters.

Key Lesson

When proposing a production change, verify what the "proven" reference environment actually does, rather than recommend a theoretical best-practice combination. Real evidence (a system that's been stable for years) is stronger justification than documentation alone, and often points to a simpler plan than "do everything the docs mention."