Elasticsearch Production Troubleshooting — Work Index
This is the entry point into a series of Elasticsearch production troubleshooting pages, covering a real
end-to-end incident investigation: from a recurring Zabbix flush-latency alert, through building an
automated diagnostic tool, to a swap/JVM misconfiguration that caused a production master-node outage,
to a full safe remediation across an 8-node cluster. Each linked page is self-contained and technical;
this page is the map connecting them in the order the work actually happened.
Skills Demonstrated Across This Work
Elasticsearch REST API Bash scripting jq / JSON processing Linux memory internals JVM heap tuning Shard allocation & cluster topology Zabbix monitoring Production incident response Root cause analysis Technical documentation
Work Sequence
| # | Topic | What it covers |
|---|---|---|
| 1 | Manual Troubleshooting: Elasticsearch Flush Latency | Starting point — a recurring Zabbix "flush latency too high" alert, and the raw curl-based investigation methodology used to trace it: cluster health, node stats, thread pools, hot threads, merge/translog state. |
| 2 | Building an Automated Elasticsearch Diagnostic Script | Turning the manual investigation into a reusable, version-adaptive Bash/jq tool: auto-discovers cluster topology, detects the ES version live, classifies nodes as problem/OK from real-time data, and generates a single readable incident report. |
| 3 | Elasticsearch Memory Metrics — Why "RAM Used%" Can Be Misleading | A real discrepancy between what Elasticsearch reports as memory-used and what the OS/monitoring tool reports as memory-available — and the Linux memory-internals reasoning that explains both numbers are correct, just measuring different things. |
| 4 | Investigating High Deletion Ratio & Index Update Churn | Using the Indexing and Merge Stats APIs to distinguish "documents are being deleted" from "documents are being updated constantly" — and using recovery history to explain why an index's stats counters had unexpectedly reset. |
| 5 | Elasticsearch Production Swap Configuration | Comparing swap configuration between a stable legacy cluster and a new one, referencing official Elasticsearch guidance, and producing a phased, low-risk plan to remove swap in production. |
| 6 | JVM Heap Sizing — OOM Incident, Root Cause & Safe Rollout Procedure | What happened after the swap change: a real master-node out-of-memory crash, the root-cause trail back to a missing explicit heap setting, and the safe, one-node-at-a-time procedure used to fix it across the entire cluster without data loss. |
Result
By the end of this sequence: a reusable diagnostic tool exists for future incidents; a documented, Elastic-guidance-backed standard exists for swap and JVM heap configuration across the cluster; a real production incident was root-caused, fixed, and verified with zero data loss; and every step is written up as a standalone technical reference for future use.