Manual Troubleshooting: Elasticsearch Flush Latency

Step 1 — Confirm the alert is real, not a monitoring artifact

curl -s "http://<endpoint>:9200/_cluster/health?pretty"
curl -s "http://<endpoint>:9200/_cat/nodes?v&h=name,ip,heap.percent,ram.percent,cpu,load,disk.avail,uptime"

One line per node — heap, RAM, CPU, disk, uptime — the fastest first-pass view of which node looks unhealthy.

Step 2 — Take two samples to see CURRENT latency, not a diluted average

# Sample 1
curl -s "http://<endpoint>:9200/_nodes/stats/indices?filter_path=nodes.*.name,nodes.*.indices.flush"   > flush1.json

sleep 60

# Sample 2
curl -s "http://<endpoint>:9200/_nodes/stats/indices?filter_path=nodes.*.name,nodes.*.indices.flush"   > flush2.json

# Compare flush.total and flush.total_time_in_millis between the two files, per node
diff <(jq . flush1.json) <(jq . flush2.json)

Step 3 — Check the fast-to-rule-out causes first: JVM/GC

curl -s "http://<endpoint>:9200/_nodes/<node>/stats/jvm?pretty"

Look at heap_used_percent and gc.collectors.old.collection_count / collection_time_in_millis. Stop-the-world GC pauses freeze every operation on the node, including flushes — a node with high old-gen GC activity is a strong early lead.

Step 4 — Check for a merge storm

curl -s "http://<endpoint>:9200/_nodes/<node>/stats/indices/merges?pretty"
curl -s "http://<endpoint>:9200/_cat/segments?v&bytes=mb"

Healthy segment count is roughly 1–5 per shard after merging. Numbers well above that (20–40+) mean Lucene's merge process is falling behind, and every flush has to compete with ongoing merges for the same disk I/O.

Step 5 — Check disk itself

curl -s "http://<endpoint>:9200/_nodes/<node>/stats/fs?pretty"

Free space, and — where the OS exposes it — io_stats for actual read/write operation counts. A node doing dramatically more disk reads than its peers may be missing page-cache hits.

Step 6 — Check translog size

curl -s "http://<endpoint>:9200/_nodes/<node>/stats/indices/translog?pretty"

A large uncommitted translog means the next flush has more data to persist and will naturally take longer — worth comparing against the default 512MB flush threshold to see how close a node is to triggering its next (likely large, likely slow) flush.

Step 7 — Check thread pool pressure

curl -s "http://<endpoint>:9200/_cat/thread_pool/bulk,index,flush?v&h=node_name,name,active,queue,rejected,completed"

Any nonzero rejected count means the node is overwhelmed at the queue level, not just I/O-bound — a higher-priority finding than a slow disk.

Step 8 — Confirm with hot threads

curl -s "http://<endpoint>:9200/_nodes/<node>/hot_threads?threads=5&ignore_idle_threads=true"

If flush/merge threads are visibly blocked inside a disk-write stack, that directly confirms I/O contention as the mechanism, not just a correlated symptom. An empty hot-threads result is also informative — it means the bottleneck is I/O wait, not CPU, since the hot-threads API only surfaces CPU-burning threads.

Step 9 — Check shard placement for a "hot index"

curl -s "http://<endpoint>:9200/_cat/shards?v&bytes=gb"
curl -s "http://<endpoint>:9200/_cat/indices?v&s=store.size:desc&bytes=gb"

Cross-reference which node holds the busiest index's primary shards. A node absorbing a disproportionate share of one index's writes will show elevated flush activity for a structural reason, not a hardware fault.

Key Lesson

Multiple nodes alerting simultaneously points toward a shared cause (storage backend, a hot index whose shards span those nodes, cluster-wide indexing spike) rather than independent per-node hardware faults happening to occur at the same time. Check for that pattern before investigating each node in isolation.