Investigating High Deletion Ratio & Index Update Churn

Sequence	4 of 6
Context	Two indices showing 30–40% "deleted docs" ratio — unclear whether this meant real deletions or something else
Skills	Lucene/Elasticsearch internals, Indexing & Merge Stats APIs, shard recovery analysis

Background Concept

In Lucene, documents are never modified in place. An update to an existing document ID is internally a delete of the old version plus an insert of the new one. This means a high "deleted docs" ratio can come from two very different sources — explicit deletes, or ordinary update traffic — and the fix for each is completely different.

Step 1 — Separate explicit deletes from update-driven churn

curl -s "http://<endpoint>:9200/<index>/_stats/indexing?pretty"

Key fields: index_total (all index/update operations ever) and delete_total (explicit deletes only). Compare index_total against the index's current live document count:

churn_ratio = index_total / live_docs_count

A real example: index_total = 1,275,640,368 against 25,932,401 live docs — a ratio of ~49:1. Each live document had been re-indexed roughly 49 times on average, strong evidence the deletion ratio is driven by update churn, not cleanup of removed data.

Step 2 — Check whether merges are actually keeping up

curl -s "http://<endpoint>:9200/<index>/_stats/merge?pretty"

Key fields: total_time_in_millis and total_throttled_time_in_millis.

throttled_pct = total_throttled_time_in_millis / total_time_in_millis * 100

A real example showed ~74% of all merge time spent deliberately throttled — Elasticsearch itself detected I/O contention and chose to slow merges rather than let them compete too aggressively with indexing/search. This is the system protecting write throughput, not a misconfiguration.

Step 3 — Check the merge policy threshold governing cleanup

curl -s "http://<endpoint>:9200/<index>/_settings?include_defaults=true&pretty"   | grep -A2 "merge\|deletes_pct_allowed"

deletes_pct_allowed (Lucene default: 33.0) is the threshold at which a segment becomes a priority for merging. A deletion ratio hovering around 30–40% is roughly the expected steady-state for a high-churn index sitting near this threshold — not evidence of a broken system on its own.

Step 4 — Measure whether it's actively growing or old debt

# Sample 1
curl -s "http://<endpoint>:9200/_cat/indices/<index>?h=index,docs.count,docs.deleted"   > sample1.txt
sleep 300
# Sample 2
curl -s "http://<endpoint>:9200/_cat/indices/<index>?h=index,docs.count,docs.deleted"   > sample2.txt
diff sample1.txt sample2.txt

Step 5 — Investigate an unexpectedly zeroed stats counter

One index's indexing/merge stats came back as exactly zero across the board — not "quiet," but reset. Per-shard stat counters live in memory and reset whenever a shard gets a fresh physical copy on a node (restart, relocation, or recovery) — the document data persists, only the running counters reset.

curl -s "http://<endpoint>:9200/_cat/shards/<index>?v&h=index,shard,prirep,state,node"
curl -s "http://<endpoint>:9200/_cat/recovery/<index>?v&active_only=false"

The recovery history showed type: peer recoveries — a full network copy of shard data between nodes, not a fast local-disk restart — with byte counts matching the shard sizes almost exactly. Tracing the source_node column further revealed the primary shards had actually moved onto their current node from elsewhere, meaning the observed shard-concentration pattern was the result of that relocation event, not an original design choice.

Key Lesson

A high deletion ratio is a symptom with at least three independent possible drivers — explicit deletes, update churn, and I/O-throttled merges — each needing a different fix (application-level change, merge policy tuning, or a scheduled forcemerge). The Indexing Stats and Merge Stats APIs let you distinguish them with real numbers rather than assuming a cause. Zeroed-out stats counters are themselves a finding — they point to a relocation event worth investigating, not "no data available."