Building an Automated Elasticsearch Diagnostic Script

Sequence	2 of 6
Goal	Turn the manual flush-latency investigation into a reusable, safe, version-adaptive tool
Skills	Bash, jq, Elasticsearch REST API, defensive scripting, read-only tooling design

Design Principles

Principle	Why
Strictly read-only	Every request is a plain HTTP GET. No write operations anywhere in the script — safe to run on production, including during an active incident.
No hardcoded topology	An early version hardcoded node names and assumed one node was always a "healthy baseline." That assumption broke the first time a different node became the problem. The script now discovers cluster topology and classifies problem/OK nodes fresh from live data every run.
Version-adaptive	Elasticsearch's API changed the indexing thread pool name (`bulk`/`index` → `write`) at 6.3, among other differences across major versions. The script probes the live cluster's actual version and API shape rather than assuming one.
No baked-in conclusions	Findings are computed fresh from each run's data using configurable thresholds — never a fixed narrative about a specific past incident that would go stale.

Step 1 — Cluster Topology Auto-Discovery

Query node attributes/roles directly rather than trusting a static list:

curl -s "http://<endpoint>:9200/_nodes?filter_path=nodes.*.name,nodes.*.roles,nodes.*.attributes"

Parsed with jq to sort nodes into data/master/client roles automatically — works across both the older attributes-based role format and the newer roles array.

Step 2 — Live Version Detection

curl -s "http://<endpoint>:9200/"

Parses the version.number field from the root endpoint response (grep/sed, no jq dependency for this step, since it must work even before confirming jq is installed). The detected major version then selects the correct thread-pool name and other version-specific API paths for the rest of the run.

Step 3 — Probing an Ambiguous API Endpoint Instead of Guessing

A disputed claim came up during this work: does the merge-stats endpoint use the singular metric name (merge) or the plural (merges)? Rather than trust either claim, the script tests both live and uses whichever one actually returns 200:

CODE_PLURAL=000
if [ "" = "200" ]; then
  MERGE_METRIC="merges"
else
  MERGE_METRIC="merge"
fi

This turned out to matter in practice: on a real ES 6.6.2 cluster, only the singular merge endpoint worked. Live-probing avoided shipping a script with a wrong hardcoded assumption.

Step 4 — Live Classification, Not Fixed Assumptions

Two flush-stat samples, a configurable interval apart, computed as a delta rather than a cumulative average (see the manual-troubleshooting page for why this matters):

# jq computes, per node:
#   delta_flushes = sample2.flush.total - sample1.flush.total
#   delta_time    = sample2.flush.total_time_in_millis - sample1.flush.total_time_in_millis
#   current_avg_ms = delta_time / delta_flushes
# Then classifies each node PROBLEM / OK against a configurable threshold (default 200ms,
# matching the Zabbix trigger).

Step 5 — Deeper Diagnostics, Auto-Targeted by Evidence

Rather than deep-diving every index in a large cluster, the script targets whichever indices are already flagged by deletion ratio or shard-concentration checks — computed fresh each run:

# Pseudocode
DEEP_DIVE_INDICES = (indices with deletion_ratio >= threshold)
                   UNION (indices where one node holds >= threshold% of primaries)
for each index in DEEP_DIVE_INDICES:
    collect _stats/indexing, _stats/merge, _settings?include_defaults=true, _cat/recovery

Step 6 — A Single Generated Report, Not Just Raw Files

All collected JSON/text is processed by jq into one readable report with computed, threshold-based findings (not a fixed narrative) — cluster overview, per-node resource table, flush latency (cumulative + live delta), shard placement, segment concentration, deletion/churn analysis, and a generic (non-prescriptive) recommended-next-steps checklist.

A Real Bug Caught by Testing Against Mock Data

jq operator-precedence trap: an expression like X + "GB" as | BODY does not parse as (X + "GB") as — jq's as binds tighter than +, so it actually parses as X + ("GB" as | BODY). This silently corrupted output (a numeric prefix leaking onto the next field) until caught by building a mock API server and testing the exact jq expression in isolation before shipping.

# Wrong —  only ever equals "GB"
(size_calc) + "GB" as  | [.index, ]

# Correct — parenthesize the whole expression before "as"
((size_calc) + "GB") as  | [.index, ]

Result

A single script that: discovers topology, detects ES version live, adapts its own API calls accordingly, classifies problem nodes from real-time data, and produces one report — safe to run repeatedly on production with zero write operations.