Elasticsearch Memory Metrics — Why "RAM Used%" Can Be Misleading

Sequence	3 of 6
Context	A diagnostic report flagged all data nodes at 99% RAM used [CRIT]; a manager cross-checked against Zabbix and found ~35.7% used instead
Skills	Linux memory internals, Elasticsearch API behavior, cross-source data verification

The Discrepancy

Source	Reported value	Formula
Elasticsearch `os.mem.used_percent`	~99%	`(MemTotal − MemFree) / MemTotal`
Zabbix "Available memory %"	~35.7% available (~64% used)	`MemAvailable / MemTotal`

Step 1 — Get the raw numbers directly from the OS

free -h
cat /proc/meminfo | grep -Ei "^(MemTotal|MemFree|MemAvailable|Buffers|Cached):"

Example real output:

MemTotal:       32006520 kB
MemFree:          219128 kB
MemAvailable:   11428244 kB
Cached:         12757968 kB

Step 2 — Reproduce both percentages by hand

# Elasticsearch's calculation
(32006520 - 219128) / 32006520 * 100 = 99.3%

# Zabbix's calculation
11428244 / 32006520 * 100 = 35.7%

Both are correct arithmetic on the same real data — they are simply answering two different questions.

Step 3 — Understand why they differ: page cache

Linux fills unused RAM with filesystem cache (visible as Cached above) to speed up repeated disk reads — in this case, that means Elasticsearch's own Lucene segment files. Elasticsearch's memory API counts that cache as "used." The Linux kernel's own MemAvailable figure correctly subtracts out the reclaimable portion of that cache, since it can be instantly freed the moment an application actually needs it.

This is documented, intended Elasticsearch behavior — confirmed via Elastic's own issue tracker (elastic/elasticsearch#27157): "it's intended that used memory includes the pages allocated to the filesystem cache."

Step 4 — Confirm which number actually matters

MemAvailable (the Zabbix-style number) is the one that reflects real memory pressure — it accounts for what the kernel could actually hand to a new process right now. A node showing 99% by Elasticsearch's metric, with a stable, healthy MemAvailable, is not in danger — it's simply well-cached, which is a sign of a healthy, active node, not a struggling one.

Step 5 — Watch for a second, independent cause too

On one cluster investigated this way, the gap between the two calculations was larger than page caching alone explained. That turned out to warrant checking whether Elasticsearch was seeing a constrained memory pool (e.g. a cgroup/systemd memory limit smaller than the host's real total) — a reminder not to assume the first plausible explanation accounts for the entire gap. Confirmed via:

curl -s "http://<node>:9200/_nodes/<node>/stats/os?filter_path=nodes.*.name,nodes.*.os.mem" | jq .
# Compare .os.mem.total_in_bytes against the real MemTotal from /proc/meminfo

Fixing a Diagnostic Tool's Reporting Based on This Finding

Once confirmed, a reporting tool that had been flagging RAM as [CRIT] based on Elasticsearch's own metric was corrected: the misleading auto-flag was removed, replaced with the raw number plus an explanatory note, and any root-cause logic that had assumed high "used%" meant "no free page cache" (backwards reasoning) was removed as well.

Key Lesson

When two monitoring sources disagree, don't assume one is broken — check whether they're measuring different things. Reproduce both numbers by hand from raw source data before trusting either interpretation, and be ready to explain the exact formula behind each.