A company runs a Kubernetes cluster with 50 worker nodes, each hosting multiple microservices. They use Prometheus for metrics collection and Grafana for dashboards. Recently, the Prometheus server has been experiencing out-of-memory (OOM) kills during peak hours, causing gaps in metric collection. The cluster has a dedicated monitoring namespace. The team has already increased the Prometheus pod's memory limits to 8GB, but OOMs still occur. The metrics retention is set to 15 days. The cardinality of certain metrics (e.g., HTTP request labels with user IDs) is very high. The team needs to resolve the OOM issue without losing critical alerting capability for at least the last 7 days of data. Which action should they take first?
Recording rules reduce cardinality by aggregating metrics, lowering memory usage while preserving aggregated data for alerting.
Why this answer
Recording rules allow Prometheus to pre-aggregate high-cardinality metrics (e.g., HTTP request labels with user IDs) at a lower granularity, reducing the number of unique time series stored in memory. This directly addresses the OOM issue caused by cardinality explosion without discarding raw data entirely, preserving the ability to query aggregated metrics for alerting over the required 7-day window.
Exam trap
The trap here is confusing memory pressure (caused by cardinality) with storage pressure (caused by retention), leading candidates to incorrectly choose reducing retention (Option C) instead of addressing the root cause of high cardinality via recording rules.
How to eliminate wrong answers
Option B is wrong because dropping high-cardinality metrics entirely using relabel_configs would remove critical data needed for alerting and debugging, violating the requirement to retain alerting capability for at least 7 days. Option C is wrong because reducing retention to 7 days frees disk space, not memory; Prometheus OOMs are caused by in-memory time series cardinality, not storage volume. Option D is wrong because enabling vertical pod autoscaler would only adjust CPU/memory limits dynamically, but the fundamental issue is cardinality—more memory without reducing cardinality will still lead to OOM kills.