Hard Difficulty Questions

Using a deployment name label.

Why it fails: A deployment name label carries a bounded set of values per workload, so it contributes low cardinality rather than high. It is tempting because deployment labels are standard practice for slicing metrics by application version, which is correct when you need per-deployment breakdowns, but that granularity stays small.

Using a cluster region label.

Why it fails: A cluster region label has only a handful of possible values, so it cannot drive high cardinality. It is tempting because region labels are legitimate for aggregating metrics across geographically distributed clusters, which is the right choice when you need per-region dashboards rather than per-object series.

Using a service name label.

Why it fails: A service name label is bounded by the number of services, typically dozens, so it produces low cardinality. It is tempting because service labels are the standard way to attribute metrics to a logical service, which is correct when you need per-service request rates rather than per-pod or per-container series.

Delete the alerting rule

Why it fails: Deleting the rule stops the alert permanently, not just during the maintenance window, and removes detection afterwards. Silences are time-bounded and label-matched, so they suit temporary suppression; deleting a rule is chosen only when the alert should never fire again.

Modify the routing tree

Why it fails: Routing trees decide which receiver gets a firing alert after it fires; they cannot stop the alert firing during maintenance. Silences, created via the API, UI or amtool with a matcher and time window, suppress notifications, so routing-tree edits suit re-directing alerts, not maintenance windows.

Add a 'skip' label to the rule

Why it fails: Alertmanager has no 'skip' label semantics; labels select routes and receivers, they do not suppress evaluation. Silence resources, matched by label, are the mechanism for maintenance windows, so a skip label would be chosen only by someone assuming arbitrary labels alter firing behaviour.

>>

Why it fails: Label matchers require =, !=, =~ or !~ between the label name and value; '>>' is not a PromQL operator at all. It tempts as a plausible-looking comparison token, and would be right only if PromQL supported greater-than matching, which it does not.

==

Why it fails: '==' is not a valid matcher operator; equality matching uses a single '=' (for example job="api"). It tempts because '==' is standard equality syntax in most programming languages, so it would be correct if PromQL reused that convention rather than its own matcher grammar.

It caches the inner query results in memory for 30 minutes.

Why it fails: Prometheus evaluates the subquery independently at each 1-minute step across the 30-minute window, executing the inner expression repeatedly; it does not memoise results. Caching feels plausible because recording rules persist computed series, but that is a deliberate pre-computation, not subquery behaviour.

It increases the global scrape interval to 1 minute for the next 30 minutes.

Why it fails: The scrape interval is configured in prometheus.yml and applies to targets; a subquery's resolution and range affect only query evaluation steps, never scrape scheduling. The confusion arises because the 1m in the subquery resembles a scrape interval, but it defines the evaluation step for the inner expression.

It converts the range vector into a scalar through statistical interpolation.

Why it fails: Prometheus evaluates subqueries by executing the inner expression at each step of the outer range, producing a range vector of instant vectors; it never interpolates scalars. Statistical interpolation is tempting because scalar conversion exists via scalar(), but that function applies to single instant vectors, not subquery syntax.

Subqueries execute faster than standard instant queries because they bypass the storage engine.

Why it fails: Subqueries are evaluated by the PromQL engine over the storage engine's returned samples, adding an inner range-vector evaluation layer rather than bypassing storage. It is tempting because subqueries nest a range selector inside an instant query, which superficially suggests a shortcut around normal sample retrieval.

Subqueries can only be executed against range vectors, never instant vectors.

Why it fails: Subqueries return instant vectors, which is precisely why they exist: wrapping a range vector expression so its result can feed functions expecting instant vectors. It is tempting because the inner expression must be a range vector, so the syntax visually resembles range-vector-only evaluation.

The resolution parameter in a subquery is mandatory.

Why it fails: The resolution parameter is optional; when omitted, Prometheus defaults it to the global evaluation_interval. It is tempting because resolution appears prominently in subquery syntax and controls step alignment, so it reads as a required argument rather than an overridable default.

In the Alertmanager 'receivers' configuration

Why it fails: Receivers define the individual integrations, but the severity-based branching itself belongs in the route tree, where child routes match labels such as severity to select a receiver. It is tempting because receivers name the Slack and email destinations, yet they hold no matching logic.

In the Grafana 'Alerting' UI

Why it fails: Grafana's Alerting UI manages Grafana-managed alerts and their notification policies, not Alertmanager routing for Prometheus alert rules. It is tempting because Grafana can display and forward alerts, but the severity split must live in Alertmanager's route configuration.

In the Prometheus 'alerting' rule file

Why it fails: Alerting rule files only define alert expressions and labels; they contain no receiver or routing configuration. It is tempting because severity labels originate there, but the actual decision of which destination receives each severity is made downstream in Alertmanager's route tree.

To keep the alert firing after resolution

Why it fails: Alert resolution is driven by the alert's status from the source, not by routing configuration. The 'continue' field only decides whether evaluation proceeds to subsequent sibling routes after a match, which matters when one alert must trigger several receivers instead of stopping at the first.

To ignore duplicate alerts

Why it fails: Duplicate suppression is performed by Alertmanager's deduplication and grouping logic, not by the 'continue' field. The field governs whether matching continues to further sibling routes after one matches, which is useful when an alert must also reach additional receivers beyond the first matching route.

To restart the Alertmanager process

Why it fails: The 'continue' field controls route matching behaviour, determining whether an alert proceeds to subsequent sibling routes after a match rather than stopping at the first. It has no effect on the Alertmanager process lifecycle; restarting is handled by the service manager or orchestration platform, not by route configuration.

The ?= operator performs fuzzy string matching.

Why it fails: PromQL defines only =, !=, =~ and !~ as label matchers; no ?= operator exists, so this syntax is invalid. It is tempting because fuzzy or substring matching feels useful for loosely related label values, but regex matching via =~ already covers that need.

The =~ operator is used exclusively for numeric comparisons.

Why it fails: =~ performs regex matching against label values, not numeric comparison; numbers are compared with arithmetic and comparison operators outside selectors. It is tempting because regex syntax superficially resembles comparison symbols, but label matchers filter series, they do not evaluate numeric thresholds.

The == operator is valid inside label selectors for filtering.

Why it fails: Label selectors accept only =, !=, =~ and !~; == is not valid matcher syntax and causes a parse error. It is tempting because == is the equality operator in PromQL expressions and many programming languages, but inside curly braces the single = form is required.

It reverses the vector matching direction to evaluate from right to left.

Why it fails: Vector matching always evaluates from left to right; group_right controls cardinality expansion.

It drops all labels from the left-hand side vector.

Why it fails: group_right keeps all labels from the right side and matching labels from the left.

It ensures that the output contains only time series found exclusively on the right side.

Why it fails: It still performs an inner or outer join depending on modifiers, but specifically populates based on the right-hand side cardinality.

histogram_quantile(http_request_duration_seconds_bucket[5m], 0.99)

Why it fails: The quantile parameter must be the first argument, and the range vector must be wrapped in a rate or increase function.

quantile_over_time(0.99, http_request_duration_seconds_bucket[5m])

Why it fails: quantile_over_time calculates the quantile of raw values over time for a single series, not aggregated histogram buckets.

rate(histogram_quantile(0.99, http_request_duration_seconds_bucket[5m]))

Why it fails: histogram_quantile returns an instant vector of gauges; wrapping it in rate() is incorrect because gauges cannot be passed to rate().

Data retention policy

Why it fails: Retention policy governs how long stored samples are kept before deletion; it does not govern how many distinct time series are created, so it cannot explain the memory spike. It is tempting because retention settings do affect disk usage, and would be the answer if old data were accumulating beyond the configured retention window.

Metric saturation

Why it fails: Metric saturation describes a resource or signal reaching capacity, not the cardinality explosion caused by a unique user ID label; each distinct value creates a new time series. It is tempting because memory did spike, but the violated concept is high cardinality, which would be avoided by excluding unbounded values from labels.

Service discovery limits

Why it fails: Service discovery limits concern how Prometheus finds scrape targets, not how many label combinations each target emits; the unique user ID multiplies time series independently of target count. It is tempting because discovery does affect scrape volume, and would be relevant if targets were being registered or deregistered incorrectly.

The query speed will increase.

Why it fails: Adding a high-cardinality label such as user_id multiplies time series, increasing memory, storage and query cost, so speed degrades rather than improves. It is tempting because labels enable filtering, which feels like it should speed up targeted queries, but cardinality growth overwhelms that benefit.

Prometheus will reject the metric.

Why it fails: Prometheus accepts the metric; it does not validate label cardinality or reject series at ingestion. The real risk is cardinality explosion degrading performance and storage. It is tempting because invalid metric names or malformed exposition lines are rejected, so label misuse appears similar, but user_id is syntactically valid.

The metric will stop recording.

Why it fails: High-cardinality labels such as user_id create a separate time series per user, exhausting memory and slowing queries, but the counter keeps recording. It is tempting because labels normally add useful dimensions for filtering; with a bounded set of values, such as status codes, labelling is entirely correct.

It returns an error if no data points are found.

Why it fails: irate with too few samples yields no result rather than an error; it needs at least two points in the range to compute a rate. It tempts because query errors do occur elsewhere, such as malformed expressions, but insufficient samples simply return empty output.

The result becomes a scalar.

Why it fails: irate always returns an instant vector, never a scalar, regardless of range size; a too-small range just produces no samples. It tempts because scalar() converts vectors, which is correct when you deliberately need a single numeric value for arithmetic.

The value is capped at 1.

Why it fails: irate values are rates, not bounded to 1; a too-small range yields no output, not a capped value. It tempts because ratios and percentages naturally cap at 1, which is correct when computing proportions rather than per-second rates.

group

Why it fails: group is not part of the default Prometheus ecosystem label set; those are job, instance, and (for some targets) exported labels. It tempts because grouping appears in alerting and recording rules, but that is a rule-evaluation concept, not a label attached to scraped metrics.

owner

Why it fails: owner is not a default Prometheus ecosystem label; the standard trio is job, instance, and the metric's own name. It tempts because ownership metadata is common in Kubernetes and cloud exports, but Prometheus itself does not attach it during scraping.

muted

Why it fails: Prometheus alert states are inactive, pending, and firing; muted is an Alertmanager silence concept, suppressing notifications without altering the alert's state in Prometheus. It tempts because silences do mute notifications, which is correct when you need to suppress noisy alerts during maintenance windows.

sleeping

Why it fails: Prometheus alert states are inactive, pending, and firing; sleeping is not one. It tempts because it sounds like the inactive state, but inactive means the expression returns no result, whereas sleeping is not a defined Prometheus term at all.

It returns an error.

Why it fails: PromQL's rate() does not error on counter resets; it detects the decrease and extrapolates as though the counter restarted from zero, which can yield inflated or negative-looking results. Returning an error is what happens with invalid range selectors or type mismatches, not reset handling.

It requires a manual offset.

Why it fails: rate() needs no manual offset; it automatically adjusts for counter resets within the range vector by treating each decrease as a reset. Offset modifiers exist for comparing a query against historical data, such as week-over-week capacity trends, not for correcting reset behaviour.

It stops recording the metric.

Why it fails: rate() never stops recording metrics; it is a read-time query function that computes a per-second average from stored samples and does not write anything. Recording rules are what persist derived series, and they would be the relevant mechanism if the goal were precomputing rates.

repeat_interval

Why it fails: repeat_interval governs how often a firing alert is re-notified; it does not suppress notifications, so it cannot inhibit an alert. It is tempting because it controls notification timing, and it would be the correct choice when the question asks how to reduce repeated notifications for an ongoing incident.

group_wait

Why it fails: Inhibition depends on label matchers in the inhibit_rule and the alert's own labels; group_wait merely delays the initial notification for a new alert group, so it cannot suppress anything. It is tempting because it is a real route setting, correct when you want to accumulate alerts before the first notification.

group_interval

Why it fails: Inhibition matching relies on the alert's label set and the inhibit rule's equal/not-equal matchers; group_interval only controls how long Alertmanager waits before sending batched notifications for a group, so it never suppresses a firing alert. It is tempting because it is a genuine route configuration field, correct when tuning notification batching cadence.

Alert directly on the log stream using a log aggregator.

Why it fails: Log aggregators index discrete events, so alerting on a raw stream gives no rate calculation over a window and is prone to noise. It tempts because logs hold the 5xx detail, but the requirement is a rate-based alert, which belongs on a counter metric scraped by Prometheus and evaluated with rate().

Use Prometheus to scrape the log file directly.

Why it fails: Prometheus scrapes HTTP metrics endpoints exposing counters; it does not read log files, so no 5xx counter would be collected. It tempts because Prometheus does alerting well, but the data must first be exposed as metrics, not parsed from logs by the scraper.

Increase log verbosity to trace the request until the error is resolved.

Why it fails: Raising verbosity to trace a single request produces diagnostic detail, not the error-rate metric the alert requires; rate-based alerting needs a counter such as prometheus_http_requests_total scraped by Prometheus and evaluated in a PromQL rate() expression. Verbosity tuning suits ad-hoc debugging of an already-identified failure.

To store the median

Why it fails: '_sum' accumulates the total of observed values; the median is a separate percentile metric, not a sum. Confusing them is tempting because both summarise a distribution, and a median metric would be the right choice when you need the middle value of latency samples rather than their total.

To store the maximum value

Why it fails: The '_sum' metric aggregates the sum of all observed values, not the maximum; a histogram's maximum is exposed via the '_max' metric. It is tempting because histograms do track per-bucket counts and sums, so '_sum' sounds like a total, but that total is the sum of values, not the largest one.

To store the total count

Why it fails: '_sum' stores the running total of all observed values, not the count; the count is held by '_count'. Storing the total is tempting because summing samples is a common aggregation, and '_sum' would be the correct metric when you need the cumulative value, such as total bytes transferred.

It sets the timeout for external API calls

Why it fails: That is handled by the receiver timeout.

It defines how long to store alert history

Why it fails: Alert history is not stored in Alertmanager.

It suppresses duplicate alerts

Why it fails: That is handled by grouping.