Alerting And Dashboarding practice questions

predict_linear()

Why it fails: This is for forecasting future values, not showing the current one.

sum_over_time()

Why it fails: This aggregates values over a duration, which is not suitable for a current gauge.

rate()

Why it fails: rate() is for calculating per-second growth of counters, not for displaying a single instantaneous gauge value.

increase()

Why it fails: increase() calculates the total increase, not the per-second rate.

avg()

Why it fails: avg() is an aggregation, not a rate calculation.

sum()

Why it fails: sum() is an aggregation, not a rate calculation.

To set the alert threshold

Why it fails: Alert thresholds are configured in the alert rule, not the Legend field, which only controls series labelling and display. It is tempting because both appear on the same panel configuration screen, but the Legend has no evaluation logic and cannot trigger notifications when a metric crosses a value.

To enable auto-refresh

Why it fails: Auto-refresh is set by the dashboard's refresh interval control, not the Legend field, which governs series naming and display. It is tempting because both sit in panel or dashboard settings, but the Legend has no scheduling mechanism and cannot trigger periodic query re-execution.

To filter which metrics are shown

Why it fails: Legend controls how series are labelled and displayed on the panel, not which metrics are queried; filtering is done through the query itself or template variables. It is tempting because legends do affect what appears visually, but they only annotate already-returned series, never restrict the underlying data set.

It sets the timeout for external API calls

Why it fails: That is handled by the receiver timeout.

It defines how long to store alert history

Why it fails: Alert history is not stored in Alertmanager.

It suppresses duplicate alerts

Why it fails: That is handled by grouping.

By tailing the prometheus.log file

Why it fails: Logs do not provide the real-time status of all rules.

Using 'curl' on the TSDB files

Why it fails: TSDB files are binary and not meant for direct querying.

Using 'promtool check rules'

Why it fails: This command checks syntax, not live status.

The rule file has invalid YAML indentation

Why it fails: A labels conflict arises when the recording rule's output labels clash with labels already present on the input series, not from malformed YAML. Invalid indentation would produce a parse or syntax error at load time, so it is tempting only because both faults stop the rule from evaluating.

The evaluation interval is too low

Why it fails: The evaluation interval controls how often the rule runs, not how its output labels are constructed, so a short interval cannot produce a labels conflict. It is tempting because a low interval does cause evaluation problems, but those manifest as missed or overlapping evaluations rather than label clashes.

The metric name is already in use

Why it fails: A labels conflict concerns colliding label names between the rule expression's output and its input series, not reuse of a metric name. Name collisions would surface as duplicate series or overwrite behaviour, so this is tempting only because both faults involve conflicting identifiers within the rule.

receiver: 'email'

Why it fails: Receivers are defined in a separate section.

alerts: { to: 'email' }

Why it fails: This is not valid Alertmanager config.

route: [ 'email' ]

Why it fails: Incorrect syntax structure.

Check the Prometheus metrics for total alerts

Why it fails: This doesn't show routing specifics.

Restart the server

Why it fails: Restarting won't help diagnose routing logic.

Delete all silences

Why it fails: Silences might be blocking alerts, but they aren't the primary debugging tool.

group_wait

Why it fails: Affects initial notification delay.

alert_limit

Why it fails: Not a valid field.

group_interval

Why it fails: Affects batching of notifications.

match_re

Why it fails: Not part of the inhibition block syntax.

repeat

Why it fails: Not part of the inhibition block.

repeat_interval

Why it fails: repeat_interval governs how often a firing alert is re-notified; it does not suppress notifications, so it cannot inhibit an alert. It is tempting because it controls notification timing, and it would be the correct choice when the question asks how to reduce repeated notifications for an ongoing incident.

group_wait

Why it fails: Inhibition depends on label matchers in the inhibit_rule and the alert's own labels; group_wait merely delays the initial notification for a new alert group, so it cannot suppress anything. It is tempting because it is a real route setting, correct when you want to accumulate alerts before the first notification.

group_interval

Why it fails: Inhibition matching relies on the alert's label set and the inhibit rule's equal/not-equal matchers; group_interval only controls how long Alertmanager waits before sending batched notifications for a group, so it never suppresses a firing alert. It is tempting because it is a genuine route configuration field, correct when tuning notification batching cadence.

Installing the Prometheus plugin in Grafana

Why it fails: Prometheus support is built-in; no external plugin is required.

Configuring alert rules in the Prometheus YAML file

Why it fails: This is a Prometheus configuration, not a Grafana-Prometheus integration method.

The 'resolve_timeout' parameter

Why it fails: resolve_timeout governs how long Alertmanager waits before marking an alert resolved after no new notifications arrive; it does not delay initial firing during flapping. It is tempting because it controls alert lifecycle timing, and it would be correct for tuning recovery detection, but suppressing repeated flapping notifications requires grouping or repeat-interval control.

The 'group_wait' parameter

Why it fails: group_wait delays the initial notification for a new alert group, so it affects first delivery timing, not repeated re-firing during flapping. It is tempting because it batches notifications and would be correct for grouping related alerts from one incident, but damping flapping requires repeat_interval or for-duration control.

The 'inhibit_rules' block

Why it fails: inhibit_rules suppress a target alert while a matching source alert fires, which silences related alerts rather than damping repeated flapping of the same rule. It is tempting because it reduces notification noise, and it would be correct when one outage should mute dependent alerts, but here the same alert must still fire, just less often.

In the Alertmanager memory

Why it fails: Alertmanager holds alert state, silences and notification routing, not recorded metric samples; recording rule output is written to Prometheus's time series database. It tempts because recording rules frequently feed alerting expressions, so Alertmanager consumes the metric, but consuming is not storing.

In the Grafana database

Why it fails: Recording rules write their output into Prometheus's own time series storage, not Grafana's database; Grafana only queries and visualises that data. It tempts because Grafana dashboards display the recorded series, making it appear the metric lives there, but Grafana stores dashboards and configuration, not samples.

In a local flat file

Why it fails: Recording rule results are appended to Prometheus's time series database as new series, not written to any local flat file. It tempts because Prometheus does persist blocks on disk, but those are its internal TSDB format queried via PromQL, not a flat file the rule writes.

The alert is dropped

Why it fails: Alerts lacking a grouped label are grouped under an empty value rather than discarded; notification still proceeds through the matching route. Dropping occurs only when routing rules or inhibition explicitly suppress the alert, not because a group_by label is absent from its label set.

The alert triggers an error in the logs

Why it fails: Alertmanager treats a missing label as an empty string value for grouping purposes; it neither errors nor logs a failure, so alerts still route and notify normally. Logging errors would be expected only for malformed configuration syntax or invalid matcher expressions rejected at load time.

The routing rule fails

Why it fails: A missing label is simply treated as an empty value, so grouping and routing proceed normally. Routing failure is tempting because absent labels can break inhibition matching, but group_by tolerates them; the option would fit a question about required matchers in a route.

Delete the alerting rule

Why it fails: Deleting the rule stops the alert permanently, not just during the maintenance window, and removes detection afterwards. Silences are time-bounded and label-matched, so they suit temporary suppression; deleting a rule is chosen only when the alert should never fire again.

Modify the routing tree

Why it fails: Routing trees decide which receiver gets a firing alert after it fires; they cannot stop the alert firing during maintenance. Silences, created via the API, UI or amtool with a matcher and time window, suppress notifications, so routing-tree edits suit re-directing alerts, not maintenance windows.

Add a 'skip' label to the rule

Why it fails: Alertmanager has no 'skip' label semantics; labels select routes and receivers, they do not suppress evaluation. Silence resources, matched by label, are the mechanism for maintenance windows, so a skip label would be chosen only by someone assuming arbitrary labels alter firing behaviour.

Route tree

Why it fails: The route tree matches alerts to receivers and applies grouping, timing and child routes; it cannot express dependency-based suppression between distinct alerts. Inhibition handles that by muting DiskSpaceLow when InstanceDown is active for the same host. Route trees would be correct for directing alerts by label to different teams.

Repeat intervals

Why it fails: Repeat intervals control how often an already-notified alert is re-sent while still firing; they do not suppress one alert because another is active. Inhibition performs that dependency-based muting. Repeat intervals would be correct when tuning notification frequency for long-running unresolved alerts.

Grouping

Why it fails: Grouping batches related alerts into a single notification to reduce noise; it does not suppress one alert based on another's state. Inhibition is the mechanism that silences DiskSpaceLow while InstanceDown is firing for the same host. Grouping would be correct when consolidating many alerts from one cluster into fewer messages.

Decrease the evaluation_interval

Why it fails: Lowering evaluation_interval makes recording rules execute more frequently, increasing CPU consumption rather than reducing it. It is tempting because the setting controls rule evaluation cadence, and lengthening it would cut load, but the option says decrease, which shortens the interval between evaluations.

Move recording rules to Alertmanager

Why it fails: Alertmanager only routes, groups and silences alerts; it cannot evaluate recording rules, which are PromQL expressions executed by Prometheus itself. It is tempting because Alertmanager handles alerting load, but offloading rule evaluation requires a separate Prometheus instance or federation, not the alert router.

Delete all indices

Why it fails: Deleting indices removes stored time-series data, so historical metrics become unavailable, violating the requirement to keep data available. It is tempting because deleting data frees disk and can reduce query load, but the CPU is consumed by rule evaluation, not by index storage.

JSON

Why it fails: Prometheus rule files use YAML, parsed by the rule loader; JSON is accepted only for API queries and HTTP responses, not rule definitions. JSON is tempting because it is a common configuration and data-interchange format, and it would be the correct choice when defining Grafana dashboards or querying the Prometheus HTTP API.

TOML

Why it fails: Prometheus rule files require YAML syntax; TOML is not supported by the rule loader. TOML is tempting because it is a legitimate configuration format, and it would be the correct choice for defining Cargo manifests or Hugo site configuration, not alerting rule groups.

XML

Why it fails: Prometheus alerting rules are written in YAML; XML is not parsed by the rule loader at all. XML is tempting as a widely used structured markup format, and it would be the correct choice for defining Maven build files or SOAP service contracts, not Prometheus rule groups.

Alertmanager is down

Why it fails: Alertmanager handles routing, grouping and silencing of fired alerts; it does not evaluate rules, so its availability cannot cause evaluation errors. It is tempting because both components sit in the alerting pipeline, but it would be correct when alerts fire but notifications fail to reach receivers.

Too many dashboards

Why it fails: Dashboard count has no bearing on the rule evaluation engine; dashboards query data but do not execute alerting rules. It is tempting because heavy dashboard querying can load Prometheus, but it would be correct when the symptom is slow queries or high memory use, not evaluation errors.

Prometheus is out of disk space

Why it fails: Disk exhaustion on the Prometheus server prevents rule evaluation from completing, but the alerting rule evaluation error specifically indicates a malformed or invalid rule expression, such as a PromQL syntax or type error. It is tempting because resource pressure does degrade Prometheus, yet that surfaces as scrape or write failures, not rule evaluation errors.