CCNA Alerting And Dashboarding Questions

Recording rules precompute expensive expressions into new series, so dashboards and alerts reference a single stored metric instead of re-running complex PromQL. This satisfies the stem's benefit by simplifying complex queries and reducing repeated evaluation cost at query time.

Adding a Prometheus data source in Grafana queries the Prometheus HTTP API directly, so no intermediary exporter or federation layer is needed. This satisfies the stem's requirement to visualise existing Prometheus metrics, since Grafana's built-in data source plugin translates PromQL queries and renders the returned time series natively.

Raising evaluation_interval lengthens the gap between rule evaluations, so each rule query executes less often and consumes proportionally less CPU. Recorded series remain written to storage, satisfying the constraint of keeping data available while cutting load.

Alertmanager validates configuration on reload; if parsing fails, it rejects the new file and keeps the previously loaded configuration running. Alerts therefore continue routing normally, satisfying the stem's scenario without dropping notifications or halting the service.

Variables enable users to dynamically switch or select data sources in a panel.

Time series panels plot metric values against a timestamped x-axis, rendering successive data points as a continuous line. This directly satisfies the requirement to visualise how a metric changes over time, unlike stat, gauge or table panels which show only current or aggregated values.

The `repeat_interval` parameter sets how frequently Alertmanager re-notifies about an alert that remains firing after the initial notification, provided the alert has not been resolved or updated. It satisfies the scenario's requirement to define resend frequency for ongoing incidents, distinct from `group_interval`, which governs batching delays between different alert groups.

Inhibition suppresses a lower-severity alert while a matching higher-severity alert is firing, using equal label matchers. This satisfies the stem's requirement by muting ServiceDown whenever GlobalNetworkOutage is active, preventing duplicate notification storms during broad outages.

Alertmanager evaluates group_by labels against each alert's label set. A label absent from an incoming alert resolves to an empty string, so all alerts missing that label collapse into a single group keyed by a null value.

Inhibition rules suppress a target alert when a matching source alert is firing, using equal label matchers such as instance. This directly satisfies the stem's requirement to mute DiskSpaceLow while InstanceDown is active for the same host, which silences and routing cannot achieve.

A Prometheus alerting rule's 'expr' field holds the PromQL expression; the alert fires when that expression's result satisfies the rule's 'for' duration. The threshold comparison itself is written inside the PromQL expression, not in a separate numeric field.

Alertmanager configuration routes alerts by matching label sets, so a route with a `service` matcher directs each alert to the correct receiver, satisfying the requirement to notify different teams per service label. This logic belongs in Alertmanager's routing tree, not in Prometheus scrape configs or Grafana dashboards.

The `expr` field holds the PromQL expression Prometheus evaluates on each evaluation interval; a non-empty result triggers the alert. It is mandatory in every alerting rule, satisfying the stem's requirement for a valid field alongside `alert` and `for`.

The --cluster.peer flag joins Alertmanager instances into a gossip cluster, so notification state and silences are shared; if communication between peers fails, each node still evaluates and dispatches its own alerts. This satisfies the requirement that alerts continue being sent during cluster partition.

Alertmanager's `inhibit_rules` block suppresses a target alert when a matching source alert is firing, using `equal` labels to scope the match. Setting `source_matchers: [alertname="InstanceDown"]`, `target_matchers: [alertname="HighErrorRate"]` and `equal: [instance]` silences HighErrorRate only for the same instance, satisfying the stem's per-instance constraint.

Prometheus alerting rules are defined in YAML files, typically within a `groups` structure containing `alert` and `expr` fields. This satisfies the stem's requirement for the specific file format, since Prometheus natively parses rule files as YAML — not JSON or TOML — and loads them via `rule_files` in the main configuration.

Grouping aggregates alerts sharing common labels, such as alertname or cluster, into one notification, reducing noise during incidents. Configured via group_by within the route block, it batches related firing alerts so receivers get a single consolidated message rather than many.

receiver is a required route parameter naming the notification integration, such as email or Slack, that delivers alerts matching that route. Every route must define one, making it a core element of the Alertmanager routing tree.

Grouping by instance creates one output series per distinct instance value, so cardinality scales linearly with cluster size. This satisfies the stem's inefficiency concern: each additional instance multiplies stored series, increasing memory and query cost.

The group_wait parameter sets how long Alertmanager buffers alerts belonging to a new group before sending the first notification, which is precisely the global delay mechanism requested. It applies across all routes unless overridden, satisfying the stem's requirement for a global notification delay.

Alertmanager's route configuration matches alerts by label and directs them to named receivers, so severity-based routing to Slack versus email belongs there. Prometheus rule files only define firing conditions; they cannot perform receiver routing, making Alertmanager the correct location.

By default, an alert stops at the first matching route. Setting continue: true lets evaluation proceed to sibling routes at the same level, so the alert can match multiple routes and trigger their respective receivers.

By default a matching alert stops at the first route. Setting continue to true tells Alertmanager to keep evaluating sibling routes after a match, so the same alert is also delivered through those additional routes rather than being consumed by the first.

Firing is the active state an alert enters once its PromQL expression evaluates true for the configured 'for' duration, meaning the condition currently holds. It contrasts with inactive and pending, so this option correctly describes the live breach the rule detects.

Receivers is a top-level Alertmanager configuration block listing named notification integrations, such as email, Slack or PagerDuty, each referenced by routes. Without at least one receiver defined, Alertmanager cannot dispatch any notification, making it a required valid component.

The 'for' field requires the alert's condition to remain true for a specified duration before firing, so brief flapping spikes are ignored while sustained problems still alert. This suppresses transient noise without silencing the rule entirely, unlike inhibition or muting.

Alertmanager's routing tree has a root node, and every alert enters there; without a matching child route, the alert inherits the root route's receiver and continues to be notified. This satisfies the question's scenario of an alert with no explicitly defined route.

group_by is a core Alertmanager route parameter that partitions alerts into separate notification groups by label values, controlling which alerts are batched together. It directly satisfies the question's demand for a valid grouping parameter, alongside group_wait and group_interval.

Basic Auth is the standard mechanism: Grafana stores the username and password in the Prometheus data source configuration and sends them in the Authorization header, which the reverse proxy validates before forwarding requests to the Prometheus server.

Webhook receivers let Alertmanager POST a JSON payload to any HTTP endpoint, satisfying the requirement for a standard notification route. This is one of the built-in receiver types configured under `receivers:` in the Alertmanager configuration file, alongside email and Slack, requiring no external integration component.

Alertmanager supports receivers such as email, Slack, PagerDuty, webhook and Opsgenie for delivering notifications. SQL is a database technology, not a notification integration, so it cannot be configured as a valid receiver type in the Alertmanager routing configuration.

Labels attach key-value metadata to an alert, which Alertmanager matches against routes, silences and inhibition rules to determine where and whether notifications are delivered. This satisfies the stem's routing and filtering requirement, distinguishing labels from annotations, which only carry descriptive text.

The `expr` field holds the PromQL expression that Prometheus evaluates on each evaluation interval; when the result is non-empty, the alert transitions to pending or firing. It is mandatory in every alerting rule, satisfying the stem's requirement for a valid component of an Alerting Rule file.

The Alertmanager UI is a built-in web interface served by Alertmanager itself, showing active alerts, silences and grouping state. It satisfies the stem because Prometheus's own expression browser does not display fired alerts in this way.

Recording rules evaluate their expression and write the resulting samples back into the Prometheus time-series database under the new metric name. The rule does not create a separate store; results persist in the TSDB alongside scraped metrics.

Prometheus evaluates each alerting rule's expression against the query engine; malformed syntax or invalid functions in the PromQL cause evaluation to fail, producing exactly this error. Correcting the expression's syntax resolves the alerting rule evaluation error.

Recording rules precompute the expensive PromQL expression and persist the result as a new time series, so dashboards read a cheap, already-aggregated metric instead of re-evaluating the complex query on every refresh, directly cutting dashboard query latency.

A recording rule precomputes the rate expression and stores the result as a new time series, so dashboards read a small precalculated series instead of evaluating the full underlying expression at render time. This reduces query latency on dashboards, satisfying the stated preference.

keep_firing_for holds an alert in firing state for a set duration after its condition stops being true, preventing rapid transitions between firing and resolved. This directly addresses flapping, which the stem identifies as the concern.

The 'description' annotation is used to provide details about the alert.

The `for` field sets a pending duration, during which the alert must remain continuously firing before Alertmanager receives it. Configuring `for: 5m` satisfies the stem's requirement to delay notification until the condition has persisted five minutes, filtering transient spikes that would otherwise trigger premature alerts.

A Silence matches alerts by label set and suppresses notifications for a defined time window, which maps directly onto a scheduled maintenance period. Unlike inhibition, which depends on another firing alert, a Silence is time-bounded and configured in advance, satisfying the maintenance-window constraint.

A recording rule referencing a non-existent metric produces no output series, because the expression evaluates against an empty vector and yields nothing. This satisfies the stem's impact directly: no data is written, so dependent dashboards and alerts querying that recorded series return empty results rather than errors.