PCA Alerting And Dashboarding — 20 Questions

Question 1

You are creating a Grafana dashboard and need to display a gauge showing the current CPU usage percentage from a Prometheus data source. Which function is most appropriate?

Accepted Answer

No function (raw query). For a gauge showing current CPU usage percentage, you need the instantaneous value, not a derived rate or prediction. CPU usage percentage is typically exposed by exporters like node_exporter as a gauge metric (e.g., node_cpu_seconds_total is a counter, but the percentage is often calculated via a recording rule or a query like 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100). However, if the metric is already a gauge representing the current percentage (e.g., from a custom exporter or a precomputed metric), applying any function would alter the value. The raw query returns the exact current value, which is what a gauge panel should display. Functions like rate() are for counters, predict_linear() for forecasting, and sum_over_time() for aggregating over a time window—none are appropriate for a single current percentage.

Answer

predict_linear()

Answer

sum_over_time()

Answer

rate()

Question 2

Which PromQL function is used to convert a gauge into a rate of increase?

Accepted Answer

rate(). The rate() function is specifically designed to calculate the per-second rate of increase for counter metrics.

Answer

increase()

Answer

avg()

Answer

sum()

Question 3

When configuring a dashboard panel, what is the significance of the 'Legend' field?

Accepted Answer

To rename the metric for display purposes. The legend field allows you to customize the labels displayed in the graph tooltip and legend section.

Answer

To set the alert threshold

Answer

To enable auto-refresh

Answer

To filter which metrics are shown

Question 4

What does the 'resolve_timeout' setting in Alertmanager do?

Accepted Answer

It determines how long to wait before sending a resolved notification. If an alert stops firing, Alertmanager waits for the resolve_timeout duration to ensure the alert has truly resolved before notifying about the resolution.

Answer

It sets the timeout for external API calls

Answer

It defines how long to store alert history

Answer

It suppresses duplicate alerts

Question 5

How can you view the current status of all alerts directly from the Prometheus server CLI?

Accepted Answer

By visiting the /alerts endpoint in the browser. The Prometheus UI on port 9090 has an 'Alerts' tab that shows every configured rule and its current status.

Answer

By tailing the prometheus.log file

Answer

Using 'curl' on the TSDB files

Answer

Using 'promtool check rules'

Question 6

You have a recording rule that fails to evaluate because of a 'labels conflict'. What is the most likely cause?

Accepted Answer

The aggregation results in duplicate label sets. A label conflict occurs when the recording rule tries to create a metric with labels that are already present or restricted, or when the aggregation produces duplicate label sets.

Answer

The rule file has invalid YAML indentation

Answer

The evaluation interval is too low

Answer

The metric name is already in use

Question 7

Which of these is the correct syntax for a simple Alertmanager route?

Accepted Answer

route: { receiver: 'email' }. The route uses a 'match' or 'match_re' block to filter alerts and a 'receiver' field to specify where they go.

Answer

receiver: 'email'

Answer

alerts: { to: 'email' }

Answer

route: [ 'email' ]

Question 8

How can you debug an Alertmanager routing issue?

Accepted Answer

Use 'amtool alert show'. The Alertmanager web UI allows you to see which alerts are arriving and provides information on which route they matched.

Answer

Check the Prometheus metrics for total alerts

Answer

Restart the server

Answer

Delete all silences

Question 9

Which TWO of the following are valid ways to prevent alert flapping?

Accepted Answer

keep_firing_for. Option A, keep_firing_for, is correct because it is an Alertmanager rule parameter that keeps an alert in firing state for a specified duration after the condition stops being true, preventing rapid transitions between firing and resolved states that cause flapping. Option D, for, is correct because it is a Prometheus alerting rule parameter that requires the alert condition to be continuously true for a specified duration before the alert fires, filtering out transient spikes that would otherwise cause flapping. Option B, group_wait, is incorrect because it only controls how long Alertmanager waits to send the initial notification for a new group of alerts, affecting notification batching rather than alert state stability. Option C, alert_limit, is incorrect because it is not a valid Prometheus or Alertmanager configuration parameter for preventing flapping. Option E, group_interval, is incorrect because it only sets the wait time before sending notifications about new alerts added to an existing group, which affects notification timing but not the underlying alert flapping behavior.

Answer

group_wait

Answer

alert_limit

Answer

group_interval

Question 10

Which THREE of the following fields exist in the Alertmanager 'inhibit_rules' block?

Accepted Answer

source_match. Inhibition rules require 'source_match', 'target_match', and 'equal' fields to function.

Answer

match_re

Answer

repeat

Question 11

Which TWO of the following are necessary to successfully inhibit an alert in Alertmanager?

Accepted Answer

target_matchers. Inhibition requires a 'target_matchers' (the alert to be silenced) and 'source_matchers' (the alert that triggers the silence), plus common labels to correlate the two.

Answer

repeat_interval

Answer

group_wait

Answer

group_interval

Question 12

Which THREE of the following are standard ways to integrate Grafana with Prometheus?

Accepted Answer

Using PromQL to query metrics in panels. Grafana connects to Prometheus via the Data Source interface, supports PromQL queries in panels, and can visualize metrics from Prometheus exporters.

Answer

Installing the Prometheus plugin in Grafana

Answer

Configuring alert rules in the Prometheus YAML file

Question 13

You have a Prometheus alerting rule that triggers too frequently during flapping states. Which feature should you use to prevent this without silencing the alert entirely?

Accepted Answer

The 'for' field. The 'for' field in a Prometheus alerting rule specifies how long the condition must be true before the alert fires. By setting a duration (e.g., '5m'), it prevents alerts from triggering during brief flapping states, as the condition must persist for that period. This reduces noise without silencing the alert entirely.

Answer

The 'resolve_timeout' parameter

Answer

The 'group_wait' parameter

Answer

The 'inhibit_rules' block

Question 14

You have a recording rule named 'job:node_cpu:avg_rate_5m'. Where is this metric stored once the rule is executed?

Accepted Answer

In the Prometheus TSDB. Recording rules in Prometheus evaluate a PromQL expression on a schedule and write the resulting samples back into the Prometheus TSDB as new time series. The metric 'job:node_cpu:avg_rate_5m' is therefore stored in the same local TSDB as scraped metrics, queryable via PromQL like any other series. This is the core purpose of recording rules: precomputing expensive queries for faster dashboards and alerts.

Answer

In the Alertmanager memory

Answer

In the Grafana database

Answer

In a local flat file

Question 15

When configuring Alertmanager 'group_by' settings, what happens if you include a label that is not present on an incoming alert?

Accepted Answer

The alert is grouped under a null value. Alertmanager's group_by directive groups alerts by the values of the specified labels. If a label listed in group_by is absent from an incoming alert, Alertmanager treats its value as empty/null and still groups the alert under that null value rather than dropping or erroring. This means alerts missing the label are grouped together in a single notification group.

Answer

The alert is dropped

Answer

The alert triggers an error in the logs

Answer

The routing rule fails

Question 16

You need to prevent an alert from firing if a maintenance window is active. How should you approach this in Alertmanager?

Accepted Answer

Use a Silence. Silences are the standard way to prevent specific alerts from firing during a known maintenance period.

Answer

Delete the alerting rule

Answer

Modify the routing tree

Answer

Add a 'skip' label to the rule

Question 17

You have a global Alertmanager configuration where you want to suppress 'DiskSpaceLow' alerts if 'InstanceDown' is active for the same host. Which mechanism accomplishes this?

Accepted Answer

Inhibition rules. Inhibition rules in Alertmanager are designed to suppress a set of alerts when another set of alerts is already firing. They match on labels, so you can define an inhibition rule that mutes 'DiskSpaceLow' alerts if an 'InstanceDown' alert with the same host label is active. This is exactly the use case described: suppressing dependent alerts when a more fundamental alert is firing.

Answer

Route tree

Answer

Repeat intervals

Answer

Grouping

Question 18

You notice that recording rules are consuming too much CPU on your Prometheus server. What is the most effective way to reduce the load while keeping data available?

Accepted Answer

Increase the evaluation_interval. Recording rules in Prometheus are evaluated at the interval defined by 'evaluation_interval' in the rule group. Increasing the evaluation_interval means rules run less frequently, which directly reduces CPU load on the Prometheus server while keeping the recorded time series available for querying.

Answer

Decrease the evaluation_interval

Answer

Move recording rules to Alertmanager

Answer

Delete all indices

Question 19

Which file format is used to define Prometheus alerting rules?

Accepted Answer

YAML. Prometheus alerting rules are defined in YAML format. The rule files are specified in the Prometheus configuration and contain groups of rules with expressions and durations. YAML is the standard for Prometheus configuration files.

Answer

JSON

Answer

TOML

Answer

XML

Question 20

You are seeing 'Alerting rule evaluation error' in your logs. What is the most likely cause?

Accepted Answer

Invalid PromQL in the expression. The error message 'Alerting rule evaluation error' originates from the Prometheus rule evaluation engine when it cannot successfully execute the PromQL expression defined in a rule. The most common cause is a syntactically or semantically invalid PromQL expression — for example, referencing a non-existent metric, using an unsupported function, or mismatching label matchers. Prometheus logs this error per rule per evaluation cycle, so the fix is to validate the expression in the Prometheus expression browser or with promtool check rules.

Answer

Alertmanager is down

Answer

Too many dashboards

Answer

Prometheus is out of disk space