Prometheus Certified Associate (PCA, CNCF/Linux Foundation) (PCA) (PCA) — Questions 76–150

`sum` aggregates a vector's samples across matching label sets into a single instant vector, optionally grouped by labels via `by` or `without`. PromQL defines it as a core aggregation operator, so it satisfies the question's requirement for a valid operator.

blackbox_exporter probes endpoints over HTTP, HTTPS, TCP and ICMP from wherever its instance runs, exposing results as Prometheus metrics. Deploying multiple instances across regions satisfies the stem's geographic requirement, letting you alert on probe_success and latency per location.

`increase()` extrapolates a counter's total growth across the specified 1h window, correctly handling counter resets and returning the actual number of errors that occurred. `rate()` would instead give a per-second average, and `delta()` cannot handle resets, so neither satisfies the requirement for a total count over the last hour.

file_sd_configs makes Prometheus read target lists from JSON or YAML files on disk, refreshing them periodically without a restart. This satisfies the stem's constraint of discovering targets from a file rather than via API-based service discovery.

Errors counts failed requests, whether explicit failures, wrong content or policy rejections. It is a Golden Signal because it directly exposes reliability degradation, letting SRE teams alert on failure rates rather than waiting for user reports.

The rule_files directive lists the files Prometheus loads to evaluate alerting rules and recording rules. Alerting rules generate alerts sent to Alertmanager; recording rules precompute frequently used expressions into new time series, satisfying the need to define both rule types.

Descriptive, clear metric names satisfy the stem's requirement for defining metric names by making each series self-documenting, so engineers can identify what a measurement represents without external context. This reduces ambiguity during querying and alerting, directly supporting maintainability across dashboards and Prometheus-style label sets.

clamp_max() caps each sample's value at the given maximum, leaving lower values untouched. Applied to an instant vector, it returns the same series with values above the threshold replaced by that maximum, satisfying the requirement to bound peaks rather than filter or drop them.

GaugeVec wraps a Gauge with a label dimension, exposing one child metric per label-value combination via WithLabelValues. This satisfies the stem's requirement to distinguish the two types: a plain Gauge holds a single unlabelled value, whereas GaugeVec permits per-label tracking, such as by endpoint or status code.

PromQL is Prometheus's native functional query language, used across the expression browser, HTTP API and alerting rules to select and aggregate time series. It satisfies the stem's requirement as the primary language for querying Prometheus data.

Prometheus alerting rules are defined in YAML files, typically within a `groups` structure containing `alert` and `expr` fields. This satisfies the stem's requirement for the specific file format, since Prometheus natively parses rule files as YAML — not JSON or TOML — and loads them via `rule_files` in the main configuration.

A bare metric selector returns the instant vector of current samples for http_requests_total at query evaluation time, which is exactly the current value requested. No rate or aggregation function is needed for a raw counter reading.

Latency without CPU correlation points to resource contention outside the processor: storage throughput, IOPS limits, or network bandwidth and packet loss. Examining disk I/O and network saturation metrics identifies whether the bottleneck sits in I/O waits or transport, which CPU graphs cannot reveal.

Prometheus's server process binds to TCP port 9090 by default, serving its web UI, HTTP API and metrics endpoint there. This is the port scraped for Prometheus's own self-metrics and the one exposed in the default --web.listen-address setting.

Grouping aggregates alerts sharing common labels, such as alertname or cluster, into one notification, reducing noise during incidents. Configured via group_by within the route block, it batches related firing alerts so receivers get a single consolidated message rather than many.

The '_sum' metric accumulates the total of all observed values, complementing '_count' and '_bucket', which track observation counts and distribution boundaries. It satisfies the requirement to store the summed observations rather than counts or buckets.

receiver is a required route parameter naming the notification integration, such as email or Slack, that delivers alerts matching that route. Every route must define one, making it a core element of the Alertmanager routing tree.

Histogram buckets store cumulative counts per bucket, not individual observations, so the 99th percentile cannot be read directly. Merging histograms across instances requires summing bucket counts and interpolating within the target bucket, which introduces approximation error and cannot recover the true percentile.

The _count suffix exposes the number of observations aggregated into that summary or histogram series, independent of the sum or bucket values. It lets you compute averages and rates by dividing other suffixes by this sample total.

The `--no-collector.textfile` flag disables the textfile collector at startup, satisfying the requirement to switch off a specific collector. node_exporter accepts `--no-collector.<name>` for any registered collector, so the textfile collector is excluded from the metrics endpoint without affecting others.

A Gauge records an instantaneous value that can rise or fall over time, such as memory usage or queue depth, unlike counters that only increase. This matches the stem's requirement to track values that fluctuate up and down.

Prometheus scrapes each target within a configured scrape_timeout; if the exporter's response exceeds that window, the scrape is cancelled and marked as failed. Raising the timeout, or speeding up the exporter's collection, resolves the timeout errors.

Alertmanager receives alerts pushed by Prometheus and handles their routing, grouping, inhibition and silencing before dispatching notifications to receivers such as email, Slack or PagerDuty. This satisfies the stem's requirement for the component managing notification delivery, since Prometheus itself only evaluates rules and fires alerts.

A NaN gauge value signals an undefined or missing measurement, not a zero reading. The metric exists but its current state cannot be computed, so the gauge renders no valid numeric value. This matches the undefined gauge state the stem describes.

Prometheus pulls metrics from targets at controlled intervals.

Pushgateway accepts pushed metrics without verifying that the source service is still running, so Prometheus scrapes stale values indefinitely. This directly satisfies the stem's constraint: long-running services lose Prometheus's liveness detection, since the target's up metric reflects the gateway, not the actual service instance.

Histogram buckets are cumulative: each _bucket series counts all observations less than or equal to its upper bound, so values increase monotonically across buckets. This cumulative layout is what enables histogram_quantile() to interpolate quantiles from the bucket counts.

Observability infers internal system state purely from emitted outputs such as logs, metrics and traces, without inspecting internals directly. This lets operators answer novel questions about behaviour the system was never explicitly instrumented to report.

Grouping by instance creates one output series per distinct instance value, so cardinality scales linearly with cluster size. This satisfies the stem's inefficiency concern: each additional instance multiplies stored series, increasing memory and query cost.

absent() returns a non-empty result when its input vector has no samples, letting alerting rules fire on missing data. This suits detecting a metric that has stopped reporting within a given range, which is exactly the scenario described.

The group_wait parameter sets how long Alertmanager buffers alerts belonging to a new group before sending the first notification, which is precisely the global delay mechanism requested. It applies across all routes unless overridden, satisfying the stem's requirement for a global notification delay.

The Prometheus server discovers targets through service discovery, then scrapes each instance and marks it down when unreachable. This suits ephemeral, auto-scaling microservices because target health and liveness are managed centrally without instances needing to know the server's address.

The WAL stores data that hasn't been flushed to disk blocks yet.

`prometheus.MustRegister()` adds a collector to the default registry, which the `/metrics` endpoint exposes automatically. It panics on duplicate registration, surfacing programming errors immediately rather than silently dropping the metric. This satisfies the stem's requirement for the standard registration mechanism in the Go client library.

Requests per second measures demand volume hitting the web service, making it the most direct indicator of the Traffic golden signal. It reveals load changes and saturation trends that latency or error-rate metrics alone would not expose for a request-driven service.

Alertmanager's route configuration matches alerts by label and directs them to named receivers, so severity-based routing to Slack versus email belongs there. Prometheus rule files only define firing conditions; they cannot perform receiver routing, making Alertmanager the correct location.

Multiplication binds tighter than addition in PromQL, so in `vector1 + vector2 * vector3` the `*` operation is evaluated first, then the result is added to `vector1`. PromQL follows standard arithmetic precedence, giving `*` and `/` higher precedence than `+` and `-`, satisfying the stem's precedence constraint.

Short-lived batch jobs finish before Prometheus can scrape them, so their final metrics would be lost. Pushing those metrics to the Pushgateway preserves them for the Prometheus server to scrape afterwards, satisfying the ephemeral-job constraint.

Recording rules precompute expensive or frequently used PromQL expressions and store the results as new time series. Dashboards and alerts then query those precalculated series instead of re-evaluating heavy aggregations across raw data at request time, cutting query latency and load on Prometheus.

Memory scales with the number of distinct label-value series, so cardinality is the constraint. relabel_configs can drop or rewrite labels before ingestion, collapsing many series into fewer and directly cutting the stored time series that drive memory usage.

Tombstones record deleted time ranges within a series, letting Prometheus's TSDB skip those blocks during queries without immediately rewriting data on disk. This satisfies the stem's deletion-marking purpose: the series persists physically, but tombstoned intervals are excluded from reads until compaction removes them.

Each distinct dynamic URL parameter value creates its own label set, so cardinality scales with the number of unique parameter combinations. This satisfies the stem's requirement for a common high-cardinality source, since unbounded parameter variation multiplies time series far beyond a fixed label set.

The = operator performs exact string equality against a label's value, matching only samples whose label equals the given string. It is one of Prometheus's core label matching operators, alongside !=, =~ and !~, satisfying the selector requirement.

Traffic measures demand on the system, so requests per second directly quantifies the volume of incoming work. Latency, errors and saturation describe other golden signals, but only request rate satisfies the Traffic SLO requirement in the stem.

When a target disappears, Prometheus writes a stale marker into the TSDB, causing queries to stop returning that series' last sample immediately after the lookback delta. This prevents graphs from displaying flat-lined or misleading data once the target is genuinely down.

Grafana queries Prometheus as a data source and renders its time series as dashboards, graphs and alerts. Prometheus itself offers only a basic expression browser, so pairing it with Grafana provides the visualisation layer the scenario requires for exploring collected metrics.

Metrics are numeric samples recording state at a point in time, aggregated and cheap to store. Logs are timestamped records of discrete events with full context. This axis — sampled state versus discrete event — is the fundamental distinction between the two telemetry types.

By default, an alert stops at the first matching route. Setting continue: true lets evaluation proceed to sibling routes at the same level, so the alert can match multiple routes and trigger their respective receivers.

`group_left` is a many-to-one vector matching modifier, permitting each right-hand sample to pair with multiple left-hand samples while retaining left-side labels. PromQL binary vector matching requires it, alongside `on`/`ignoring` and `group_right`, to resolve cardinality mismatches between operands, satisfying the stem's request for valid matching keywords.

Prometheus rate() detects a decrease between samples and treats it as a counter reset, assuming the counter restarted at zero. It then adds the post-reset value to the pre-reset value, so the restart's lost increments are not counted as a negative rate, preserving an accurate request rate.

Errors is the golden signal tracking failed requests, and HTTP 404 responses count as errors for a specific endpoint. Monitoring error rate per endpoint reveals whether clients are requesting nonexistent resources, satisfying the detection requirement.

By default a matching alert stops at the first route. Setting continue to true tells Alertmanager to keep evaluating sibling routes after a match, so the same alert is also delivered through those additional routes rather than being consumed by the first.

Latency is a duration measured repeatedly over time, so time-series metrics capture request durations as timestamped samples, enabling percentile and histogram analysis. This satisfies the stem's requirement, unlike logs, which record discrete events without native aggregation.

The Blackbox exporter performs active probing from outside the target, and its probe modules support HTTP, TCP, DNS, ICMP and gRPC checks. Supporting HTTP, TCP and DNS satisfies the multi-protocol probing capability the question asks about.

Firing is the active state an alert enters once its PromQL expression evaluates true for the configured 'for' duration, meaning the condition currently holds. It contrasts with inactive and pending, so this option correctly describes the live breach the rule detects.

Prometheus defaults its scrape interval to one minute when no explicit value is set, so a target without a custom `scrape_interval` is polled every 60 seconds. This satisfies the stem's request for the default, distinguishing it from shorter intervals such as 15s or 30s that must be configured manually.

The module block selects the probe type—HTTP, TCP, ICMP or DNS—and its parameters such as preferred IP protocol, timeouts and valid status codes. It therefore determines the protocol-specific behaviour the blackbox_exporter applies when testing the web endpoint.

Receivers is a top-level Alertmanager configuration block listing named notification integrations, such as email, Slack or PagerDuty, each referenced by routes. Without at least one receiver defined, Alertmanager cannot dispatch any notification, making it a required valid component.

Client-side quantiles are computed within the instrumented application before export, so the Summary metric type satisfies the stem's requirement for locally calculated quantiles rather than server-side aggregation. This distinguishes it from histogram-based types, where quantiles are derived later from bucketed data in the backend.

Evaluating rules is a core Prometheus server responsibility: it periodically runs the alerting and recording rules loaded from rule_files, producing alerts and precomputed series. This sits alongside scraping and storing time series in the server's own duties.

In the pull model Prometheus actively initiates each scrape over HTTP against configured targets, rather than passively receiving pushed data. This satisfies the stem's requirement by naming the direction of initiation, the defining axis separating pull from push architectures.

`sum` aggregates sample values across matching series, and both `by` and `without` modifiers are valid for it, letting you group results by specified labels or strip them from the output. This satisfies the stem's requirement for operators supporting both clause modifiers.

The instance label records the specific target endpoint scraped, typically host:port, distinguishing multiple replicas or jobs sharing the same metric names. This satisfies the need to identify which individual target produced a given sample during querying and alerting.

rate() computes the per-second average rate of increase of a counter time series across the range vector, handling counter resets automatically. This satisfies the constraint of calculating per-second average increase, unlike irate() which uses only the last two samples.

Scrape intervals are set per job within `scrape_configs`, where each job's `scrape_interval` overrides the global default. This satisfies the stem's requirement to define the interval for a specific target, since targets are grouped under a job and inherit that job's interval unless individually relabelled.

Aggregating raw metrics into broader categories directly reduces cardinality by collapsing many distinct label combinations into fewer series. For example, grouping individual HTTP status codes into 2xx, 4xx and 5xx classes cuts the number of unique time series Prometheus must track, satisfying the stem's requirement for a standard cardinality-reduction technique.

A short range vector such as [10s] samples only the last two scrape points, so irate divides the freshest delta by that tiny interval. This captures high-frequency spikes but produces volatile, noisy results, satisfying the stem's stated impact of an undersized range.

histogram_quantile() calculates a quantile from histogram buckets, interpolating within the bucket containing the requested rank. Applied to a histogram's le-labelled bucket counters with rate(), it returns the P99 latency estimate, which no counter or summary function provides directly.

The write-ahead log persists incoming samples to disk before they reach in-memory blocks, so a crash can be recovered by replaying it. This satisfies the durability requirement, unlike the head block, which is volatile until flushed.

The offset modifier shifts the entire query evaluation back by the specified duration, so `http_requests_total offset 1h` returns the sample value as it existed exactly one hour before the current evaluation timestamp. This satisfies the stem's requirement for the precise time Prometheus queries when an offset is applied.

The 'for' field requires the alert's condition to remain true for a specified duration before firing, so brief flapping spikes are ignored while sustained problems still alert. This suppresses transient noise without silencing the rule entirely, unlike inhibition or muting.

label_replace() rewrites labels on time series using a regex with capture groups, letting you add or modify a destination label from a source label. It operates within PromQL queries, satisfying the requirement for a valid PromQL label-modification method.

max_over_time evaluates the range vector over the trailing one-hour window and returns the highest sample per series, giving the peak gauge value across all instances. The [1h] duration supplies the sliding window the stem specifies.

CPU usage percentage is a numeric measurement sampled over time, which is exactly what Prometheus stores as a time series. It satisfies the stem's requirement for metric data, unlike logs or traces, because its value is quantifiable and aggregatable.

Prometheus Server executes PromQL queries directly against its time-series database, retrieving and aggregating metric samples on demand. This satisfies the stem's requirement precisely: PromQL is Prometheus's native query language, evaluated by the server's query engine rather than by exporters, Alertmanager, or client libraries.