Prometheus Certified Associate (PCA, CNCF/Linux Foundation) (PCA) (PCA) — Questions 151–225

Every distinct label combination creates a separate time series, so high cardinality multiplies the series Prometheus must track. Each series consumes memory in the head block and index, driving up RAM usage on the server.

The by (instance) modifier groups series by the instance label and sums each group's CPU counters, retaining instance while collapsing other labels such as cpu and mode. This satisfies the stem's requirement to aggregate across nodes yet keep instance.

topk(3, ...) returns the three series with the highest instantaneous values, and wrapping rate() over the 5m range converts the cumulative counter process_cpu_seconds_total into a per-second CPU usage rate, satisfying the requirement to rank jobs by CPU consumption.

The `le` pseudo-label means "less than or equal to", recording each bucket's inclusive upper bound. This satisfies the stem's constraint by exposing the cumulative count of observations falling at or below that threshold, letting you compute per-bucket counts by subtracting adjacent series.

`sum()` aggregates a metric's samples across every matched instance into a single total, satisfying the requirement to calculate the overall sum. Unlike `avg()`, which divides by sample count, or `max()`/`min()`, which select extremes, `sum()` adds each instance's contribution, yielding the combined total across all instances.

The offset modifier shifts the evaluation instant backwards by the given duration, so errors_total offset 1h returns the value from one hour ago. This satisfies the scenario's requirement to query a point in the past.

count() is the aggregation operator that returns the number of series in each group, so count(up) yields the total active series matching the selector. sum() would add sample values instead of counting series, which does not satisfy the requirement.

Alertmanager's routing tree has a root node, and every alert enters there; without a matching child route, the alert inherits the root route's receiver and continues to be notified. This satisfies the question's scenario of an alert with no explicitly defined route.

In a Prometheus histogram, the `le` label defines each bucket's inclusive upper bound, so a bucket labelled `le="0.5"` counts every observation less than or equal to 0.5 seconds. This satisfies the stem's requirement by naming the comparison operator that determines which observations fall into each cumulative bucket.

group_by is a core Alertmanager route parameter that partitions alerts into separate notification groups by label values, controlling which alerts are batched together. It directly satisfies the question's demand for a valid grouping parameter, alongside group_wait and group_interval.

Scalars carry no labels, so arithmetic between a vector and scalar applies element-wise to every sample, leaving the vector's label set untouched. This satisfies the stem's requirement that dividing by `(1024 * 1024)` converts bytes to mebibytes without dropping or altering any series identity.

The 'or' operator returns the union of two instant vectors, taking the left-hand series and adding any right-hand series whose label sets do not already match. 'and' and 'unless' perform intersection and complement instead, so they cannot produce a union.

An instant vector holds one sample per time series at a single evaluation timestamp, unlike a range vector which holds many samples per series. This single-sample-per-series structure is the defining property that makes instant vectors suitable for instant queries and comparisons.

Changing a metric's label set alters its identity, so Prometheus treats it as a distinct series rather than mutating the existing one. The original series stops receiving samples and a new time series begins under the modified labels, satisfying the stem's constraint that label changes during a metric's lifetime create separate series.

Emitting a counter metric per 5xx error lets Prometheus apply rate() over the series, so alerting rules fire on error frequency rather than raw log volume. This satisfies the requirement to trigger on the rate of errors, which log-based alerting cannot express directly.

rate() computes the per-second average rate of increase of a counter over the given range, handling counter resets and extrapolation at the range boundaries. The 10m range selector supplies the required window, yielding a per-second figure rather than a raw increase.

Basic Auth is the standard mechanism: Grafana stores the username and password in the Prometheus data source configuration and sends them in the Authorization header, which the reverse proxy validates before forwarding requests to the Prometheus server.

Logs capture discrete, timestamped events with full contextual detail — error messages, stack traces, request identifiers and variable values — that aggregated metrics cannot retain. Metrics reveal that a failure occurred; logs explain why, satisfying the need to diagnose specific incidents rather than merely detect anomalies.

Traces capture the full causal path of a single request as it propagates across microservices, preserving parent-child span relationships and per-hop timings. High-cardinality metrics aggregate values and cannot reconstruct that request-scoped journey, so traces uniquely satisfy the need for end-to-end context when debugging one slow request.

In Prometheus relabelling, `$1` references the first regex capture group, satisfying the `label_replace` requirement to extract and substitute a matched substring. The `replacement` field uses `$n` syntax, where `n` is the capture group index, so `$1` inserts the first parenthesised match into the target label.

A gauge represents a value that can rise or fall arbitrarily over time, such as the current count of active connections. Counters only increase monotonically, so they cannot accurately reflect a connection total that decreases when sessions close.

Email addresses are effectively unique per user, so each distinct value spawns its own time series. That unbounded label-value growth multiplies active series and index size, which is precisely the high-cardinality condition the stem asks about.

Webhook receivers let Alertmanager POST a JSON payload to any HTTP endpoint, satisfying the requirement for a standard notification route. This is one of the built-in receiver types configured under `receivers:` in the Alertmanager configuration file, alongside email and Slack, requiring no external integration component.

Labels form part of the metric's unique time series identity. Embedding unbounded dynamic values such as request IDs, user IDs or timestamps creates a new series per value, exploding cardinality and overwhelming the time-series database with memory and storage pressure.

Saturation quantifies how full a resource is, typically expressed as queued work awaiting service, such as requests in a queue or threads waiting. It signals impending degradation before latency or errors appear, matching the Four Golden Signals definition the stem requests.

Alertmanager supports receivers such as email, Slack, PagerDuty, webhook and Opsgenie for delivering notifications. SQL is a database technology, not a notification integration, so it cannot be configured as a valid receiver type in the Alertmanager routing configuration.

Labels attach key-value metadata to an alert, which Alertmanager matches against routes, silences and inhibition rules to determine where and whether notifications are delivered. This satisfies the stem's routing and filtering requirement, distinguishing labels from annotations, which only carry descriptive text.

A sample is the smallest unit of data in the Prometheus model: a single float64 value paired with a millisecond-precision timestamp. A time series is an ordered sequence of such samples identified by its metric name and labels.

Each distinct label combination creates a separate time series held in memory. With constant traffic but rising cardinality, new series accumulate faster than stale ones are garbage-collected, so memory grows linearly. This directly explains the symptom described in the stem.

Metrics are numeric, aggregated and cheap to store, making them suited to dashboards and trend analysis over time. Logs capture individual discrete events with rich context, better for debugging specific incidents. This aggregate-versus-event axis is the defining difference.

The != operator is Prometheus's label matcher for inequality, so environment!="production" selects every time series of http_requests_total whose environment label holds any value other than production. This satisfies the stem's requirement to exclude production while retaining all other environments.

Prometheus retains scraped samples for 15 days by default, after which the oldest blocks are deleted. This satisfies the stem's constraint because no explicit retention flag is required; the default applies unless overridden via storage.tsdb.retention.time.

Delta computes the difference between a sample and its predecessor, so it requires a range vector to supply that historical window; instant vectors hold only one value per series and cannot feed it. This satisfies the stem's requirement for functions designed specifically around range vectors.

Blackbox_exporter performs active probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP, returning metrics such as response time, status code and TLS certificate expiry. This satisfies the stem's requirement for the primary function, since it monitors externally observable behaviour rather than internal host metrics.

Gauge is one of Prometheus's four native metric types, representing a value that can rise or fall arbitrarily, such as temperature or memory usage. It is exposed through client libraries and stored directly in the time series database.

Pushgateway retains pushed metrics indefinitely, so stale series from terminated batch jobs persist until removed via the API or by deleting the entire group. This satisfies the scenario's requirement to identify genuine Pushgateway behaviour: unlike pull-based exporters, no scrape-based expiry or automatic garbage collection occurs.

A unique transaction ID label creates a new time series for every transaction, so cardinality grows without bound as traffic increases. This directly satisfies the stem's high-cardinality cause, since Prometheus indexes each distinct label combination separately in its TSDB.

prometheus.MustRegister() registers a collector with the default registry and panics on error, which is the idiomatic call in Go client libraries. It ensures the metric is exposed on the /metrics endpoint without requiring explicit error handling.

PromQL follows standard arithmetic precedence, so multiplication and division bind more tightly than addition and subtraction. Thus in an expression mixing * and +, the multiplication is evaluated first, matching conventional mathematical ordering rather than strict left-to-right evaluation.

Alerting on causes (CPU, disk) fires on conditions users may never feel. Symptom-based alerts target observable service degradation, such as elevated error rates or latency, so pages reflect actual user impact and reduce noise from transient internal fluctuations.

When a scrape exceeds scrape_timeout, Prometheus aborts the HTTP request and records the target as down, incrementing up=0 and scrape failure metrics. The sample set is discarded, so the scrape is marked failed rather than partially ingested.

`rate()` calculates the per-second average rate of increase of a counter over a specified range, handling counter resets and extrapolating to the range boundaries. This directly satisfies the stem's requirement to convert a monotonically increasing counter into a per-second rate, unlike `irate()`, which uses only the last two samples.

Scraping requires the Prometheus server to reach the target's metrics endpoint over the network. Firewalls, DNS failures, routing problems or unreachable hosts prevent the HTTP request completing, so the scrape fails outright. This satisfies the stem's requirement for a common scraping failure cause.

The `expr` field holds the PromQL expression that Prometheus evaluates on each evaluation interval; when the result is non-empty, the alert transitions to pending or firing. It is mandatory in every alerting rule, satisfying the stem's requirement for a valid component of an Alerting Rule file.

Prometheus scrapes targets on a fixed interval, so metrics must be exposed consistently rather than only when they change. Continuously exposing the current value lets each scrape capture the state, satisfying the periodic-exposure requirement without gaps.

stddev is a valid PromQL aggregation operator, computing the standard deviation across the grouped sample values. It accepts a by or without clause and operates over an instant vector, matching the aggregation requirement in the stem.

Registering the metric with the OpenTelemetry meter provider creates the instrument the application emits values through, satisfying the requirement to expose a new metric. Without a registered instrument, recorded measurements have no defined name, type or unit, so no metric is exported to the backend.

Gauges represent a single numerical value that can arbitrarily increase or decrease, such as temperature or current memory usage. Prometheus exposes this as a valid metric type, satisfying the question's requirement for three valid types alongside counter and histogram.

Counters reset to zero on restart, so raw deltas go negative. PromQL's rate() detects the reset and treats the drop as a wrap-around, extrapolating the increase across the interval rather than reporting a spurious negative slope, which would otherwise distort the per-second average.

The Alertmanager UI is a built-in web interface served by Alertmanager itself, showing active alerts, silences and grouping state. It satisfies the stem because Prometheus's own expression browser does not display fired alerts in this way.

The =~ operator applies RE2 regular expression matching, and the anchored pattern ^5.* requires the status label value to begin with the digit 5, matching any trailing characters. This selects exactly the time series whose status starts with 5.

Rate() suits alerting on slow-moving counters because it extrapolates across the full lookback window, producing a stable per-second average even when few samples exist. Irate() uses only the last two raw samples, so sparse counters yield erratic, spiky values that trigger false alerts. This satisfies the stem's slow-moving constraint.

Prometheus evaluates the inner expression independently at each step across the 30-minute window, spaced at the 1-minute resolution, producing a range vector. This satisfies the subquery's requirement to apply an instant-vector function over time, which a plain range selector cannot do.

Grafana supplies the visualisation layer Prometheus lacks natively, rendering its time-series data into configurable panels, graphs and dashboards. This satisfies the stem's requirement for a dashboarding feature, since Prometheus itself offers only basic expression-browser graphing rather than persistent, shareable, customisable dashboards.

delta() computes the difference between the first and last sample values within a range vector, showing absolute change for a gauge over the window. It satisfies the stem's requirement for a function analysing gauge changes over time.

Subqueries bridge the gap between instant and range vectors: wrapping an instant vector expression in brackets with a range gives functions like max_over_time() the range vector input they require, which a bare instant vector cannot supply.

A gauge records a value that can rise or fall arbitrarily at any moment, such as the current count of active sessions. Counters only increase or reset, histograms and summaries track distributions of observations, so neither represents an instantaneous fluctuating total.

Recording rules evaluate their expression and write the resulting samples back into the Prometheus time-series database under the new metric name. The rule does not create a separate store; results persist in the TSDB alongside scraped metrics.

Static configuration files let Prometheus scrape targets listed explicitly in prometheus.yml, without any external registry. This satisfies the scenario's need for a file-based service discovery mechanism, one of the standard target sources alongside Consul and Kubernetes API discovery.

`increase()` extrapolates the counter's growth across the 5-minute range vector, automatically handling counter resets and scaling to the window boundaries. This satisfies the stem's requirement for the rise in `http_requests_total` over exactly 5 minutes, unlike `delta()`, which ignores resets and suits gauges rather than counters.

`max_over_time` is the correct range-vector function here: it evaluates each time series returned by `node_cpu_load` across the 30-minute window and returns the highest sample value, satisfying the "maximum over the past 30 minutes" constraint. `max()` instead aggregates across series at a single instant, not over time.

Tracking memory consumption is a core node_exporter use case because it exposes host-level metrics such as node_memory_MemAvailable_bytes and node_memory_MemTotal_bytes from /proc/meminfo. This satisfies the scenario's requirement for operating system memory visibility, which application-level instrumentation cannot provide.

The Pushgateway accepts metrics from short-lived jobs that Prometheus cannot reliably scrape within their 30-second lifetime. Because the batch job terminates before a scrape interval elapses, it must push its metrics to the Pushgateway, which retains them for Prometheus to collect afterwards.

histogram_quantile() estimates the 0.99 quantile from the le bucket series, and rate() over the 5m range converts cumulative bucket counters into per-second rates. This satisfies the constraint of computing the 99th percentile latency from a histogram rather than a summary.

Prometheus evaluates each alerting rule's expression against the query engine; malformed syntax or invalid functions in the PromQL cause evaluation to fail, producing exactly this error. Correcting the expression's syntax resolves the alerting rule evaluation error.

An instant vector selector returns one sample per matching series at the evaluation timestamp, filtered by metric name and label matchers such as job="api-server". It does not range over time or aggregate; it yields current values for the selected series.

topk is an aggregation operator that returns the k time series with the highest instantaneous values for the given metric, sorted descending. It directly satisfies the 'top 5' ranking constraint, unlike sum, avg or count, which aggregate across series rather than selecting individual ones.

Underscore-prefixed metric names are reserved for internal use by the monitoring system, so user-defined metrics cannot claim them. This satisfies the stem's constraint on naming impact: such metrics are hidden from standard queries and dashboards, and Prometheus-style tooling treats them as non-public, preventing collisions with system-generated series.

Exponentiation binds tighter than multiplication and addition, so PromQL evaluates `^` first, then `*`, then `+`, matching standard mathematical precedence. This satisfies the stem's requirement to identify the highest-precedence operator among the three, since `^` outranks both `*` and `+` in every expression.

Node Exporter runs on Linux hosts and exposes hardware and kernel metrics such as CPU, memory, disk and network statistics in Prometheus format. It is one of the most widely deployed official exporters, making it a common exporter type.

deriv() fits a linear regression over the range, treating values as ordinary numbers. When a counter resets to zero, that drop is read as a genuine negative slope, yielding large negative rates instead of the reset-corrected increase rate() provides.

Running identical Prometheus instances in parallel satisfies the horizontal scaling requirement: each replica scrapes the same targets independently, so queries can be served by any instance and a failed replica does not lose data. This removes the single-node bottleneck without needing sharding or federation.

Each unique user ID creates a separate time series, so memory grows with user count rather than staying bounded. Label values must have low cardinality; unbounded values like user IDs explode the index and memory footprint, which is exactly the spike observed.

Each unique label value combination creates a distinct time series. Unbounded or high-churn label names multiply series combinatorially, overwhelming Prometheus memory and storage. Fixing label names constrains that growth, directly preventing the cardinality explosion the stem asks about.

A gauge represents a value that can rise or fall arbitrarily, such as current memory usage, which fluctuates as a process allocates and releases memory. Counters and histograms cannot capture this instantaneous, bidirectional measurement correctly.